Algebraic geometry and statistical learning theory

This page intentionally left blank

CAMBRIDGE MONOGRAPHS ON APPLIED AND COMPUTATIONAL MATHEMATICS Series Editors M. J. ABLOWITZ, S. H. DAVIS, E. J. HINCH, A. ISERLES, J. OCKENDEN, P. J. OLVER

25

Algebraic Geometry and Statistical Learning Theory

The Cambridge Monographs on Applied and Computational Mathematics reflect the crucial role of mathematical and computational techniques in contemporary science. The series publishes expositions on all aspects of applicable and numerical mathematics, with an emphasis on new developments in this fast-moving area of research. State-of-the-art methods and algorithms as well as modern mathematical descriptions of physical and mechanical ideas are presented in a manner suited to graduate research students and professionals alike. Sound pedagogical presentation is a prerequisite. It is intended that books in the series will serve to inform a new generation of researchers. The series includes titles in the Library of Computational Mathematics, published under the auspices of the Foundations of Computational Mathematics organisation. The Library of Computational Mathematics is edited by the following editorial board: Felipe Cucker (Managing Editor), Ron Devore, Nick Higham, Arieh Iserles, David Mumford, Allan Pinkus, Jim Renegar, Mike Shub.

Also in this series: Simulating Hamiltonian Dynamics, B. Leimkuhler and Sebastian Reich Collocation Methods for Volterra Integral and Related Functional Differential, Hermann Brunner Topology for Computing, Afra J. Zomorodian Scattered Data Approximation, Holger Wendland Matrix Preconditioning Techniques and Applications, Ke Chen Spectral Methods for Time-Dependent Problems, Jan Hesthaven, Sigal Gottlieb and David Gottlieb The Mathematical Foundations of Mixing, Rob Sturman, Julio M. Ottino and Stephen Wiggins Curve and Surface Reconstruction, Tamal K. Dey Learning Theory, Felipe Cucker and Ding Xuan Zhou

Algebraic Geometry and Statistical Learning Theory SUMIO WATANABE Tokyo Institute of Technology

CAMBRIDGE UNIVERSITY PRESS

Cambridge, New York, Melbourne, Madrid, Cape Town, Singapore, São Paulo, Delhi, Dubai, Tokyo Cambridge University Press The Edinburgh Building, Cambridge CB2 8RU, UK Published in the United States of America by Cambridge University Press, New York www.cambridge.org Information on this title: www.cambridge.org/9780521864671 © S. Watanabe 2009 This publication is in copyright. Subject to statutory exception and to the provision of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press. First published in print format 2009 ISBN-13

978-0-511-65153-3

eBook (NetLibrary)

ISBN-13

978-0-521-86467-1

Hardback

Cambridge University Press has no responsibility for the persistence or accuracy of urls for external or third-party internet websites referred to in this publication, and does not guarantee that any content on such websites is, or will remain, accurate or appropriate.

Contents

Preface

page vii

1

Introduction 1.1 Basic concepts in statistical learning 1.2 Statistical models and learning machines 1.3 Statistical estimation methods 1.4 Four main formulas 1.5 Overview of this book 1.6 Probability theory

1 1 10 18 26 41 42

2

Singularity theory 2.1 Polynomials and analytic functions 2.2 Algebraic set and analytic set 2.3 Singularity 2.4 Resolution of singularities 2.5 Normal crossing singularities 2.6 Manifold

48 48 50 53 58 66 72

3

Algebraic geometry 3.1 Ring and ideal 3.2 Real algebraic set 3.3 Singularities and dimension 3.4 Real projective space 3.5 Blow-up 3.6 Examples

77 77 80 86 87 91 99

4

Zeta function and singular integral 4.1 Schwartz distribution 4.2 State density function v

105 105 111

vi

Contents

4.3 4.4 4.5

Mellin transform Evaluation of singular integral Asymptotic expansion and b-function

116 118 128

5

Empirical processes 5.1 Convergence in law 5.2 Function-valued analytic functions 5.3 Empirical process 5.4 Fluctuation of Gaussian processes

133 133 140 144 154

6

Singular learning theory 6.1 Standard form of likelihood ratio function 6.2 Evidence and stochastic complexity 6.3 Bayes and Gibbs estimation 6.4 Maximum likelihood and a posteriori

158 160 168 177 203

7

Singular learning machines 7.1 Learning coefficient 7.2 Three-layered neural networks 7.3 Mixture models 7.4 Bayesian network 7.5 Hidden Markov model 7.6 Singular learning process 7.7 Bias and variance 7.8 Non-analytic learning machines

217 217 227 230 233 234 235 239 245

8

Singular statistics 8.1 Universally optimal learning 8.2 Generalized Bayes information criterion 8.3 Widely applicable information criteria 8.4 Singular hypothesis test 8.5 Realization of a posteriori distribution 8.6 From regular to singular

249 249 252 253 258 264 274

Bibliography Index

277 284

Preface

In this book, we introduce a fundamental relation between algebraic geometry and statistical learning theory. A lot of statistical models and learning machines used in information science, for example, mixtures of probability distributions, neural networks, hidden Markov models, Bayesian networks, stochastic context-free grammars, and topological data analysis, are not regular but singular, because they are nonidentifiable and their Fisher information matrices are singular. In such models, knowledge to be discovered from examples corresponds to a singularity, hence it has been difficult to develop a mathematical method that enables us to understand statistical estimation and learning processes. Recently, we established singular learning theory, in which four general formulas are proved for singular statistical models. Firstly, the log likelihood ratio function of any singular model can be represented by the common standard form even if it contains singularities. Secondly, the asymptotic behavior of the evidence or stochastic complexity is clarified, giving the result that the learning coefficient is equal to the maximum pole of the zeta function of a statistical model. Thirdly, there exist equations of states that express the universal relation of the Bayes quartet. We can predict Bayes and Gibbs generalization errors using Bayes and Gibbs training errors without any knowledge of the true distribution. And lastly, the symmetry of the generalization and training errors holds in the maximum likelihood and a posteriori estimators. If one-point estimation is applied to statistical learning, the generalization error is equal to the maximum value of a Gaussian process on a real analytic set. This book consists of eight chapters. In Chapter 1, an outline of singular learning theory is summarized. The main formulas proved in this book are overviewed without mathematical preparation in advance. In Chapter 2, the definition of a singularity is introduced. Resolution of singularities is the essential theorem on which singular learning theory is constructed. In Chapter 3, vii

viii

Preface

several basic concepts in algebraic geometry are briefly explained: ring and ideal, correspondence between algebra and geometry, and projective spaces. The algorithm by which a resolution map is found using recursive blow-ups is also described. In Chapter 4, the relation between the singular integral and the zeta function of a singular statistical model is clarified, enabling some inequalities used in Chapter 6 to be proved. In Chapter 5, function-valued random variables are studied and convergence in law of empirical processes is proved. In Chapter 6, the four main formulas are proved: the standard form of the likelihood ratio function, the asymptotic expansion of the stochastic complexity, the equations of states in a Bayes quartet, and the symmetry of generalization and training errors in one-point estimation. In Chapters 7 and 8, applications of singular learning theory to information science are summarized and discussed. This book involves several mathematical fields, for example, singularity theory, algebraic geometry, Schwartz distribution, and empirical processes. However, these mathematical concepts are introduced in each chapter for those who are unfamiliar with them. No specialized mathematical knowledge is necessary to read this book. The only thing the reader needs is a mathematical mind seeking to understand the real world. The author would like to thank Professor Shun-ichi Amari for his encouragement of this research. Also the author would like to thank Professor Bernd Sturmfels for his many helpful comments on the study and this book. In this book, the author tries to build a bridge between pure mathematics and real-world information science. It is expected that a new research field will be opened between algebraic geometry and statistical learning theory. Sumio Watanabe

1 Introduction

In this book, we study a system which perceives the real world. Such a system has to estimate an information source by observation. If the information source is a probability distribution, then the estimation process is called statistical learning, and the system is said to be a statistical model or a learning machine. A lot of statistical models have hierarchical layers, hidden variables, a collection of modules, or grammatical structures. Such models are nonidentifiable and contain singularities in their parameter spaces. In fact, the map from a parameter to a statistical model is not one-to-one, and the Fisher information matrix is not positive definite. Such statistical models are called singular. It has been difficult to examine the learning process of singular models, because there has been no mathematical theory for such models. In this book, we establish a mathematical foundation which enables us to understand the learning process of singular models. This chapter gives an overview of the book before a rigorous mathematical foundation is developed.

1.1 Basic concepts in statistical learning To describe what statistical learning is, we need some basic concepts in probability theory. For the reader who is unfamiliar with probability theory, Section 1.6 summarizes the key results.

1.1.1 Random samples Let N be a natural number and RN be the N -dimensional real Euclidean space. We study a case when information data are represented by vectors in RN . 1

2

Introduction

Samples Dn = {X1, X2 , …, Xn}

Statistical estimation

Random sampling Generalization error K(q||p* )

True q(x)

Estimated p*(x)

Fig. 1.1. Statistical learning

Firstly, using Figure 1.1, let us explain what statistical learning is. Let (
, B, P ) be a probability space and X :
→ RN be a random variable which is subject to a probability distribution q(x)dx. Here q(x) is a probability density function and dx is the Lebesgue measure on RN . We assume that random variables X1 , X2 , . . . , Xn are independently subject to the same probability distribution as X, where n is a natural number. In statistical learning theory, q(x) is called a true probability density function, and random variables X1 , X2 , . . . , Xn are random samples, examples, random data, or training samples. The probability density function of the set of independent random samples is given by q(x1 )q(x2 ) · · · q(xn ). In practical applications, we obtain their realizations by observations. The natural number n is said to be the number of random samples. The set of random samples or random data is denoted by Dn = {X1 , X2 , . . . , Xn }. The main purpose of statistical learning is to construct a method to estimate the true probability density function q(x) from the set Dn . In this book, we study a method which employs a parametric probability density function. A conditional probability density function p(x|w) of x ∈ RN for a given parameter w ∈ W is called a learning machine or a statistical model, where W is the set of all parameters. Sometimes the notation p(x|w) = pw (x) is used. Then w → pw gives a map from the parameter to the probability density function. We mainly study the case when W is a subset of the d-dimensional real Euclidean space Rd or a d-dimensional real analytic manifold. An a priori probability density function ϕ(w) is defined on W . We assume that, for any w ∈ W , the support of the probability distribution p(x|w) is equal to that of

1.1 Basic concepts in statistical learning

3

q(x) and does not depend on w. That is to say, for any w ∈ W , {x ∈ RN ; p(x|w) > 0} = {x ∈ RN ; q(x) > 0}, where S is the closure of a set S in RN . In statistical learning or statistical inference, the main study concerns a method to produce a probability density function p ∗ (x) on RN based on the set of random samples Dn , using the parametric model p(x|w). Such a function Dn → p∗ (x) is called a statistical estimation method or a learning algorithm. Note that there are a lot of statistical estimation methods and learning algorithms. The probability density function p∗ (x), which depends on the set of random variables Dn , is referred to as the estimated or trained probability density function. Generally, it is expected that the estimated probability density function p ∗ (x) is a good approximation of the true density function q(x), and that it becomes better as the number of random samples increases.

1.1.2 Kullback–Leibler distance In order to compare two probability density functions, we need a quantitative value which shows the difference between two probability density functions. Definition 1.1 (Kullback–Leibler distance) For given probability density functions q(x), p(x) > 0 on an open set A ⊂ RN , the Kullback–Leibler distance or relative entropy is defined by
q(x) q(x) log dx. K(qp) = p(x) A If the integral is not finite, K(qp) is defined as K(qp) = ∞. Theorem 1.1 Assume that q(x), p(x) > 0 are continuous probability density functions on an open set A. Then the following hold. (1) For arbitrary q(x), p(x), K(qp) ≥ 0. (2) K(qp) = 0 if and only if q(x) = p(x) for any x ∈ A. Proof of Theorem 1.1 Let us introduce a real function S(t) = − log t + t − 1

(0 < t < ∞).

4

Introduction

Then S(t) ≥ 0, and S(t) = 0 if and only if t = 1. Since  p(x)dx = 1,
 p(x)  q(x) S dx, K(qp) = q(x) A



q(x)dx =

which shows (1). Assume K(qp) = 0. Since S(p(x)/q(x)) is a nonnegative and continuous function of x, S(p(x)/q(x)) = 0 for any x ∈ A, which is equiv
alent to p(x) = q(x). Remark 1.1 The Kullback–Leibler distance is called the relative entropy in physics. In information theory and statistics, the Kullback–Leibler distance K(qp) represents the loss of the system p(x) for the information source q(x). The fact that K(qp) is not symmetric for q(x) and p(x) may originate from the difference of their roles. Historically, relative entropy was first defined by Boltzmann and Gibbs in statistical physics in the nineteenth century. In the twentieth century it was found that relative entropy plays a central role in information theory and statistical estimation. We can measure the difference between the true density function q(x) and the estimated one p ∗ (x) by the Kullback–Leibler distance:
q(x) K(qp ∗ ) = q(x) log ∗ dx. p (x) In statistical learning theory, K(qp ∗ ) is called the generalization error of the method of statistical estimation Dn → p∗ . In general, K(qp ∗ ) is a measurable function of the set of random samples Dn , hence it is also a real-valued random variable. The training error is defined by 1 q(Xi ) log ∗ , n i=1 p (Xi ) n

Kn (qp ∗ ) =

which is also a random variable. One of the main purposes of statistical learning theory is to clarify the probability distributions of the generalization and training errors for a given method of statistical estimation. The expectation values E[K(qp∗ )] and E[Kn (qp ∗ )] are respectively called the mean generalization error and the training error. If the mean generalization error is smaller, the statistical estimation method is more appropriate. The other purpose of statistical learning theory is to establish a mathematical relation between the generalization error and the training error. If the generalization error can be estimated from the training error, we can select the suitable model or hyperparameter among several statistical possible models.

1.1 Basic concepts in statistical learning

5

Definition 1.2 (Likelihood function) For a given set of random samples Dn and a statistical model p(x|w), the likelihood function Ln (w) of w ∈ W ⊂ Rd is defined by Ln (w) =

n 

p(Xi |w).

i=1

If p(x|w) = q(x), then Ln (w) is equal to the probability density function of Dn . Definition 1.3 (Log likelihood ratio function) For a given true distribution q(x) and a parametric model p(x|w), the log density ratio function f (x, w), the Kullback–Leibler distance K(w), and the log likelihood ratio function Kn (w) are respectively defined by f (x, w) = log
K(w) =

q(x) , p(x|w)

q(x)f (x, w)dx, 1 f (Xi , w), n i=1

(1.1) (1.2)

n

Kn (w) =

(1.3)

where Kn (w) is sometimes referred to as an empirical Kullback–Leibler distance. From the definition, E[f (X, w)] = E[Kn (w)] = K(w). By using the empirical entropy 1 log q(Xi ), n i=1 n

Sn = −

(1.4)

the likelihood function satisfies 1 (1.5) − log Ln (w) = Kn (w) + Sn . n The empirical entropy Sn does not depend on the parameter w, hence maximization of the likelihood function Ln (w) is equivalent to minimization of Kn (w). Remark 1.2 If a function S(t) satisfies S  (t) > 0 and S(1) = 0, then
 p(x)  q(x) S dx q(x) A

6

Introduction

has the same property as the Kullback–Leibler distance in Theorem 1.1. For example, using S(t) = (1 − t a )/a for a given a, 0 < a < 1, a generalized distance is defined by
 1 − (p(x)/q(x))a  (a) dx. K (qp) = q(x) a For example, if a = 1/2, Hellinger’s distance is derived,
  K (1/2) (qp) = ( q(x) − p(x))2 dx. In general Jensen’s inequality claims that, for any measurable function F (x),
 
q(x)F (x)dx , q(x)S(F (x))dx ≥ S where the equality holds if and only if F (x) is a constant function on q(x) > 0. Hence K (a) (qp) ≥ 0 and K (a) (qp) = 0 if and only if q(x) = p(x) for all x. Hence K (a) (qp) indicates a difference of p(x) from q(x). The Kullback– Leibler distance is formally obtained by a → +0. For arbitrary probability density functions q(x), p(x), the Kullback–Leibler distance satisfies K(qp) ≥ K (a) (qp), because
 af (x, w) + e−af (x,w) − 1  K(qp) − K (a) (qp) = q(x) dx ≥ 0. a Moreover, if K(qp) = 0 then Ka (qp) = 1. a→+0 K(qp) lim

Therefore, from the learning theory of K(qp), we can construct a learning theory of K (a) (qp). Remark 1.3 If E[K(w)] < ∞ then, by the law of large numbers, the convergence in probability Kn (w) → K(w) holds for each w ∈ W . Furthermore, if E[K(w)2 ] < ∞ then, by the central limit theorem, √ n(Kn (w) − K(w)) converges in law to the normal distribution, for each w ∈ W . Therefore, for each w, the convergence in probability 1 − log Ln (w) → K(w) − S n

1.1 Basic concepts in statistical learning

7

holds, where S is the entropy of the true distribution q(x),
S = − q(x) log q(x)dx. It might seem that minimization of Kn (w) is equivalent to minimization of K(w). If these two minimization problems were equivalent, then maximization of Ln (w) would be the best method in statistical estimation. However, minimization and expectation cannot be commutative. E[min Kn (w)] = min E[Kn (w)] = min K(w). w

w

w

(1.6)

Hence maximization of Ln (w) does not mean minimization of K(w). This is the basic reason why statistical learning does not result in a simple optimization problem. To clarify the difference between K(w) and Kn (w), we have to study the meaning of the convergence Kn (w) → K(w) in a functional space. There are many nonequivalent functional topologies. For example, sup-norm, Lp -norm, weak topology of Hilbert space L2 , Schwartz distribution topology, and so on. It strongly depends on the topology of the function space whether the convergence Kn (w) → K(w) holds or not. The Bayes estimation corresponds to the Schwartz distribution topology, whereas the maximum likelihood or a posteriori method corresponds to the sup-norm. This difference strongly affects the learning results in singular models.

1.1.3 Fisher information matrix Definition 1.4 (Fisher information matrix) For a given statistical model or a learning machine p(x|w), where x ∈ RN and w ∈ Rd , the Fisher information matrix I (w) = {Ij k (w)} (1 ≤ j, k ≤ d) is defined by


  ∂  ∂ log p(x|w) log p(x|w) p(x|w) dx Ij k (w) = ∂wj ∂wk

if the integral is finite. By the definition, the Fisher information matrix is always symmetric and positive semi-definite. It is not positive definite in general. In some statistics textbooks, it is assumed that the Fisher information matrix is positive definite, and that the Cramer–Rao inequality is proven; however, there are a lot of statistical models and learning machines in which Fisher information matrices

8

Introduction

have zero eigenvalue. The Fisher information matrix is positive definite if and only if d  ∂ log p(x|w) j =1 ∂wj is linearly independent as a function of x on the support of p(x|w). Since ∂ ∂ log p(x|w) = − f (x, w), ∂wj ∂wj the Fisher information matrix is positive definite if and only if d  ∂ f (x, w) j =1 ∂wj  is linearly independent as a function of x. By using p(x|w)dx = 1 for an arbitrary w, it is easy to show that
  ∂2 log p(x|w) p(x|w) dx. Ij k (w) = − ∂wj ∂wk If q(x) = p(x|w0 ), then Ij k (w0 ) =

∂2 K(w0 ). ∂wj ∂wk

Therefore, the Fisher information matrix is equal to the Hessian matrix of the Kullback–Leibler distance at the true parameter. Remark 1.4 If the Fisher information matrix is positive definite in a neighborhood of the true parameter w0 , K(w) > 0 (w = w0 ) holds, and the Kullback– Leibler distance can be approximated by the positive definite quadratic form, then K(w) ≈ 12 (w − w0 ) · I (w0 )(w − w0 ), where u · v shows the inner product of two vectors u, v. If the Fisher information matrix is not positive definite, then K(w) cannot be approximated by any quadratic form in general. This book establishes the mathematical foundation for the case when the Fisher information matrix is not positive definite. Remark 1.5 (Cramer–Rao inequality) Assume that random samples {Xi ; i = n  1, 2, . . . , n} are taken from the probability density function p(xi |w), where i=1

w = (w1 , w2 , . . . , wd ) ∈ Rd . A function from random samples to the parameter space {uj (x1 , x2 , . . . , xn ); j = 1, 2, . . . , d} ∈ Rd

1.1 Basic concepts in statistical learning

9

is called an unbiased estimator if it satisfies
E[uj (X1 , X2 , . . . , Xn ) − wj ] ≡ (uj (x1 , x2 , . . . , xn ) − wj ) ×

n 

p(xi |w)dxi = 0

i=1

for arbitrary w ∈ Rd . Under certain are conditions which ensure that (∂/∂wk ) are commutative for arbitrary j, k, 0=



dxj and

∂ E[uj (X1 , X2 , . . . , Xn ) − wj ] ∂wk

= E[(uj − wj )

n  ∂ log p(Xi , w)] − δj k . ∂w k i=1

Therefore, δj k = E[(uj − wj )

n  ∂ log p(Xi , w)]. ∂wk i=1

For arbitrary d-dimensional vectors a = (aj ), b = (bk ), (a · b) = E

d



d  d 

aj (uj − wj )

j =1

k=1 i=1

bk

 ∂ log p(Xi , w) . ∂wk

By applying the Cauchy–Schwarz inequality (a · b)2 ≤ n (a · V a)(b · I (w)b),

(1.7)

where V = (Vj k ) is the covariance matrix of u − w, Vj k = E[(uj − wj )(uk − wk )] and I (w) is the Fisher information matrix. If I (w) is positive definite, by putting a = I (w)1/2 c, b = I (w)−1/2 c, it follows that c2 ≤ n (c · I (w)1/2 V I (w)1/2 c) holds for arbitrary vector c, hence V ≥

I (w)−1 . n

(1.8)

10

Introduction

This relation, the Cramer–Rao inequality, shows that the covariance matrix of any unbiased estimator cannot be made smaller than the inverse of the Fisher information matrix. If I (w) has zero eigenvalue and b is an eigenvector for zero eigenvalue, eq.(1.7) shows that either V is not a finite matrix or no unbiased estimator exists. For statistical models which have a degenerate Fisher information matrix, we have no effective unbiased estimator in general.

1.2 Statistical models and learning machines 1.2.1 Singular models Definition 1.5 (Identifiablity) A statistical model or a learning machine p(x|w) (x ∈ RN , w ∈ W ⊂ Rd ) is called identifiable if the map W  w → p( |w) is one-to-one, in other words, p(x|w1 ) = p(x|w2 ) (∀x ∈ Rd ) =⇒ w1 = w2 . A model which is not identifiable is called nonidentifiable or unidentifiable. Definition 1.6 (Positive definite metric) A statistical model or a learning machine p(x|w) (x ∈ RN , w ∈ W ⊂ Rd ) is said to have a positive definite metric if its Fisher information matrix I (w) is positive definite for arbitrary w ∈ W. If a statistical model does not have a positive definite metric, it is said to have a degenerate metric. Definition 1.7 (Singular statistical models) Assume that the support of the statistical model p(x|w) is independent of w. A statistical model p(x|w) is said to be regular if it is identifiable and has a positive definite metric. If a statistical model is not regular, then it is called strictly singular. The set of singular statistical models consists of both regular and strictly singular models. Mathematically speaking, identifiability is neither a necessary nor a sufficient condition of positive definiteness of the Fisher information matrix. In fact, if p(x|a) (x, a ∈ R1 ) is a regular statistical model, then p(x|a 3 ) is identifiable but has a degenerate Fisher information matrix. Also p(x|a 2 ) (|a| > 1) has a nondegenerate Fisher information matrix but is nonidentifiable. These are trivial examples in which an appropriate transform or restriction of a parameter makes models regular. However, a lot of statistical models and learning machines used in information science have simultaneously nonidentifiability and a degenerate metric.

1.2 Statistical models and learning machines

11

Moreover, they contain a lot of singularities which cannot be made regular by any transform or restriction. In this book, we mainly study singular statistical models or singular learning machines. The following statistical models are singular statistical models. (1) (2) (3) (4) (5) (6) (7) (8) (9) (10)

Layered neural networks Radial basis functions Normal mixtures Binomial and multinomial mixtures Mixtures of statistical models Reduced rank regressions Boltzmann machines Bayes networks Hidden Markov models Stochastic context-free grammar

These models play the central role of information processing systems in artificial intelligence, pattern recognition, robotic control, time series prediction, and bioinformatics. They determine the preciseness of the application systems. Singular models are characterized by the following features. (1) (2) (3) (4) (5) (6)

They are made by superposition of parametric functions. They have hierarchical structures. They contain hidden variables. They consist of several information processing modules. They are designed to obtain hidden knowledge from random samples. They estimate the probabilistic grammars.

In singular statistical models, the knowledge or grammar to be discovered corresponds to singularities in general. Figure 1.2 shows an example of the correspondence between parameters and probability distributions in normal mixtures. Remark 1.6 (Equivalence relation) The condition that p(x|w1 ) = p(x|w2 ) for arbitrary x does not mean ∂ k p(x|w1 ) ∂ k p(x|w2 ) = ∂w1k ∂w2k

(k = 1, 2, 3, . . .).

Even if p(x|w1 ) ≈ p(x|w2 ), their derivatives are very different in general. The preciseness of statistical estimation is determined by the derivative of p(x|w), hence results of statistical estimations are very different if p(x|w1 ) ≈ p(x|w2 ).

12

Introduction

Fig. 1.2. Map from parameter to probability distribution

One can introduce an equivalence relation ∼ into the set of parameters W , w1 ∼ w2 ⇐⇒ p(x|w1 ) = p(x|w2 ) (∀x). Then the map (W/∼ )  w → pw is one-to-one. However, the quotient set (W/∼ ) is neither the Euclidean space nor a manifold. Therefore, it is still difficult to construct statistical learning theory on (W/∼ ). In this book, we show that there is a birational map in algebraic geometry which enables us to establish singular learning theory. Remark 1.7 (No asymptotic normality) If a model is regular, then the Bayes a posteriori distribution can be approximated by the normal distribution  n  1 exp − (w − w0 )I (w0 )(w − w0 ) , Zn 2 where w0 is the unique parameter such that q(x) = p(x|w0 ). Also the maximum likelihood estimator and the maximum a posteriori estimator are asymptotically subject to the normal distribution. Such a property is called asymptotic normality. However, singular statistical models do not have such a property, with the result that almost all statistical theories using asymptotic normality do not hold in singular statistical models. Remark 1.8 (True generic condition) In a lot of statistical models and learning machines, the set of parameters at which the Fisher information matrices are degenerate W(0) = {w ∈ W ; det(I (w)) = 0} is a measure zero subset in Rd . Hence one might suppose that, in generic cases, the true parameter w0 is seldom contained in W(0) , and that the learning theory assuming det(I (w0 )) > 0 may be sufficient in practical applications. However,

1.2 Statistical models and learning machines

13

this consideration is wrong. On the contrary, in general cases, we have to optimize a statistical model or a learning machine by comparing several probable models and hyperparameters. In such cases, we always examine models under the condition that the optimal parameter lies in a neighborhood of W(0) . Especially in model selection, hyperparameter optimization, or hypothesis testing, we need the theoretical results of the case w0 ∈ W(0) because we have to determine whether w0 ∈ W(0) or not. Therefore, the superficial generic condition det(I (w)) > 0 does not have true generality. Remark 1.9 (Singular theory contains regular theory) Statistical theory of regular models needs identifiability and a nondegenerate Fisher information matrix. In this book, singular learning theory is established on the assumption that neither identifiability nor a positive definite Fisher information matrix is necessary. Of course, even if a model is regular, the singular learning theory holds. In other words, a regular model is understood as a very special example to which singular learning theory can be applied. From the mathematical point of view, singular learning theory contains regular learning theory as a very special part. For example, the concepts AIC (Akaike’s information criterion) and BIC (Bayes information criterion) in regular statistical theory are completely generalized in this book.

1.2.2 Density estimation Let us introduce some examples of regular and singular statistical models. Example 1.1 (Regular model) A parametric probability density function of (x, y) ∈ R2 for a given parameter w = (a, b) ∈ R2 defined by p(x, y|a, b) =

 (x − a)2 + (y − b)2  1 exp − 2π 2

is a regular statistical model, where the set of parameters is W = {(a, b) ∈ R2 }. This is a two-dimensional normal distribution. For given random samples (Xi , Yi ), the likelihood function is  1  1 2 2 exp − {(X − a) + (Y − b) } . i i (2π)n 2 i=1 n

Ln (a, b) =

If the true distribution is given by (a0 , b0 ), q(x, y) = p(x, y|a0 , b0 ),

14

Introduction

the log likelihood ratio function is Kn (a, b) =

a 2 − a02 + b2 − b02 2 n n 1  1    − (a − a0 ) Xi − (b − b0 ) Yi . n i=1 n i=1

The Kullback–Leibler distance is K(a, b) = 12 {(a − a0 )2 + (b − b0 )2 }. Note that K(a, b) = 0 if and only if a = a0 and b = b0 . The Fisher information matrix

1 0 I (a, b) = 0 1 is positive definite for an arbitrary (a, b). Example 1.2 (Singular model) Let us introduce another parametric probability density function of x ∈ R1 defined by (x−b)2  1  x2 p(x|a, b) = √ (1 − a) e− 2 + a e− 2 . 2π

The set of parameters is W = {w = (a, b); 0 ≤ a ≤ 1, −∞ < b < ∞}. This model is called a normal mixture. If the true distribution is given by q(x) = p(x|a0 , b0 ), then the log likelihood ratio function is Kn (a, b) = and

n 1   1 + a0 (exp(b0 Xi − b02 /2) − 1)  log n i=1 1 + a (exp(bXi − b2 /2) − 1)


K(a, b) =

log

 1 + a (exp(b x − b2 /2) − 1)  0 0 0 q(x) dx. 1 + a (exp(bx − b2 /2) − 1)

If a0 b0 = 0, then K(a, b) = 0 is equivalent to a = a0 and b = b0 . In such cases, the Fisher information matrix I (a0 , b0 ) is positive definite. However, if a0 b0 = 0, then K(a, b) = 0 is equivalent to ab = 0, and the Fisher information matrix I (a0 , b0 ) = 0. The function K(a, b) can be expanded as K(a, b) = 12 a 2 b2 + · · · ,

1.2 Statistical models and learning machines

y

a

b

15

Hidden variable

c Visible variable

x1

x2

x3

Fig. 1.3. Bayesian network with hidden unit

which shows that K(a, b) cannot be approximated by any quadratic form. If we make a model selection algorithm or a hypothesis test procedure for this √ model, then we have to study the case K(a, b) ∼ = 1/ n where n is the number of random samples. Hence we have to evaluate the effect of the singularity in the set ab = 0. Example 1.3 (Bayesian network with a hidden unit) Let X1 , X2 , X3 and Y are random variables which take values {−1, 1}. The Bayesian network shown in Figure 1.3 is defined by the probability distribution of X = (X1 , X2 , X3 ) and Y , p(x, y|w) =

1 exp(ax1 y + bx2 y + cx3 y), Z(a, b, c)

where w = (a, b, c) and Z(a, b, c) is a normalizing constant. Let X be a set of visible units and Y a hidden unit. The probability distribution of X is given by the marginal distribution, p(x|w) = =

 1 exp(ax1 y + bx2 y + cx3 y) Z(a, b, c) y=±1 1 cosh(ax1 + bx2 + cx3 ). 2 Z(a, b, c)

By using tanh(axi ) = tanh(a)xi for xi = ±1, and cosh(u + v) = cosh(u) cosh(v) + sinh(u) sinh(v), sinh(u + v) = sinh(u) cosh(v) + cosh(u) sinh(v), we have p(x|w) = 18 {1 + t(a)t(b)x1 x2 + t(b)t(c)x2 x3 + t(c)t(a)x3 x1 },

16

Introduction

where t(a) = tanh(a). Assume that the true distribution is given by q(x) = p(x|0, 0, 0) = 1/8. Then the Kullback–Leibler distance is K(a, b, c) = 12 (a 2 b2 + b2 c2 + c2 a 2 ) + · · · . Therefore q(x) = p(x|a, b, c) ⇐⇒ a = b = 0,

or

b = c = 0,

or

c = a = 0.

The Fisher information matrix is equal to zero at (0, 0, 0). If we want to judge whether the hidden variable Y is necessary to explain a given set of random samples, we should clarify the effect of the singularity of K(a, b, c) = 0.

1.2.3 Conditional probability density Example 1.4 (Regular model) A probability density function of (x, y) ∈ R2 , 1 p(x, y|a, b) = q0 (x) √ exp(− 12 (y − ax − b)2 ), 2π

(1.9)

is a statistical model, where the set of parameters is W = {w = (a, b) ∈ R2 } and q0 (x) is a constant probability density function of x. This model is referred to as a line regression model. If the true distribution is q(x, y), the true conditional probability density function q(y|x) = 

q(x, y) q(x, y  ) dy 

is estimated by the conditional probability density function 1 p(x|y, a, b) = √ exp(− 12 (y − ax − b)2 ). 2π

(1.10)

The two models eq.(1.9) and eq.(1.10) have the same log likelihood ratio function and the same Kullback–Leibler distance, hence the two models are equivalent from a statistical point of view. In other words, estimation of the conditional density function of y for a given x can be understood as the estimation of a joint probability density function of (x, y), if q(x) is not estimated. If the true distribution is given by w0 = (a0 , b0 ), q(x, y) = p(x, y|a0 , b0 ).

1.2 Statistical models and learning machines

17

The log likelihood ratio function is 1   (a 2 − a02 )  1  2  (b2 − b02 ) Xi + Xi + (ab − a0 b0 ) 2 n i=1 2 n i=1 n

Kn (a, b) =

n

n 1 

− (a − a0 )

n 1    Xi Yi − (b − b0 ) Yi . n i=1 n i=1

The Kullback–Leibler distance is
1 (ax + b − a0 x − b0 )2 q0 (x) dx K(a, b) = 2 = 12 (w − w0 ) · I (w − w0 ), where

I=

and

m2 m1

m1 , 1


mi =

x i q0 (x) dx.

The Fisher information matrix is always equal to I , which does not depend on the true parameter w0 . It is positive definite if and only if m2 = m21 . In other words, I is degenerate if and only if the variance of q0 (x) is equal to zero. Example 1.5 (Singular model) Another example of a statistical model of y ∈ R1 for x ∈ R1 is 1 p(x, y|a, b) = q0 (x) √ exp(− 12 (y − a tanh(bx))2 ), 2π where the set of parameters is W = {(a, b) ∈ R2 }. This model is the simplest three-layer neural network. If the true distribution is given by w = (a0 , b0 ), q(x, y) = p(x, y|a0 , b0 ), the log likelihood ratio function is 1  {(Yi − a tanh(bXi ))2 − (Yi − a0 tanh(b0 Xi ))2 }, 2n i=1 n

Kn (a, b) =

and the Kullback–Leibler distance is
1 (a tanh(bx) − a0 tanh(b0 x))2 q0 (x) dx. K(a, b) = 2

18

Introduction

s(bx) a

b x

x

y c

as(bx)+cx

Fig. 1.4. Layered neural network

If a0 b0 = 0, then K(a, b) = 0 if and only if a = a0 and b = b0 , and the Fisher information matrix I (a0 , b0 ) is positive definite. However, if a0 b0 = 0, then K(a, b) = 0 is equivalent to ab = 0, and the Fisher information matrix is degenerate. In practical applications of three-layer neural networks, we have to decide whether a three-layer neural network H 

ah tanh(bh x + ch )

h=1

almost approximates the true regression function or not. In such cases, the more precisely the model approximates the true regression function, the more degenerate the Fisher information matrix is. Therefore, we cannot assume that the Fisher information matrix is positive definite in the model evaluation. Example 1.6 (Layered neural network) Let x, y ∈ R1 and w = (a, b, c) ∈ R3 . The statistical model shown in Figure 1.4, 1 p(y|x, w) = √ exp(− 12 (y − as(bx) − cx)2 ), 2π where s(t) = t + t 2 , is a layered statistical model. If the true distribution is q(y|x) = p(y|x, 0, 0, 0) and if q(x) is the standard normal distribution, then K(a, b, c) = 12 (ab + c)2 + 32 a 2 b2 . Hence q(y|x) = p(y|x, w) ⇐⇒ ab = c = 0. The Fisher information matrix is equal to zero at (0, 0, 0).

1.3 Statistical estimation methods In this section, let us introduce some statistical estimation methods, Bayes and Gibbs estimations, the maximum likelihood and a posteriori estimations.

1.3 Statistical estimation methods

19

1.3.1 Evidence Let Dn = {X1 , X2 , . . . , Xn } be a set of random samples. For a given set of a statistical model p(x|w) and an a priori probability density function ϕ(w), the a posteriori probability density function p(w|Dn ) with the inverse temperature β > 0 is defined by p(w|Dn ) =

n  1 ϕ(w) p(Xi |w)β , Zn i=1

where Zn is the normalizing constant determined so that p(w|Dn ) is a probability density function of w,
Zn =

dw ϕ(w)

n 

p(Xi |w)β .

i=1

If β = 1, p(w|Dn ) is called a strict Bayes a posteriori density; if β = 1, it is a generalized version. When β → ∞, it converges to δ(w − w), ˆ where wˆ is the maximum likelihood estimator. Note that Zn is a measurable function of Dn , hence it is also a random variable. The random variable Zn is called the evidence, the marginal likelihood, or the partition function. Remark 1.10 (Meaning of evidence) If β = 1, Zn = Zn (X1 , X2 , . . . , Xn ) satisfies
dx1 dx2 · · · dxn Zn (x1 , x2 , . . . , xn ) = 1. Therefore, Zn with β = 1 defines a probability density function of Dn for a given pair of p(x|w) and ϕ. In other words, Zn can be understood as a likelihood function of the pair (p(x|w), ϕ(w)). The predictive distribution p(x|Dn ) is defined by
p(x|Dn ) = p(x|w) p(w|Dn ) dw. The Bayes estimation is defined by p∗ (x) = p(x|Dn ), in other words, the Bayes estimation is the map Dn → p∗ (x) = p(x|Dn ).

20

Introduction

The Bayes generalization error Bg is the Kullback–Leibler distance from q(x) to p ∗ (x),
q(x) Bg = q(x) log dx. p(x|Dn ) Here Bg is a measurable function of Dn , hence it is also a random variable. The Bayes training error Bt is defined by 1 q(Xi ) log . n i=1 p(Xi |Dn ) n

Bt =

In Bayes learning theory, there are several important observables. The stochastic complexity, the minus log marginal likelihood, or the free energy is defined by Fn = − log Zn . Since Zn with β = 1 can be understood as the likelihood of the pair p(x|w) and ϕ(w), Fn with β = 1 is the minus log likelihood of them. In analysis of Bg , Bt , and Fn , we have some useful equations. The normalized evidence is defined by Zn0 =

Zn n 

q(Xi )

. β

i=1

Then, by using eq.(1.5), the a posteriori distribution is rewritten as p(w|Dn ) =

1 exp(−nβKn (w)) ϕ(w), Zn0

where Kn (w) is the log likelihood ratio function defined in eq.(1.3), and
Zn0 = dw ϕ(w) exp(−nβKn (w)). In the same way, the normalized stochastic complexity is defined by Fn0 = − log Zn0 . The empirical entropy is given by 1 log q(Xi ). n i=1 n

Sn = − Then

Fn = Fn0 − nβSn .

(1.11)

1.3 Statistical estimation methods

21

By the definition of the predictive distribution, it follows that
dw ϕ(w) p(Xn+1 |w) p(Xn+1 |Dn ) =

n 

p(Xi |w)β

i=1


dw ϕ(w)

n 

.

(1.12)

p(Xi |w)

β

i=1

Theorem 1.2 For an arbitrary natural number n, the Bayes generalization error with β = 1 and its mean satisfy the following equations.  0  − Fn0 , Bg = EXn+1 Fn+1  0  − E[Fn0 ]. E[Bg ] = E Fn+1 Proof of Theorem 1.2 From eq.(1.12) with β = 1, p(Xn+1 |Dn ) =

Zn+1 Zn

holds for an arbitrary natural number n. The logarithm of this equation results in − log p(Xn+1 |Dn ) = Fn+1 − Fn . Also,


p(Xn+1 |Dn ) = q(Xn+1 )

dw ϕ(w) exp(−(n + 1)Kn+1 (w))
dw ϕ(w) exp(−nKn (w))

shows Z0 p(Xn+1 |Dn ) . = n+1 q(Xn+1 ) Zn0 Therefore, log

q(Xn+1 ) 0 − Fn0 . = Fn+1 p(Xn+1 |Dn )

(1.13)

Based on eq.(1.13), the two equations in the theorem are respectively given by
the expectations of Xn+1 and Dn+1 . This theorem shows that the Bayes generalization error with β = 1 is equal to the increase of the normalized stochastic complexity.

22

Introduction

1.3.2 Bayes and Gibbs estimations Let Ew [ · ] be the expectation value using the a posteriori distribution p(w|Dn ). In Bayes estimation, the true distribution is estimated by the predictive distribution Ew [p(x|w)]. In the other method of statistical estimation, Gibbs estimation a parameter w is randomly chosen from p(w|Dn ), then the true distribution is estimated by p(x|w). Gibbs estimation depends on a random choice of the parameter w. Hence, to study its generalization error, we need the expectation value over random choices of w. Bayes and Gibbs estimations respectively have generalization and training errors. The set of four errors is referred to as the Bayes quartet. Definition 1.8 (Bayes quartet) For the generalized a posteriori distribution p(w|Dn ), the four errors are defined as follows. (1) The Bayes generalization error, 

q(X) Bg = EX log , Ew [p(X|w)] is the Kullback–Leibler distance from q(x) to the predictive distribution. (2) The Bayes training error, 1 q(Xi ) log , n i=1 Ew [p(Xi |w)] n

Bt =

is the empirical Kullback–Leibler distance from q(x) to the predictive distribution. (3) The Gibbs generalization error,



q(X)  Gg = Ew EX log , p(X|w) is the mean Kullback–Leibler distance from q(x) to p(x|w). (4) The Gibbs training error, Gt = Ew

n

1 

n

i=1

log

q(Xi )  , p(Xi |w)

is the mean empirical Kullback–Leibler distance from q(x) to p(x|w). Remark 1.11 The Bayes a posteriori distribution p(w|Dn ) depends on the set of random samples, Dn . Hence the Bayes quartet is a set of random variables. The most important variable among them is the Bayes generalization error because it is used in practical applications; however, the other variables have important information about statistical estimation. In fact, we prove that there are mathematical relations among them.

1.3 Statistical estimation methods

23

Theorem 1.3 (Representation of Bayes quartet) By using the log density ratio function f (x, w) = log(q(x)/p(x|w)), the four errors are rewritten as 

Bg = EX − log Ew [e−f (X,w) ] , 1 − log Ew [e−f (Xi ,w) ], n i=1 n

Bt =

Gg = Ew [K(w)], Gt = Ew [Kn (w)]. Proof of Theorem 1.3 The first and the second equations are derived from log

q(X) = − log Ew [e−f (X,w) ]. Ew [p(X|w)]

The third and the fourth equations are derived from the definitions of the Kullback–Leibler distance K(w) and the empirical one Kn (w).
Remark 1.12 (Generalization errors and square error) Let p(y|x, w) be a conditional probability density of y ∈ RN for a given x ∈ RM defined by  1  1 2 p(y|x, w) = √ exp − |y − h(x, w)| , N 2σ 2 2π σ 2 where h(x, w) is a function from RM × Rd to RN , | · | is the norm of RN , and σ > 0 is a constant. Let us compare generalization errors with the square error. If the true conditional distribution is p(y|x, w0 ), then the log density ratio function is 1 {|y − h(x, w)|2 − |y − h(x, w0 )|2 } f (x, y, w) = 2σ 2 1 = {2(y − h(x, w0 )) · (h(x, w0 ) − h(x, w)) 2σ 2 + |h(x, w0 ) − h(x, w)|2 }.

(1.14)

The Kullback–Leibler distance is 1 K(w) = EX [ |h(X, w0 ) − h(X, w)|2 ]. 2σ 2 The Gibbs generalization error is 1 EX [Ew [ |h(X, w0 ) − h(X, w)|2 ]]. 2σ 2 The Bayes generalization error is Gg =

Bg = EX EY [− log Ew [e−f ]],

(1.15)

24

Introduction

where f = f (X, Y, w). On the other hand, the regression function of the estimated distribution Ew [p(y|x, w)] is equal to



yp(y|x, w)dy y Ew [p(y|x, w)] dy = Ew = Ew [h(x, w)]. Let us define the square error of the estimated and true regression functions by 1 EX [|Ew [h(x, w)] − h(X, w0 )|2 ]. (1.16) 2σ 2 In general, Bg = Eg . However, asymptotically, Bg ∼ = Eg . In fact, on a natural assumption, the a posteriori distribution p(w|Xn ) converges so that f → 0, hence 



f2 + o(f 2 ) Bg = EX EY − log Ew 1 − f + 2  

2 f = EX EY Ew f − + 12 Ew [f ]2 + o(f 2 ) . 2 Eg =

By EX EY Ew (f − f 2 /2) = o(f 2 ) using eq.(1.14), Bg = 12 EX EY [Ew [f (X, Y, w)]2 ] + o(f 2 )]], hence Bg ∼ = Eg .

1.3.3 Maximum likelihood and a posteriori Let q(x), p(x|w), and ϕ(w) be the true distribution, a statistical model, and an a priori probability density function, respectively. The generalized log likelihood function is given by Rn (w) = −

n 

log p(Xi |w) − an log ϕ(w),

i=1

where {an } is a sequence of nonnegative real values. By using a log density ratio function f (x, w) = log(q(x)/p(x|w)), the generalized log likelihood function can be rewritten as Rn (w) = Rn0 (w) + nSn ,

1.3 Statistical estimation methods

25

where Rn0 (w) is given by Rn0 (w)

=

n 

f (Xi , w) − an log ϕ(w).

(1.17)

i=1

Note that, in singular statistical models, sometimes inf Rn (w) = −∞, w

which means that there is no parameter that minimizes Rn (w). If a parameter wˆ that minimizes Rn (w) exists, then a statistical estimation method ˆ Dn → p(x|w) is defined. The generalization error Rg and the training error Rt of this method are respectively defined by
q(x) dx, Rg = q(x) log p(x|w) ˆ n 1 q(Xi ) Rt = log . n i=1 p(Xi |w) ˆ By using K(w) and Kn (w) in equations (1.2) and (1.3) respectively, they can be rewritten as ˆ Rg = K(w), ˆ Rt = Kn (w). Definition 1.9 (Maximum likelihood and maximum a posteriori) (1) If an = 0 for arbitrary n, then wˆ is called the maximum likelihood (or ML) estimator and the statistical estimation method is called the maximum likelihood (or ML) method. (2) If an = 1 for arbitrary n, then wˆ is called the maximum a posteriori estimator (or MAP) and the method is called the maximum a posteriori (or MAP) method. (3) If an is an increasing function of n, then wˆ is the generalized maximum a posteriori estimator and the method is called the generalized maximum a posteriori method. Remark 1.13 (Formal relation between Bayes and ML) (1) If β → ∞, both Bayes and Gibbs estimations formally result in the maximum likelihood estimation. (2) In regular statistical models in which the maximum likelihood estimator has asymptotic normality, the leading terms of the asymptotic generalization

26

Introduction

errors of Bayes, ML, and MAP are equal to each other. However, in singular statistical models, they are quite different. Example 1.7 (Divergence of MLE) Let g(x|a, σ ) be the normal distribution on R1 ,  (x − a)2  1 g(x|a, σ ) = √ exp − . 2σ 2 2π σ 2 Let us study a normal mixture, p(x|a, b, c, σ, ρ) = a g(x|b, σ ) + (1 − a) g(x|c, ρ), where the set of parameters is W = {(a, b, c, σ, ρ) ; 0 ≤ a ≤ 1, |b|, |c| < ∞, σ, ρ > 0}. Then the likelihood function for a given Dn Ln (a, b, c, σ, ρ) =

n 

p(Xi |a, b, c, σ, ρ)

i=1

is an unbounded function, because lim Ln (a, b, X1 , σ, ρ) = ∞.

ρ→0

Therefore the normal mixture p(x|a, b, c, σ, ρ) does not have a maximum likelihood estimator for arbitrary true distribution. To avoid this problem, we should restrict the parameter set or adopt the generalized maximum a posteriori method. In singular statistical models, the maximum likelihood estimator often diverges.

1.4 Four main formulas In this section, we give an outline of singular learning theory. Because singular learning theory is quite different from regular statistical theory, the reader is advised to read this overview of the results of the book in advance. The equations and explanations in this section are intuitively described, because rigorous definitions and proofs are given in subsequent chapters.

1.4.1 Standard form of log likelihood ratio function To evaluate how appropriate the statistical models p(x|w) and ϕ(w) are for a given data set Dn = {X1 , X2 , . . . , Xn }, we have to study the case when the set

1.4 Four main formulas

Bayes a posteriori

27

Maximum likelihood

Fig. 1.5. Maximum likelihood and Bayes a posteriori

of true parameters W0 = {w ∈ W ; q(x) = p(x|w) (∀x)} = {w ∈ W ; K(w) = 0} consists of not one point but a union of several manifolds. If K(w) is a polynomial, then W0 is called an algebraic set; if K(w) is an analytic function, then W0 is called an analytic set. If W0 is not one point, neither the Bayes a posteriori distribution nor the distribution of the maximum likelihood estimators converges to the normal distribution. For example, the left-hand side of Figure 1.5 shows a Bayes a posteriori distribution when the set of true parameters is {(a, b); ab = 0}. The right-hand side shows the probability distribution of the maximum likelihood estimator. We need a method to analyze such a singular distribution. The basic term in statistical learning is the empirical Kullback–Leibler distance, 1 f (Xi , w), n i=1 n

Kn (w) =

which is a function of w ∈ W ⊂ Rd . For w ∈ W \ W0 , a random process ψn (w) =

n  K(w) − f (Xi , w) √ n K(w) i=1

is well-defined. The log likelihood ratio function is rewritten as  nKn (w) = nK(w) − nK(w) ψn (w). This representation has two mathematical problems. (1) (Geometrical problem). In a singular model, W0 is not one point but a real analytic set hence the log likelihood ratio function cannot be treated locally

28

Introduction

even if the number of training samples is sufficiently large. Moreover, since the set of true parameters contains complicated singularities, it is difficult to analyze its behavior even in each local neighborhood of W0 . (2) (Probabilistic problem). When n → ∞, under a natural condition, ψn (w) converges in law to a Gaussian process ψ(w) on the set W \ W0 . However, neither ψn (w) nor ψ(w) is well-defined on the set of true parameters W0 . Therefore it is difficult to analyze such a stochastic process near the set of true parameters. In this book, we propose an algebraic geometrical transform that is powerful enough to overcome these two problems. For a real analytic function K(w), the fundamental theorem in algebraic geometry ensures that there exists a real d-dimensional manifold M and a real analytic map g : M  u → w ∈ W such that, for each coordinate Mα of M, K(g(u)) is a direct product, 2kd 2k2 1 K(g(u)) = u2k 1 u2 · · · ud ,

where k1 , k2 , . . . , kd are nonnegative integers. Morevoer, there exists a function φ(u) > 0 and nonnegative integers h1 , h2 , . . . , hd such that   ϕ(g(u))|g  (u)| = φ(u)uh1 1 uh2 2 · · · uhd d , where |g  (u)| is Jacobian determinant of w = g(u). Note that k1 , k2 , . . . , kd and h1 , h2 , . . . , hd depend on a local coordinate. By using the notation u = (u1 , u2 , . . . , ud ), k = (k1 , k2 , . . . , kd ), h = (h1 , h2 , . . . , hd ), the function K(g(u)) and the a priori distribution ϕ(g(u))|g  (u)| are respectively expressed as K(g(u)) = u2k , ϕ(g(u))|g  (u)| = φ(u)|uh |. The theorem that ensures the existence of such a real analytic manifold M and a real analytic map w = g(u) is called Hironaka’s theorem or resolution of singularities. The function w = g(u) is called a resolution map. In Chapters 2 and 3, we give a rigorous statement of the theorem and a method to find the set (M, g), respectively. Then by using K(g(u)) = 0 =⇒ f (x, g(u)) = 0,

1.4 Four main formulas

29

we can prove that there exists a real analytic function a(x, u) such that f (x, g(u)) = a(x, u) uk (∀x). From the definition of the Kullback–Leibler distance,
f (x, g(u))q(x)dx = K(g(u)) = u2k . It follows that


a(x, u)q(x)dx = uk .

Moreover, by f (x, g(u)) = log(q(x)/p(x|g(u))),
K(g(u)) = (f (x, g(u)) + e−f (x,g(u)) − 1)q(x)dx. It is easy to show t + e−t − 1 → 12 . t→0 t2 lim

Therefore, if u2k = 0, then
2K(g(u)) a(x, u)2 q(x)dx = lim = 2. 2k u2k u →0 Here we can introduce a well-defined stochastic process on M, 1  k {u − a(Xi , u)}, ξn (u) = √ n i=1 n

from which we obtain a representation, nKn (g(u)) = nu2k −

√ k nu ξn (u).

(1.18)

By definition, ξn (u) satisfies E[ξn (u)] = 0 (∀u ∈ M), E[ξn (u)ξn (v)] = EX [a(X, u)a(X, v)] − uk v k (∀u, v ∈ M). If K(g(u)) = K(g(v)) = 0, then E[ξn (u)ξn (v)] = EX [a(X, u)a(X, v)], and E[ξn (u)2 ] = 2. By the central limit theorem, for each u ∈ M, ξn (u) converges in law to a Gaussian distribution with mean zero and variance 2. In Chapter 5, we prove the convergence in law ξn → ξ as a random variable on the space of bounded and continuous functions on M. Then the Gaussian

30

Introduction

process ξ (u) is uniquely determined by its mean and covariance. Here we attain the first main formula. Main Formula I (Standard form of log likelihood ratio function) Under natural conditions, for an arbitrary singular statistical model, there exist a real analytic manifold M and a real analytic map g : M → W such that the log likelihood ratio function is represented by 1 Kn (g(u)) = u2k − √ uk ξn (u), n

(1.19)

where ξn (u) converges in law to the Gaussian process ξ (u). Also w = g(u) gives the relation ϕ(g(u))|g  (u)| = φ(u)|uh |,

(1.20)

where φ(u) > 0 is a positive real analytic function. Remark 1.14 (1) Note that the log likelihood ratio function of any singular statistical model can be changed to the standard form by algebraic geometrical transform, which allows |g  (u)| = 0. (2) The integration over the manifold M can be written as the finite sum of the integrations over local coordinates. There exists a set of functions {σα (u)} such  that σα (u) ≥ 0, α σα (u) = 1, and the support of σα (u) is contained in Mα . By using a function φ ∗ (u) = φ(u)σα (u) ≥ 0, where dependence of α in φ ∗ is omitted, for an arbitrary integrable function F (w),

F (w)ϕ(w)dw = F (g(u))ϕ(g(u))|g  (u)|du M

W

=




F (g(u))φ ∗ (u)|uh |du.

(1.21)

α

(3) In regular statistical models, the set of true parameters consists of one point, W0 = {w0 }. By the transform w = g0 (u) = w0 + I (w0 )1/2 u K(g0 (u)) ∼ = 12 |u|2 , Kn (g0 (u)) ∼ = 12 |u|2 −

ξn √ n

· u,

where I (w0 ) is the Fisher information matrix and ξn = (ξn (1), ξn (2), . . . , ξn (d)) is defined by n  1  ∂  log p(Xi |g0 (u)) . ξn (k) = √ u=0 n i=1 ∂uk

1.4 Four main formulas

31

Here each ξn (k) converges in law to the standard normal distribution. This property is called asymptotic normality. If a statistical model has asymptotic normality, Bayes generalization and training errors, MAP, and ML estimations are obtained by using the normal distribution. However, singular statistical models do not have asymptotic normality. The standard form of the log likelihood ratio function, eq.(1.19), is the universal base for singular statistical models.

1.4.2 Evidence of singular model In singular learning theory, the zeta function of a statistical model plays an important role. Definition 1.10 (Zeta function of a statistical model) For a given set (p, q, ϕ), where p(x|w) is a statistical model, q(x) is a true probability distribution, and ϕ(w) is an a priori probability density function with compact support, the zeta function ζ (z) (z ∈ C) of a statistical model is defined by
ζ (z) = K(w)z ϕ(w) dw, where K(w) is the Kullback–Leibler distance from q(x) to p(x|w). By the definition, the zeta function is holomorphic in Re(z) > 0. It can be rewritten by using resolution map w = g(u),
ζ (z) = K(g(u))z ϕ(g(u)) |g  (u)| du. M

By using K(g(u)) = u

2k

and eq.(1.20),(1.21), in each local coordinate, 
u2kz+h φ ∗ (u)du. ζ (z) = α

Mα

It is easy to show that
0

b

1 z+h1 u2k du1 = 1

b2k1 z+h1 . 2k1 z + h1 + 1

Therefore, the Taylor expansion of φ ∗ (u) around the origin in arbitrary order shows that ζ (z) (Re(z) > 0) can be analytically continued to the meromorphic function on the entire complex plane C, whose poles are all real, negative, and rational numbers. They are ordered from the larger to the smaller, 0 > −λ1 > −λ2 > −λ3 > · · · .

32

Introduction

The largest pole (−λ1 ) is determined by h + 1 j . 1≤j ≤d 2kj

λ1 = min min α

(1.22)

Let mk be the order of the pole (−λk ). The order m1 is the maximum number of the elements of the set {j } that attain the minimum of eq.(1.22). Therefore, the zeta function has the Laurent expansion, ζ (z) = ζ0 (z) +

mk ∞  

ckm , (z + λk )mk k=1 m=1

(1.23)

where ζ0 (z) is a holomorphic function and ckm is a coefficient. Let the state density function of t > 0 be
v(t) = δ(t − K(w)) ϕ(w) dw 


=

δ(t − u2k )|uh |φ ∗ (u)du.

α

The zeta function is equal to its Mellin transform,
∞ ζ (z) = v(t) t z dt. 0

Conversely, v(t) is uniquely determined as the inverse Mellin transform of ζ (z). The inverse Mellin transform of F (z) =

(m − 1)! (z + λ)m

is equal to  f (t) =

t λ−1 (− log t)m−1 0

(0 < t < 1) . otherwise

By eq.(1.23), we obtain the asymptotic expansion of v(t) for t → 0, v(t) =

mk ∞  

 ckm t λk −1 (− log t)m−1 .

k=1 m=1  This expansion holds for arbitrary φ ∗ (u), and ckm is a linear transform of φ ∗ (u), therefore there exists a set of Schwartz distributions {Dkm (u)} whose supports are contained in M0 = g −1 (W0 ) such that the asymptotic expansion

δ(t − u2k )uh φ ∗ (u) =

mk ∞   k=1 m=1

Dkm (u) t λk −1 (− log t)m−1

1.4 Four main formulas

33

holds for t → 0. Let Yn (w)dw be a measure defined by Yn (w)dw ≡ exp(−nβKn (w)) ϕ(w) dw, then we have an asymptotic expansion, Yn (w)dw = Yn (g(u)) |g  (u)| du  √ 2k k e−nβu + nβu ξn (u) φ ∗ (u)|uh |du = α

=




∞

dt δ(t − u2k )

0

α

√

× φ ∗ (u)|uh |e−nβt+ =

∞ m k −1   α

ntβ ξn (u)

du

Dkm (u)du

k=1 r=0




∞

× 0

dt  t λk −1  n r −βt+√tβ ξn (u) e . log n n t

For simplicity we use the notation λ = λ1 , m = m1 and  du∗ = D1 m1 (u)du,

(1.24)

α∗

 where α∗ shows the sum of local coordinates that attain the minimum λ and the maximum m in eq.(1.22). Such local coordinates are called essential coordinates in this book. By using the convergence in law ξn (u) → ξ (u), the largest term of the asymptotic expansion of the a posteriori distribution is given by (log n)m−1 Yn (w) dw ∼ du∗ = nλ




∞

√

dt t λ−1 e−βt+

tβ ξ (u)

.

(1.25)

0

The normalized evidence is
Zn0 =

Yn (w) dw.

It follows that Fn0 = − log Zn0 ∼ = λ log n − (m − 1) log log n + F R (ξ ),

(1.26)

34

Introduction

where F R (ξ ) is a random variable

du∗ F R (ξ ) = − log

∞

√

dt t λ−1 e−βt+

tβ ξ (u)

 .

0

We obtain the second main result. Main Formula II (Convergence of stochastic complexity) Let (−λ) and m be respectively the largest pole and its order of the zeta function
ζ (z) = K(w)z ϕ(w)dw of a statistical model. The normalized stochastic complexity has the following asymptotic expansion, Fn0 = λ log n − (m − 1) log log n + F R (ξ ) + op (1), where F R (ξ ) is a random variable and op (1) is a random variable which satisfies the convergence in probability op (1) → 0. Therefore the stochastic complexity Fn has the asymptotic expansion Fn = nβSn + λ log n − (m − 1) log log n + F R (ξ ) + op (1), where Sn is the empirical entropy defined by eq.(1.4). Remark 1.15 If a model is regular then K(w) is equivalent to |w|2 , hence λ = d/2 and m = 1 where d is the dimension of the parameter space. The asymptotic expansion of Fn with β = 1 in a regular statistical model is well known as the Bayes information criterion (BIC) or the minimum description length (MDL). Hence Main Formula II contains BIC and MDL as a special case. If a model is singular, then λ = d/2 in general. In Chapter 3, we give a method to calculate λ and m, and in Chapter 7, we show examples in several statistical models. The constant λ is an important birational invariant, which is equal to the real log canonical threshold if ϕ(w) > 0 at singularities. Therefore Main Formula II claims that the stochastic complexity is asymptotically determined by the algebraic geometrical birational invariant.

1.4.3 Bayes and Gibbs theory In real-world problems, the true distribution is unknown in general. The third formula is useful because it holds independently of the true distribution q(x).

1.4 Four main formulas

35

The expectation value of an arbitrary function F (w) over the a posteriori distribution is defined by  F (w)Yn (w)dw Ew [F (w)] =  , Yn (w)dw where Yn (w) = exp(−nKn (w))ϕ(w). When the number of training samples goes to infinity, this distribution concentrates on the union of neighborhoods of K(w) = 0. In such neighborhoods, the renormalized a posteriori distribution Eu,t [ ] is defined for an arbitrary function A(u, t),

∞ √ du∗ dt A(u, t) t λ−1 e−βt+β t ξ (u)
0
∞ Eu,t [A(u, t)] = , √ ∗ λ−1 −βt+β t ξ (u) du dt t e 0

where du∗ is defined in eq.(1.24). Then eq.(1.25) shows the convergence in law √ √ Ew [( nf (x, w))s ] → Eu,t [( t a(x, u))s ] for s > 0, where the relations of the paramaters are w = g(u), t = nK(w) = nu2k , f (x, w) = a(x, u)uk . Based on these properties, we can derive the asymptotic behavior of the Bayes quartet from Theorem 1.3. Firstly, Gibbs generalization error is Gg = Ew [K(w)] =

1 Eu,t [t]. n

Secondly, Gibbs training error is 1 1 1 Ew [f (Xi , w)] = Eu,t [ξ (u)t 1/2 ] + op , n i=1 n n n

Gt =

where op (1/n) is a random variable which satisfies the convergence in probability, n op (1/n) → 0. Thirdly, the Bayes generalization error is Bg = EX [− log Ew [1 − f (X, w) + 12 f (X, w)2 ]] + op = EX [− log(1 − Ew [f (X, w)] +

1

1 E [f (X, w)2 ])] 2 w

n + op (1/n)

36

Introduction

Then by using − log(1 − ) =  +  2 /2 + o( 2 ) and 1 Ew [f (X, w)] = √ Eu,t [a(X, u)t 1/2 ], n 1 EX [Ew [f (X, w)]] = Ew [K(w)] = Eu,t [t], n EX [Ew [f (X, w)2 ]] = EX [Eu,t [a(X, u)2 t]] =

2 Eu,t [t] + op (1/n), n

where we used EX [a(X, u)2 ] = 2, it follows that Bg =

1 EX [Eu,t [a(X, u)t 1/2 ]2 ] + op (1/n). 2n

(1.27)

And, lastly, the Bayes training error is 1 1 [− log Ew [1 − f (Xi , w) + 12 f (Xi , w)2 ]] + op n i=1 n n

Bt = =

1 1 [− log(1 − Ew [f (Xi , w)] + 12 Ew [f (Xi , w)2 ])] + op n i=1 n

=

n 1  Eu,t [a(Xi , u)t 1/2 ] − 12 Eu,t [a(Xi , u)2 t] n2 i=1

n

1  + 12 Eu,t [a(Xi , u)t 1/2 ]2 + op n = Gt − Gg + Bg + op (1/n), where we used the law of large numbers

1 a(Xi , u)a(Xi , v) = EX [a(X, u)a(X, v)] + op (1) n i=1 n

in the last equation. By using convergence in law ξn (u) → ξ (u), we prove the convergences in law of the Bayes quartet, nBg → Bg∗ ,

nBt → Bt∗ ,

nGg → G∗g ,

nGt → G∗t ,

1.4 Four main formulas

37

where Bg∗ , Bt∗ , G∗g , G∗t are random variables represented by the random process ξ (u). Let us introduce the notation (a ∈ R),
∞ √ dt t λ−1 e−βt+aβ t , Sλ (a) =


0

du∗ Sλ (ξ (u)).

Z(ξ ) = Then Sλ (a) = β




Sλ (a) = β 2

∞

√

dt t λ−1/2 e−βt+aβ t ,

0




∞

√

dt t λ e−βt+aβ t .

0

Finally we obtain E[Bg∗ ]

   ∗  du a(X, u)Sλ (ξ (u)) 2 1 = E EX , 2β 2 Z(ξ )

E[Bt∗ ] = E[Bg∗ ] + E[G∗t ] − E[G∗g ],   ∗   du Sλ (ξ (u)) 1 ∗ , E[Gg ] = 2 E β Z(ξ )   ∗    ∗  du Sλ (ξ (u)) du ξ (u)Sλ (ξ (u)) 1 1 ∗ E[Gt ] = 2 E − E . β Z(ξ ) β Z(ξ ) These equations show that the expectations of the Bayes quartet are represented by linear sums of three expectation values over the random process ξ (u). On the other hand, ξ (u) is a Gaussian process which is represented by ξ (u) =

∞ 

bk (u)gk ,

i=1

where {gk } is a set of random variables that are independently subject to the standard normal distribution and bk (u) = E[ξ (u)gk ]. By using the partial integration E[gk F (gk )] = E[(∂/∂gk )F (gk )] for an arbitrary integrable function F ( ), we can prove   1 du∗ Sλ (ξ (u))  1 du∗ ξ (u) Sλ (ξ (u))  ∗ − 2E . E[Bg ] = 2 E β Z(ξ ) 2β Z(ξ ) Therefore four errors are given by the linear sums of two expectations of Sλ (ξ (u)) and Sλ (ξ (u)). By eliminating two expectations from four equations, we obtain two equations which hold for the Bayes quartet.

38

Introduction

Main Formula III (Equations of states in statistical estimation) There are two universal relations in Bayes quartet. E[Bg∗ ] − E[Bt∗ ] = 2β(E[G∗t ] − E[Bt∗ ]),

(1.28)

E[G∗g ] − E[G∗t ] = 2β(E[G∗t ] − E[Bt∗ ]).

(1.29)

These equations hold for an arbitrary true distribution, an arbitrary statistical model, an arbitrary a priori distribution, and arbitrary singularities. Remark 1.16 (1) Main Formula III holds in both regular and singular models. Although the four errors themselves strongly depend on q(x), p(x|w), and ϕ(w), these two equations do not. By this formula, we can estimate the Bayes and Gibbs generalization errors from the Bayes and Gibbs training errors without any knowledge of the true distributions. The constant ν(β) = β(E[G∗t ] − E[Bt∗ ])

(1.30)

is the important birational invariant called a singular fluctuation. Then Main Formula III claims that E[Bg∗ ] = E[Bt∗ ] + 2ν(β),

(1.31)

E[G∗g ] = E[G∗t ] + 2ν(β).

(1.32)

We can estimate ν(β) from samples. In fact, by defining two random variables, n  (log Ew [p(Xi |w)] − Ew [log p(Xi |w)]), V0 = i=1 n  V = (Ew [(log p(Xi |w))2 ] − Ew [log p(Xi |w)]2 ), i=1

we have V0 = nGt − nBt and E[V /2] is asymptotically equal to E[nGt ] − E[nBt ], ν(β) = βE[V0 ] + o(1) = (β/2)E[V ] + o(1). (2) If a model is regular then, for any β > 0, ν(β) = d/2,

(1.33)

where d is the dimension of the parameter space. If a model is regular, both Bayes and Gibbs estimation converge to the maximum likelihood estimation, when β → ∞. Then two equations of states result in one equation, E[nRg ] = E[nRt ] + d,

(1.34)

1.4 Four main formulas

39

where Rg and Rt are the generalization and training errors of the maximum likelihood estimator. The equation (1.34) is well known as the Akaike information criterion (AIC) of a regular statistical model, hence Main Formula III contains AIC as a very special case. In singular learning machines, eq.(1.33) does not hold in general, hence AIC cannot be applied. Moreover, Main Formula III holds even if the true distribution is not contained in the model [120].

1.4.4 ML and MAP theory The last formula concerns the maximum likelihood or a posteriori method. Let W be a compact set, and f (x, w) and ϕ(w) be respectively analytic and C 2 -class functions of w ∈ W . Then there exists a parameter wˆ ∈ W that minimizes the generalized log likelihood ratio function, Rn0 (w) =

n 

f (Xi , w) − an log ϕ(w),

i=1

where an is a nondecreasing sequence. Note that, if W is not compact, the parameter that minimizes Rn0 (w) does not exist in general. By applying the standard form of the log likelihood ratio function and a simple notation σ (u) = − log ϕ(g(u)), in each coordinate, the function Rn0 (g(u)) is represented by 1 an 1 0 Rn (g(u)) = u2k − √ uk ξn (u) + σ (u), n n n where ξn (u) → ξ (u) in law. For an arbitrary u, a new parameterization (t, v) is defined by t = uk , v = Proj(u), where the function Proj( ) maps u to v on the set {v; v 2k = 0} along the ordinary differential equation u(T ) for T ≥ 0, d u(T ) = −∇(u(T )2k ). dT

(1.35)

Here v = Proj(u) is determined by v = u(T = ∞) for the initial condition u = u(T = 0). More precisely, see Chapter 6 and Figure 6.3. In each local coordinate, 1 an 1 0 Rn (g(t, v)) = t 2 − √ t ξn (t, v) + σ (t, v). n n n

40

Introduction

Then we can prove that, for arbitrary C 1 -class function f (u) on a compact set, there exist constants C, δ > 0 such that |f (t, v) − f (0, v)| ≤ t δ ∇f , where

 ∂f    ∇f  ≡ sup sup . u ∂uj j

Let tˆ be the parameter that minimizes Rn0 (t, v), then tˆ should be in proportion √ to 1/ n, hence 1 an 1 1 0 Rn (g(tˆ, v)) = tˆ2 − √ tˆ ξ (0, v) + σ (0, v) + op . n n n n We can prove that op (1/n) does not affect the main terms. Let vˆ be the parameter that minimizes Rn0 (g(tˆ, v)). Then 1 tˆ = √ max max{0, ξ (0, v)}, ˆ 2 n α where α shows the local coordinate. If an ≡ 0, then vˆ is determined by minimizing ˆ 2. − max max{0, ξ (v)} α

Hence the generalization and training errors are given by  1 Rg = max {0, ξ (u)}2 , 4n u∈M0  1 Rt = − max {0, ξ (u)}2 , 4n u∈M0 where M0 = g −1 (W0 ) is the set of true parameters. The symmetry of generalization and training errors holds if an /np → ∞ for arbitrary p > 0. Therefore, E[nRg ] = −E[nRt ] + o(1). For the other sequence an , the same result is obtained. Main Formula IV (Symmetry of generalization and training errors) If the maximum likelihood or generalized maximum a posteriori method is applied, the symmetry of generalization and training errors holds, lim E[nRg ] = − lim E[nRt ].

n→∞

n→∞

1.5 Overview of this book

41

Remark 1.17 (1) In regular statistical models, E[nRg ] = d/2 where d is the dimension of the parameter space. In singular statistical models E[nRg ] >> d/2 in general, because it is the mean of the maximum value of a Gaussian process. If the parameter space is not compact, then the maximum likelihood estimator sometimes does not exist. Even if it exists, it often diverges for n → ∞, which means that E[nRt ] → −∞. In such a case, the behavior of the generalization error is still unknown. It is expected that the symmetry still holds, in which case E[nRg ] → +∞. Hence the maximum likelihood method is not appropriate for singular statistical models. Even if the set of parameters is compact, it is still difficult to estimate the generalization error from the training error without knowledge of the true distribution. From a statistical point of view, the maximum likelihood estimator is asymptotically the sufficient statistic in regular models. However, it is not in singular models, because the likelihood function does not converge to the normal distribution. (2) In singular statistical models, two limiting procedures n → ∞ and β → ∞ are not commutative in general. In other words, lim lim E[nBg ] = lim E[nRg ].

β→∞ n→∞

n→∞

In fact, the Bayes generalization error is determined by the sum of the essential  local coordinates α∗ , whereas the maximum likelihood generalization error  is determined by the set of all coordinates α .

1.5 Overview of this book The main purpose of this book is to establish the mathematical foundation on which the four main formulas are proved. In Chapter 2, we introduce singularity theory and explain the resolution theorem which claims that, for an arbitrary analytic function K(w), there exist a manifold and an analytic function w = g(u) such that K(g(u)) = u2k . In Chapter 3, elemental algebraic geometry is explained. The relation between algebra and geometry, Hilbert’s basis theorem, projective space, and blow-ups are defined and illustrated. We show how to find the resolution map using recursive blow-ups for a given statistical model.

42

Introduction

In Chapter 4, the mathematical relation between the zeta function and the singular integral is clarified. We need Schwartz distribution theory to connect these two concepts. Several inequalities which are used in the following sections are proved. In Chapter 5, we study the convergence in law of the empirical process to a Gaussian process, ξn (u) → ξ (u). This is the central limit theorem on the functional space. Also we introduce the partial integral on the function space. Based on mathematical foundations in chapters 2, 3, 4, and 5, the four main formulas are rigorously proved in Chapter 6. These are generalizations of the conventional statistical theory of regular models to singular models. We find two birational invariants, the maximum pole of the zeta function and the singular fluctuation, which determine the statistical learning process. Chapters 7 and 8 are devoted to applications of this book to statistics and information science.

1.6 Probability theory In this section, fundamental points of probability theory are summarized. Readers who are familiar with probability theory can skip this section. Definition 1.11 (Metric space) Let
be a set. A function D D :
×
 (x, y) → D(x, y) ∈ R is called a metric if it satisfies the following three conditions. (1) For arbitrary x, y ∈
, D(x, y) = D(y, x) ≥ 0. (2) D(x, y) = 0 if and only if x = y. (3) For arbitrary x, y, z ∈
, D(x, y) + D(y, z) ≥ D(x, z). A set
with a metric is called a metric space. The set of open neighborhoods of a point x ∈
is defined by {U (x);  > 0} where U (x) = {y ∈
; D(x, y) < }. The topology of the metric space is determined by all open neighborhoods. A metric space
is called separable if there exists a countable and dense subset. A set {xn ; n = 1, 2, 3, . . .} is said to be a Cauchy sequence if, for arbitrary δ > 0, there exists M such that m, n > M =⇒ D(xm , xn ) < δ. If any Cauchy sequence in a metric space
converges in
, then
is called a complete metric space. A complete and separable metric space is called a Polish space.

1.6 Probability theory

43

Example 1.8 In this book, we need the following metric spaces. (1) The finite-dimensional real Euclidean space Rd is a metric space with the metric D(x, y) = |x − y| ≡

d  1/2 (xi − yi )2 , i=1

where x = (xi ), y = (yi ), and | · | is a norm of Rd . The real Euclidean space Rd is a complete and separable metric space. (2) A subset of Rd is a metric space with the same metric. Sometimes a finite or countable subset in Rd is studied. (3) Let K be a compact subset in Rd . The set of all continuous function from  K to Rd 


= {f ; f : K → Rd } is a metric space with the metric D(f, g) = f − g ≡ max |f (x) − g(x)|, x∈K



where | · | is the norm of Rd . By the compactness of K in Rd , it is proved that
is a complete and separable metric space. Definition 1.12 (Probability space) Let
be a metric space. A set B composed of subsets contained in
is called a sigma algebra or a completely additive set if it satisfies the following conditions. (B contains the empty set.) (1) If A1 , A2 ∈ B then A1 ∩ A2 ∈ B. (2) If A ∈ B then Ac ∈ B (Ac is the complementary set of A). (3) If A1 , A2 , A3 . . . , ∈ B then the countable union ∪∞ k=1 Ak ∈ B. The smallest sigma algebra that contains all open sets of
is said to be a Borel field. A pair of a metric space and a sigma algebra (
, B) is called a measurable space. A function P , P : B  A → 0 ≤ P (A) ≤ 1, is called a probability measure if it satisfies (1) P (
) = 1. (2) For {Bk } which satisfies Bk ∩ Bk = ∅ (k = k  ), P (∪∞ k=1 Bk ) =

∞ 

P (Bk ).

k=1

A triple of a metric space, a sigma algebra, and a probability measure (
, B, P ) is called a probability space.

44

Introduction

Remark 1.18 Let (RN , B, P ) be a probability space, where RN is the N-dimensional real Euclidean space, B the Borel field, and P a probability distribution. If P is defined by a function p(x) ≥ 0,
p(x)dx (A ∈ B), P (A) = A

then p(x) is called a probability density function. Definition 1.13 (Random variable) Let (
, B, P ) be a probability space and (
1 , B1 ) a measurable space. A function X :
 ω → X(ω) ∈
1 is said to be measurable if X−1 (B1 ) ∈ B for arbitrary B1 ∈ B1 . A measurable function X on a probability space is called a random variable. Sometimes X is said to be an
1 -valued random variable. By the definition µ(B1 ) = P (X−1 (B1 )),

(1.36)

µ is a probability measure on (
1 , B1 ), hence (
1 , B1 , µ) is a probability space. The probability measure µ is called a probability distribution of the random variable X. Then X is said to be subject to µ. Note that µ is the probability distribution on the image space of a function of X. Equation (1.36) can be rewritten as

µ(dx) = P (da). B1

X−1 (B1 )

Remark 1.19 (1) In probability theory, the simplified notation P (f (X) > 0) ≡ P ({ω ∈
; f (X(ω)) > 0}) is often used. Then by definition, P (f (X) > 0) = µ({x ∈
1 ; f (x) > 0}). (2) The probability measure µ to which a random variable X is subject is often denoted by PX . The map X → PX is not one-to-one in general. For example, on a probability space (
, 2
, P ) where
= {1, 2, 3, 4} and P ({i}) = 1/4 (i = 0, 1, 2, 3), two different random variables  0 (i = 0, 1) X(i) = 1 (i = 2, 3)  0 (i = 0, 2) Y (i) = 1 (i = 1, 3) are subject to the same probability distribution. Therefore, in general, even if X and Y are subject to the same probability distribution, we cannot predict the realization of Y from a realization of X.

1.6 Probability theory

45

(3) In descriptions of definitions and theorems, sometimes we need only the information of the image space of a random variable X and the probability distribution PX . In other words, there are some definitions and theorems in which the explicit statement of the probability space (
, B, P ) is not needed. In such cases, the explicit definition of the probability space is omitted, resulting in a statement such as “for
1 -valued random variable X which is subject to a probability distribution PX satisfies the following equality . . .” Definition 1.14 (Expectation) Let X be a random variable from the probability space (
, B, P ) to (
1 , B1 ) which is subject to the probability distribution PX . If the integration

E[X] = X(ω)P (dω) = x PX (dx) is well defined and finite in
1 , E[X] ∈
1 is called the expectation or the mean of X. Let S be a subset of
1 . The partial expectation is defined by

E[X]S = X(ω)P (dω) = x PX (dx). X(ω)∈S

S

Remark 1.20 These are fundamental remarks. (1) Let (
1 , B1 ) and X be same as Definition 1.14 and (
2 , B2 ) be a measurable space. If f :
1 →
2 is a measurable function then f (X) is a random variable on (
, B, P ). The expectation of f (X) is equal to

E[f (X)] = f (X(ω))P (dω) = f (x) PX (dx). This expectation is often denoted by EX [f (X)]. (2) Two random variables which have the same probability distribution have the same expectation value. Hence if X and Y have the same probability distribution, we can predict E[Y ] based on the information of E[X]. (3) In statistical learning theory, it is important to predict the expectation value of the generalization error from the training error. (4) If E[|X|] = C then, for arbitrary M > 0, C = E[|X|] ≥ E[|X|]{|X|>M} ≥ ME[1]{|X|>M} = MP (|X| > M). Hence C , M which is well known as Chebyshev’s inequality. The same derivation is often effective in probability theory. P (|X| > M) ≤

46

Introduction

(5) The following conditions are equivalent. E[|X|] < ∞ ⇐⇒ lim E[|X|]{|X|≥M} = 0. M→∞

(6) If there exist constants δ > 0 and M0 > 0 such that for an arbitrary M > M0 P (|X| ≥ M) ≤

1 , M 1+δ

then E[|X|] < ∞. Definition 1.15 (Convergence of random variables) Let {Xn } and X be a sequence of random variables and a random variable on a probability space (
, B, P ), respectively. (1) It is said that Xn converges to X almost surely (almost everywhere), if   P {ω ∈
; lim Xn (ω) = X(ω)} = 1. n→∞

(2) It is said that Xn converges to X in the mean of order p > 0, if lim E[(Xn − X)p ] = 0.

n→∞

(3) It is said that Xn converges to X in probability, if lim P (D(Xn , X) > ) = 0

n→∞

for arbitrary  > 0, where D(·, ·) is the metric of the image space of X. Remark 1.21 There are well-known properties of random variables. (1) If Xn converges to X almost surely or in the mean of order p > 0, then it does in probability. (2) If Xn converges to X in probability, then it does in law. For the definition of convergence in law, see chapter 5. (3) In general, “almost surely” is neither sufficient nor necessary condition of “in the means of order p > 0.” Remark 1.22 (Limit theorem) From the viewpoint of probability theory, in this book we obtain the limit theorem of the random variables,
Fn = − log p(X1 |w)β p(X2 |w)β · · · p(Xn |w)β ϕ(w)dw, and

 Bg = EX



 p(X|w)p(X1 |w)β · · · p(Xn |w)β ϕ(w)dw  − log , q(X)p(X1 |w)β · · · p(Xn |w)β ϕ(w)dw

1.6 Probability theory

47

where X1 , . . . , Xn are independently subject to the same distribution as X. As the central limit theorem is characterized by the mean and the variance of the random variables, the statistical learning theory is characterized by the largest pole of the zeta function and the singular fluctuation. The large deviation theory indicates that Fn /n → pS, where S is the entropy of X. Main Formulas II and III show more precise results than the large deviation theory.

2 Singularity theory

A lot of statistical models and learning machines contain singularities in their parameter spaces. Singularities determine the behavior of the learning process, hence it is not until we understand singularities that we obtain statistical learning theory. In this chapter, the definition of singularities and the basic theorem for resolution of singularities are introduced. To explain the resolution of singularities, the definition of a manifold is necessary, which is included in Section 2.6.

2.1 Polynomials and analytic functions Let d be a natural number. Let R and C be the set of all real numbers and the set of all complex numbers respectively. A d-dimensional multi-index α is defined by α = (α1 , α2 , . . . , αd ), where α1 , α2 , . . . , αd are nonnegative integers. For given x, b ∈ Rd x = (x1 , x2 , . . . , xd ), b = (b1 , b2 , . . . , bd ), and aα ∈ R, we define aα (x − b)α = aα1 α2 ···αd (x1 − b1 )α1 (x2 − b2 )α2 · · · (xd − bd )αd . A sum of such terms ∞ ∞   ··· aα1 α2 ···αd (x1 − b1 )α1 · · · (xd − bd )αd f (x) = α1 =0

(2.1)

αd =0

is said to be a power series, which is written by  aα (x − b)α . f (x) = α

48

(2.2)

2.1 Polynomials and analytic functions

49

If the number of the nonzero terms in the sum of eq.(2.2) is finite, then f (x) is called a polynomial. If f (x) is a polynomial, then it uniquely determines a function f : Rd → R. The set of all polynomials with real coefficients is written by R[x1 , x2 , . . . , xd ]. If there exists an open set U ⊂ Rd which contains b such that, for arbitrary x ∈ U,  |aα | |x − b|α < ∞, α

then f (x) is called an absolutely convergent power series. If f (x) is an absolutely convergent power series, then the sum of eq.(2.1) converges indepen   dently of the order of sums α1 , α2 , . . . , αd , and uniquely determines a function f : U → R. This function is called a real analytic function. If f (x) is a real analytic function then the coefficient of the Taylor expansion around b satisfies aα =

1 ∂αf (b), α! ∂x α

where α! =

d 

αi !

i=1

 ∂ αi ∂α = . ∂x α ∂xiαi i=1 d

A real analytic function f (x) has the expansion around any b ∈ U  f (x) = aα (x − b )α , α

which absolutely converges in some open set in U . If f (x) also absolutely converges in U  , the domain of the analytic function f (x) can be extended to U ∪ U  . Such a method of extending the domain of an analytic function is called an analytic continuation. A complex analytic function f (x) of x ∈ U ⊂ Cd is also defined by using aα ∈ C. Absolute convergence and analytic continuation are defined in the same way. Definition 2.1 (Function of class C r ) Let U be an open set in d-dimensional  real Euclidean space Rd . A function f : U → Rd is said to be of class C r in U if partial derivatives ∂ n1 +n2 +···+nd f (x) ∂x n1 ∂x n2 · · · ∂x nd

50

Singularity theory

are well defined and continuous for all nonnegative integers n1 , n2 , . . . , nd such that n1 + n2 + · · · + nd ≤ r. 

If f (x) is a function of class C r for r ≥ 1, then it is of class C r for all 0 ≤ r  ≤ r. If f (x) is a function of class C r for all natural numbers r, it is said to be of class C ∞ . If a function f (x) is real analytic in U , it is said to be of class C ω . Example 2.1 A finite sum f (x, y, z) = x 3 y 5 z2 + xy 6 + z5 + 2 is a polynomial. A power series f (x, y) =

∞  ∞  xmyn m=0 n=0

m!n!

absolutely converges if |x|, |y| < ∞, which defines a real analytic function f (x, y) = ex+y . A function

    exp − 1 2 2 g(x, y) = x y  0

(xy = 0) (xy = 0)

is not a real analytic function but of class C ∞ , by defining that the arbitrary times derivative of g(x, y) at xy = 0 is zero. Although g(x, y) = 0 ⇐⇒ xy = 0 holds, there exists no pair of integers (k1 , k2 ) which satisfies g(x, y) = a(x, y)x k1 y k2 for some function a(x, y) > 0.

2.2 Algebraic set and analytic set Definition 2.2 (Real algebraic set) Let f : Rd → R be a function defined by a polynomial. The set of all points that make f (x) = 0, V(f ) = {x ∈ Rd ; f (x) = 0},

2.2 Algebraic set and analytic set

y

y

y x

O

51

x

O

a=0

a0

Fig. 2.1. Examples of real algebraic sets

z

z

z y

y x

x a0

Fig. 2.2. Example of a real algebraic set

is called a real algebraic set. Let f1 (x), f2 (x), . . . , fk (x) be polynomials. The set of all points that make all functions zero V(f1 , f2 , . . . , fk ) = {x ∈ Rd ; f1 (x) = f2 (x) = · · · = fk (x) = 0} is also called a real algebraic set. Example 2.2 Three real algebraic sets V(y 2 − x 3 − ax 2 ) = {(x, y) ∈ R2 ; y 2 − x 3 − ax 2 = 0} for a < 0, a = 0, and a > 0 are illustrated in Figure 2.1. For different a, they have very different shapes. For the case a < 0, the origin is isolated from other lines. For the case a = 0, the shape of the origin is called a cusp. In all cases, the origin is a singularity, where the singularity is defined in Definition 2.6. Figure 2.2 shows real algebraic sets V(x 2 + y 2 − z2 − a) = {(x, y, z) ∈ R3 ; x 2 + y 2 − z2 − a = 0}

52

Singularity theory

for three different a. Sometimes a real algebraic set consists of one point V(x 2 + y 2 ) = {(x, y) ∈ R2 ; x 2 + y 2 = 0} = {(0, 0)}. Different polynomials may define the same real algebraic set. V((xy + z)2 + z4 ) = V(xy, z). Definition 2.3 (Real analytic set) Let U be an open set in the real Euclidean space Rd and f : U → R be a real analytic function. The set of all zero points {x ∈ U ; f (x) = 0} is called a real analytic set. For a set of given analytic functions f1 (x), f2 (x), . . . , fk (x), the set of common zero points {x ∈ U ; f1 (x) = f2 (x) = · · · = fk (x) = 0} is also called a real analytic set. Example 2.3 These are examples of real analytic sets. {(x, y) ∈ R2 ; cos(x) − sin(y) = 0}, {(x, y, z) ∈ R3 ; exy + eyz + z3 = 0}. Another example is {(x, y, z) ∈ U ; x 2 − y log z = 0} where U = {(x, y, z); x, y ∈ R, z > 0}. A function f : R2 → R1 defined by  xy sin(1/(xy)) f (x, y) = 0

(x = 0) (x = 0)

is not a real analytic function, because it is not represented by any absolutely convergent power series at the origin. Note that the set {(x, y) ∈ R2 ; f (x, y) = 0} is not a real analytic set, because this set contains a sequence (xn , yn ) = (π/n, 1) which converges to (0, 1). However, in U = {(x, y) ∈ R2 ; 0 < x, y < 1}, {(x, y) ∈ U ; f (x, y) = 0} = {(x, y) ∈ U ; xy = 1/(nπ ), n = 1, 2, . . .} is a real analytic set.

2.3 Singularity

53

2.3 Singularity Let U be an open set in Rd and f : U → R1 be a function of C 1 class. The d-dimensional vector ∇f (x) ∈ Rd defined by   ∂f ∂f ∂f (x), (x), . . . , (x) ∇f (x) = ∂x1 ∂x2 ∂xd is said to be the gradient vector of f (x). Definition 2.4 (Critical point of a function) Let U be an open set of Rd , and f : U → R1 be a function of C 1 class. (1) A point x ∗ ∈ U is called a critical point of f if it satisfies ∇f (x ∗ ) = 0. If x ∗ is a critical point of f , then f (x ∗ ) is called a critical value. (2) If there exists an open set U  ⊂ U such that x ∗ ∈ U  and f (x) ≤ f (x ∗ )

(∀x ∈ U  ),

then x ∗ is called a local maximum point of f . If x ∗ is a local maximum point, then f (x ∗ ) is called a local maximum value. (3) If there exists an open set U  ⊂ U such that x ∗ ∈ U  and f (x) ≥ f (x ∗ )

(∀x ∈ U  ),

then x ∗ is called a local minimum point of f . If x ∗ is a local minimum point, then f (x ∗ ) is called a local minimum value. If f is a function of C 1 class, then a local maximum or minimum point is a critical point of f . However, a critical point is not always a local maximum or minimum point. Example 2.4 (1) A function on R2 f (x, y) = x 2 + y 4 + 3 has a unique local minimum point (0, 0). (2) For a function on R3 f (x, y, z) = (x + y + z)4 + 1, all points (x ∗ , y ∗ , z∗ ) which satisfy x ∗ + y ∗ + z∗ = 0 are local minimum points. (3) For a function on R2 f (x, y) = x 2 − y 2 ,

54

Singularity theory

xr+1 ,..., xd U

f

V

P A

f (A)

x1, x2 , ... , xr

Fig. 2.3. Definition of a nonsingular point

there is no local maximum or minimum point. The origin (0, 0) is a critical point of f . If |x| ≥ |y|, then f (x, y) ≥ 0, and if |x| ≤ |y|, then f (x, y) ≤ 0. Such a critical point is said to be a saddle point. Definition 2.5 (C r Isomorphism) Let U , V be open sets of the real Euclidean space Rd . If there exists a one-to-one map f : U → V such that both f and f −1 are functions of C r class, then U is said to be C r isomorphic to V , and f is called a C r isomorphism. If both f and f −1 are analytic functions, then U is said to be analytically isomorphic to V and f is called an analytic isomorphism. Example 2.5 Two open sets U = {(x, y) ; x 2 + y 2 < 1}, 



V = {(x  , y  ) ; x 2 + y 2 + 2y  ex + e2x < 1} are analytically isomorphic because x  = x, y  = y − ex is an analytic isomorphism. Definition 2.6 (Singularities of a set) Let A be a nonempty set contained in the real Euclidean space Rd . (1) A point P in A is said to be nonsingular if there exist open sets U, V ⊂ Rd and an analytic isomorphism f : U → V such that f (A ∩ U ) = {(x1 , x2 , . . . , xr , 0, 0, . . . , 0); xi ∈ R} ∩ V ,

(2.3)

where r is a nonnegative integer (Figure 2.3). If all points of A are nonsingular, then A is called a nonsingular set.

2.3 Singularity

55

(2) If a point P in A is not nonsingular, it is called a singularity or a singular point of the set A. The set of all singularities in A is called the singular locus of A, which is denoted by Sing(A) = {P ∈ A ; P is a singularity of A}. Example 2.6 (1) The set A = {(x, y); y − x 3 = 0} is nonsingular, Sing(A) = ∅. Here the origin (0, 0) is a nonsingular point of A because we can choose U = V = {(x, y); |x| < 1} and x  = x, y = y − x3 is an analytic isomorphism. The origin is not a critical point of the function f (x, y) = y − x 3 . (2) The set A = {(x, y, z); (xy + z)2 = 0} is nonsingular. The origin is a nonsingular point of A because there exist U = V = {(x, y, z); |x| < 1, |y| < 1} and an analytic isomorphism, x  = x, y  = y, z = z − xy. The origin is a critical point of the function f (x, y, z) = (xy + z)2 . (3) In the set A = {(x, y); xy = 0}, the origin is a singularity of A, which is a critical point of the function f (x, y) = xy. The singular locus is Sing(A) = {(0, 0)}. (4) In the set A = {(x, y); y 2 − x 3 = 0}, the origin is a singularity of A, which is a critical point of the function f (x, y) = y 3 − x 2 . (5) In the set A = {(x, y); x 5 − y 3 = 0}, the origin is a singularity of A, which is a critical point of f (x, y) = x 5 − y 3 . The set A has a tangent line y = 0. (6) In the set A = {(x, y, z); xyz = 0}, Sing(A) = {(x, y, z); x = y = 0, or y = z = 0, or z = x = 0}. The set B = {(x, y, z); x = y = 0} is a nonsingular set contained in Sing(A). Such a set is called a nonsingular set contained in the singular locus of A. Remark 2.1 (1) A nonsingular analytic set is a real analytic manifold, because the neighborhood of a nonsingular point is analytically isomorphic to an r-dimensional open set in real Euclidean space, where r is equal to the number in eq.(2.3). (2) At a nonsingular point, we can define a tangent plane. At a singularity, in general a tangent plane cannot be defined.

56

Singularity theory

(3) We can check whether a point P in an algebraic set is a singularity or not by a condition of the Jacobian matrix (see Theorem 2.2). (4) A critical point of a function f may not be a singularity of a real analytic set {x; f (x) = 0}. Let U be an open set in Rd and f : U → Rd , f (x) = (f1 (x), f2 (x), . . . , fd (x)), is a function of class C 1 . The Jacobian matrix at x ∈ U is a d × d matrix defined by  ∂f  ∂f1 1 (x) · · · (x) ∂xd  ∂x1. ..  ..  . J (x) =  (2.4) . . .  . ∂fd ∂fd (x) · · · ∂xd (x) ∂x1 The Jacobian determinant is defined by det J (x). The absolute value of the Jacobian determinant is denoted by |f  (x)| = | det J (x)|. Theorem 2.1 (Inverse function theorem) Let U be an open set in Rd and f : U → Rd be a function of C r class (1 ≤ r ≤ ω). If the Jacobian matrix at x0 ∈ U is invertible, then there exists an open set U  ⊂ U such that f is C r isomorphism of U  and f (U  ). (Explanation of Theorem 2.1) This theorem is the well-known inverse function theorem. See, for example, [95]. Theorem 2.2 (A sufficient condition for a nonsingular point). Let U be an open set in the real Euclidian space Rd , and f1 (x), f2 (x), . . . , fk (x) be a set of analytic functions (1 ≤ k ≤ d). We define a real analytic set by A = {x ∈ U ; f1 (x) = f2 (x) = · · · = fk (x) = 0}. If a point x0 ∈ A satisfies



···  . .. det  .  .. ∂fk (x ) · · · ∂x1 0 ∂f1 (x ) ∂x1 0

∂f1 (x ) ∂xk 0



..   .  = 0, ∂fk (x ) ∂xk 0

(2.5)

then x0 is a nonsingular point of A. Proof of Theorem 2.2 Since k ≤ d, we add (d − k) independent functions, fi (x) = xi

(k < i ≤ d).

2.3 Singularity

57

Then f (x) = (f1 (x), . . . , fd (x)) satisfies the condition of Theorem 2.1. Therefore, there exists an open set V , which contains x0 , such that f : V → f (V ) is an analytic isomorphism. If x = (x1 , x2 , . . . , xd ) ∈ A ∩ V , then f1 (x) = · · · = fk (x0 ) = 0, hence f (x) = (0, 0, . . . , 0, xk+1 , . . . , xd ) ∈ f (V ). By Definition 2.6, x0 is not a singularity.




Remark 2.2 (Implicit function theorem) From the proof of Theorem 2.2, the function f −1 : (0, 0, . . . , 0, xk+1 , . . . , xd ) → (x1 , x2 , . . . , xd ) ∈ A ∩ V can be understood as a function from xˆ = (xk+1 , . . . , xd ) ∈ Rd−k into x = ˆ If a function π : Rd → Rk is (x1 , x2 , . . . , xd ) ∈ Rd , which is denoted by g(x). defined by π(x1 , . . . , xd ) = (x1 , . . . , xk ), ˆ ≡ π(g(x)) ˆ satisfies then ϕ(x) ˆ x) ˆ =0 f1 (ϕ(x), .. . ˆ x) ˆ = 0. fr (ϕ(x), ˆ exists. This That is to say, under the condition of eq.(2.5), such a function ϕ(x) result is called the implicit function theorem. Remark 2.3 (1) Let the generalized Jacobian matrix (k × d) (k ≤ d) be  ∂f  ∂f1 1 (x ) · · · (x ) 0 0 ∂xd  ∂x1 . ..  ..  . (2.6) J (x0 ) =  . . .  . ∂fk ∂fk (x ) · · · ∂xd (x0 ) ∂x1 0 If the rank of this matrix is full, that is to say, rankJ (x0 ) = k, then there exists a set of analytic functions {gi (x); i = 1, 2, . . . , k} made of a linear combination of {fi (x); i = 1, 2, . . . , d} which satisfies the same condition as eq.(2.5) instead of {fk }, hence x0 is a nonsingular point in A. (2) Even if x0 is a nonsingular point, rankJ (x0 ) is not equal to k, in general. However, there is a set of functions f1 (x), . . . , fk (x) which ensures that x0 is nonsingular ⇐⇒ rankJ (x0 ) = k. The condition of such a set of functions is shown in chapter 3.

58

Singularity theory

Corollary 2.1 For a real analytic function f , a singularity of the real analytic set A = {x ∈ U : f (x) = 0} is a critical point of the function f . However, in general, a critical point of the function f may not be a singularity of the set A. Proof of Corollary 2.1 From Theorem 2.2, if x0 is not a critical point of f , then x0 is a nonsingular point. On the other hand, if f (x, y) = (x + y)2 , then the origin is a critical point of f , but a nonsingular point in {(x, y);
x + y = 0}. Remark 2.4 (1) (Sard’s theorem) Let f be a function of C ∞ class from an open set in Rd to Rd . Then the Lebesgue measure of the set of all critical values is zero in Rd . (2) If f is a real analytic function whose domain is restricted in a compact set then the set of all critical values is a finite set (Theorem 2.9).

2.4 Resolution of singularities Let U ⊂ Rd be an open set which contains x0 and f : U → R1 a real analytic function that satisfies f (x0 ) = 0. If x0 is not a critical point of f , in other words, ∇f (x0 ) = 0, then x0 is not a singularity of the real analytic set {x ∈ U ; f (x) = 0}. On the other hand, if ∇f (x0 ) = 0, then the point x0 may be a singularity and the real analytic set may have a very complex shape at x0 . Therefore, if ∇f (x0 ) = 0, it seems to be very difficult to analyze the function y = f (x) in the neighborhood of x0 . However, the following theorem shows that any neighborhood of a real analytic set can be understood as an image of normal crossing singularities. This theorem plays a very important role in this book. Theorem 2.3 (Hironaka’s theorem, resolution of singularities) Let f (x) be a real analytic function from a neighborhood of the origin in the real Euclidean space Rd to R1 , which satisfies f (0) = 0 and f (x) is not a constant function. Then, there exists a triple (W, U, g) where (a) W is an open set in Rd which contains 0, (b) U is a d-dimensional real analytic manifold, (c) g : U → W is a real analytic map, which satisfies the following conditions. (1) The map g is proper, in other words, for any compact set C ⊂ W , g −1 (C) is a compact set in U . (2) We use the notation W0 = {x ∈ W ; f (x) = 0} and U0 = {u ∈ U ; f (g(u)) = 0}. The real analytic function g is a real analytic isomorphism of U \ U0 and

2.4 Resolution of singularities

59

W \ W0 , in other words, it is a one-to-one and onto real analytic function from U \ U0 to W \ W0 . (3) For an arbitrary point P ∈ U0 , there is a local coordinate u = (u1 , u2 , . . . , ud ) of U in which P is the origin and f (g(u)) = S uk11 uk22 · · · ukdd ,

(2.7)

where S = 1 or S = −1 is a constant, k1 , k2 , . . . , kd are nonnegative integers, and the Jacobian determinant of x = g(u) is g  (u) = b(u)uh1 1 uh2 2 · · · uhd d ,

(2.8)

where b(u) = 0 is a real analytic function, and h1 , h2 , . . . , hd are nonnegative integers. Remark 2.5 (1) This fundamental theorem was proved by Hironaka in 1964 [40], for which he received the Fields Medal. For the proof of this theorem, see [40]. Application of this theorem to Schwartz distribution theory and differential equations was pointed out by Atiyah in 1970. In the paper [14], the resolution theorem is introduced as “for an analytic function f (x) there exists an invertible analytic function a(u) such that f (g(u)) = a(u)uk ”. Here “invertible” means not that a −1 (u) exists but that 1/a(u) exists, in other words, the inverse Schwartz distribution 1/a exists. This paper also shows that the resolution theorem of real analytic functions has complexification. A mathematical relation between this theorem and Bernstein–Sato’s b-function was shown by Kashiwara in 1976 [46], where the more direct expression “f (g(u)) = uk ” was applied. It was proposed in 1999 that this theorem is the foundation for statistical learning theory [99, 100]. (2) As is well known in linear algebra, any linear transform L on a finitedimensional vector space can be represented by a matrix of Jordan form by choosing an appropriate coordinate. The resolution of singularities claims that any analytic function can be represented by a normal crossing function by choosing an appropriate manifold. Remark 2.6 These are remarks on Theorem 2.3. (1) Figure 2.4 illustrates this theorem. (2) By using multi-indices, eq.(2.7) and eq.(2.8) are respectively expressed by f (g(u)) = S uk , g  (u) = b(u) uh . (3) The constant S and the multi-indeces k and h depend on local coordinates in general.

60

Singularity theory

R Analytic f(x)

Normal crossing f(g(u)) = S u1k1 u 2k2 ... udkd g

W

Proper analytic

Rd

Manifold U

Fig. 2.4. Resolution of singularities

(4) In this theorem, we used the notation, g −1 (C) = {u ∈ U ; g(u) ∈ C}, and W \ W0 = {x ∈ W ; x ∈ / W0 }, / U0 }. U \ U0 = {u ∈ U ; u ∈ (5) Although g is a real analytic morphism of U \ U0 and W \ W0 , it is not of U and W in general. It is not a one-to-one map from U0 to W0 in general. (6) This theorem holds for any analytic function f such that f (0) = 0, even if 0 is not a critical point of f . (7) The triple (W, U, g) is not unique. There is an algebraic procedure by which we can find a triple, which is shown in Chapter 3. The manifold U is not orientable in general. (8) If f (x) ≥ 0 in the neighborhood of the origin, then all of k1 , k2 , . . . , kd in eq.(2.7) should be even integers and S = 1, hence eq.(2.7) can be replaced by 2kd 2k2 1 f (g(u)) = u2k 1 u2 · · · ud .

(2.9)

(9) The theorem shows resolution of singularities in the neighborhood of the origin. For the other point x0 , if f (x0 ) = 0, then the theorem can be applied to x0 ∈ Rd , which implies that there exists another triple (W, U, g) such that f (g(u) − x0 ) = S uk , g  (u) = b(u) uh , where S, k, h are different from those of the origin. (10) Let K be a compact set in an open domain of the real analytic function f (x). By collecting and gluing triples {W, U, g} for all points of K, we obtain

2.4 Resolution of singularities

61

Fig. 2.5. Collection of local resolutions

a global resolution of {x ∈ K; f (x) = 0} (Figure 2.5). Since K is compact, the number of local coordinates is finite. Remark 2.7 If there exists a real analytic function v = t(u), v = t(u) = (t1 (u), t2 (u), . . . , td (u)), whose Jacobian matrix is invertible, and if f (g(u)) = t1 (u)k1 t2 (u)k2 · · · td (u)kd ,

(2.10)

then the same result as Theorem 2.3 is obtained, because f (g(t −1 (v))) = v1k1 v2k2 · · · vdkd . Specifically, if we find a real analytic function g such that f (g(u)) = a(u)uk11 uk22 · · · ukdd ,

(2.11)

where a(u) = 0, then by using an analytic morphism v = t(u) defined by v1 = |a(u)|1/k1 u1 , vi = ui (2 ≤ i ≤ d), we have f (g(t −1 (v))) = v1k1 v2k2 · · · vdkd . Therefore if we find a function g such that eq.(2.10) or eq.(2.11) holds, then we obtain the same result as Theorem 2.3.

62

Singularity theory

w

x=u y = uw

u Coordinate gluing

y x t s

x = st y=t

Projective space

Fig. 2.6. Example of desingularization

Example 2.7 (1) Let us study f (x, y) = x 2 + y 2 in R2 . The triple (W, U, g) is defined as follows. Firstly, W = R2 . Secondly, two local open sets are defined by U1 = {(x1 , y1 ); −∞ < x1 , y1 < ∞}, U2 = {(x2 , y2 ); −∞ < x2 , y2 < ∞}. The manifold U is made by gluing two local coordinates U1 ∪ U2 as in Figure 2.6. Here (x1 , y1 ) and (x2 , y2 ) are identified as a point in the manifold U if and only if x1 y1 = x2 , y1 = x2 y2 . Thirdly, the map g : U → W (U = U1 ∪ U2 ) is defined on U1 by x = x1 y1 , y = y1 , and on U2 by x = x2 , y = x2 y2 . Then the map g is well defined as a function from U to W . The manifold U is a two-dimensional projective space introduced in Chapter 3, which is not

2.4 Resolution of singularities

63

orientable. Then f (g(·)) is a well-defined function on each coordinate, ! f (g(x1 , y1 )) = y12 1 + x12 , ! f (g(x2 , y2 )) = x22 1 + y22 . Here 1 + x12 > 0 and 1 + y22 > 0 are positive real analytic functions. The Jacobian matrix of f (g(u)) on U1 is

y1 x1 J = , 0 1 hence the Jacobian determinant is g  (u) = y1 . Note that g is not proper as a function g : U1 → W , because the inverse of a compact set g −1 ({0, 0}) = {(x1 , y1 ); y1 = 0} is not compact. However, it is proper as a function g : U → W . (2) The function introduced in Example 1.3 is f (x, y, z) = x 2 y 2 + y 2 z2 + z2 x 2 , which is defined on W = {(x, y, z) ∈ R3 }. We prepare three open sets defined by U1 = {(x1 , y1 , z1 ); −∞ < x1 , y1 , z1 < ∞}, U2 = {(x2 , y2 , z2 ); −∞ < x2 , y2 , z2 < ∞}, U3 = {(x3 , y3 , z3 ); −∞ < x3 , y3 , z3 < ∞}. A map from g : U1 ∪ U2 ∪ U3 → W is defined by x = x1 y1 z1 , y = y1 z1 , z = z1 , on U1 . On U2 and U3 , g is defined by x = x2 y2 = x3 , y = y2 = y3 z3 x3 , z = z2 x2 y2 = z3 x3 . Then U is a manifold of local coordinates U1 ∪ U2 ∪ U3 and g : U → W is a well-defined function. On U1 , ! g(x1 , y1 , z1 ) = y12 z14 1 + x12 + x12 y12 z12 ,

64

Singularity theory

y1

x = x1 y1 y = y1

x1

y

U1

x y2 W

x = x2 y = x2 y2

U2 x2

Fig. 2.7. Desingularization and integral

and the Jacobian determinant on U1 is g  (x1 , y1 , z1 ) = y1 z12 . Therefore (W, U, g) gives the resolution of singularities of f (x, y, z). (3) The above two examples are typical cases of desingularization. For general functions and an algorithm to find a triple (W, U, g), see Chapter 3. Example 2.8 Let W = {(x, y); |x|, |y| ≤ 1} in R2 and U1 = {(x1 , y1 ); |x1 |, |y1 | ≤ 1}, U2 = {(x2 , y2 ); |x2 |, |y2 | ≤ 1}. The map g : U1 ∪ U2 → W is defined on U1 by x = x1 y1 , y = y1 , and on U2 by x = x2 , y = x2 y2 , which means that the integration over W is the sum of the integrations over U1 and U2 as in Figure 2.7,

f (x, y)dxdy = f (x1 y1 , y1 )|y1 |dx1 dy1 W

U1




+

f (x2 , x2 y2 )|x2 |dx2 dy2 . U2

2.4 Resolution of singularities

65

If f (x, y) has a singularity at the origin, then the integral over W is sometimes made easier on U1 ∪ U2 by resolution of singularities. Definition 2.7 (Real log canonical threshold) Let f (x) be a real analytic function defined on an open set O ⊂ Rd . Let C be a compact set which is contained in O. For each point P ∈ C such that f (p) = 0, there exists a triple (W, U, g) as in Theorem 2.3 f (g(u) − P ) = S uk11 uk22 · · · ukdd , g  (u) = b(u)uh1 1 uh2 2 · · · uhd d , where (k1 , k2 , . . . , kd ) and (h1 , h2 , . . . , hd ) depend on the point P and triple (W, U, g). The real log canonical threshold for a given compact set C is defined by h + 1 j (C) = inf min , P ∈C 1≤j ≤d kj where, if kj = 0, we define (hj + 1)/kj = ∞. Theorem 2.4 The real log canonical threshold does not depend on the triple (W, U, g). Proof of Theorem 2.4 Let us introduce a zeta function of z ∈ C
ζ (z) = |f (x)|z dx. C

Then this function is holomorphic in Re(z) > 0 and
|f (g(u))|z |g  (u)|du ζ (z) = g −1 (C)

=


α

ukz+h b(u)du,

Uα ∩g −1 (C)

where Uα is the local coordinate. By using the Taylor expansion of g  (u) for an arbitrary order at the origin of each local coordinate, it follows that the zeta function can be analytically continued to a meromorphic function on the entire complex plane whose poles are all real, negative, and rational numbers. The log canonical threshold (C) corresponds the real largest pole (− min(hj + 1)/kj )
of the zeta function, which does not depend on the triple. Remark 2.8 (1) A number which does not depend on the triple is called a birational invariant. The real log canonical threshold is a birational invariant [50, 62, 78]. In Chapter 4, we study the generalized concept of the real log canonical threshold.

66

Singularity theory

(2) In this book, we mainly study real algebraic geometry. In complex algebraic geometry, there is the same concept, the log canonical threshold, but they are not equal to each other in general.

2.5 Normal crossing singularities By resolution of singularities, any singularities can be understood as the image of normal crossing singularities. Normal crossing singularities have very good properties which enable us to construct statistical learning theory. Definition 2.8 Let U be an open set in Rd . A real analytic function f (x) on U is said to be normal crossing at x ∗ = (x1∗ , x2∗ , . . . , xd∗ ) , if there exists an open set U  ⊂ U such that f (x) = a(x)

d 

(xj − xj∗ )kj

(x ∈ U  ),

j =1

where a(x) is a real analytic function (|a(x)| > 0), and k1 , k2 , . . . , kd are nonnegative integers. Theorem 2.5 Let r be a natural number (1 ≤ r ≤ d). If a real analytic function f (x) defined on an open set U ⊂ Rd satisfies (∀x ∈ U ),

“x1 x2 · · · xr = 0 ⇐⇒ f (x) = 0,”

(2.12)

where U contains the origin O, then there exist an open set U  (O ∈ U  ⊂ U ) and a real analytic function a(x) such that f (x) = a(x) x1 x2 · · · xr . Proof of Theorem 2.5 Let W be an open set in which the Taylor expansion of f (x) absolutely converges. By using the notation xˆ = (x2 , x3 , . . . , xd ), f (x) =

∞ 

ˆ 1i . ai (x)x

i=0

By assumption, ˆ = 0 ((0, x) ˆ ∈ W ). a0 (x) Thus f ∗ (x) = f (x)/x1 is analytic in W and f (x) = x1 f ∗ (x). By the recursive procedure, we obtain the theorem.




2.5 Normal crossing singularities

67

Remark 2.9 (Factor theorem) (1) If a real analytic function f (x) of a single variable x satisfies f (a) = 0 then f (x)/(x − a) is a real analytic function, which is the well-known factor theorem. (2) If the dimension of x is bigger than 1, then such a relation does not hold in general. For example, even if x 2 + y 2 = 0 ⇐⇒ f (x, y) = 0, a real analytic function a(x, y) which satisfies f (x, y) = a(x, y)(x 2 + y 2 ) does not exist in general. In fact, there is an example, f (x, y) = x 2 + y 4 . Division by a normal crossing function can be understood as the generalization of the factor theorem. Theorem 2.6 Let r be a natural number (1 ≤ r ≤ d). Assume that a real analytic function f (x) on an open set U ⊂ Rd satisfies   (2.13) |f (x)| ≤ Ax1k1 x2k2 · · · xrkr  (x ∈ U ), where A > 0 is a constant and k1 , k2 , . . . , kr are natural numbers. Then there exist an open set W ⊂ U and a real analytic function g(x) on W such that f (x) = g(x)x1k1 x2k2 · · · xrkr

(x ∈ W ).

Proof of Theorem 2.6 By Theorem 2.5, there exists a real analytic function a(x) such that f (x) = a(x)x1 x2 · · · xr , hence by eq.(2.13)

  |a(x)| ≤ Ax1k1 −1 · · · xrkr −1 .

By the recursive procedure, we obtain the theorem.




Remark 2.10 (1) Let U be an open set. Even if |f (x)| ≤ |g(x)|

(x ∈ U )

holds for real analytic functions, f (x)/g(x) is not a real analytic function in general. For example, the Cauchy–Schwarz inequality (xy + zw)2 ≤ (x 2 + z2 )(y 2 + w 2 ) holds; however, h(x, y, z, w) =

(xy + zw)2 (x 2 + z2 )(y 2 + w 2 )

(2.14)

68

Singularity theory

is not continuous at the origin. The limit value lim h(x)

x→0

does not exist, hence x = 0 is not a removable singularity. Theorem 2.6 claims that, if g(x) is normal crossing and if |f (x)| ≤ |g(x)| holds, then the origin is a removable singularity, consequently f (x)/g(x) can be made a real analytic function. (2) Although the origin of eq.(2.14) is not a removable singularity, h(x, y, z, w) can be made a real analytic function by resolution of singularities. In fact, by using x = x1 = x2 z2 , y = y1 = y2 w2 , z = x 1 z 1 = z2 , w = y1 w1 = w2 , the origin x = 0 is a removable singularity of the function h on a manifold, and h=

(1 + z1 w1 )2 (x2 y2 + 1)2 = . (1 + z12 )(1 + w12 ) (x22 + 1)(w22 + 1)

The function that has the same property is sometimes called a blow-analytic function. (3) In statistical learning theory, we have to study the log likelihood ratio function log(q(x)/p(x|w)) divided by the Kullback–Leibler distance K(w). In singular statistical models, such a function is ill-defined at singularities; however, it can be made well-defined by resolution of singularities. Theorem 2.7 Let U ⊂ Rd be an open set which contains the origin and r be a natural number (1 ≤ r ≤ d). Assume that two real analytic functions f1 (x) and f2 (x) on an open set U satisfy f1 (x)f2 (x) = x1k1 x2k2 · · · xrkr , where k1 , k2 , . . . , kr are natural numbers. Then there exist an open set W ⊂ U and real analytic functions a1 (w), a2 (w), such that j

j

f1 (x) = a1 (x)x11 x22 · · · xrjr , f2 (x) = a2 (x)x1h1 x2h2 · · · xrhr , where a1 (x)a2 (x) = 1 (x ∈ W ) (hence a1 (x) = 0, a2 (x) = 0) j1 , . . . , jr , h1 , . . . , hr are nonnegative integers.

and

2.5 Normal crossing singularities

69

Proof of Theorem 2.7 By the assumption, {x ∈ U ; f1 (x) = 0} ⊂ {x ∈ U ; x1 x2 · · · xr = 0}. Let I1 be the set of i (1 ≤ i ≤ r) which is defined by f1 (x1 , x2 , . . . , xi = 0, . . . , xd ) ≡ 0. Then {x ∈ U ; f1 (x) = 0} = {x ∈ U ;



xi = 0}.

i∈I1

Also we define I2 for f2 (x) in the same way. By Theorem 2.5, there exist real analytic functions g1 (x) and g2 (x) such that f1 (x) = g1 (x)

r 

j

xi i ,

i=1

f2 (x) = g2 (x)

r 

xihi ,

i=1

where ji and hi are defined as follows. If i ∈ I1 then ji = 1; otherwise ji = 0. If i ∈ I2 then hi = 1; otherwise hi = 0. Therefore  k −j −h xi i i i . g1 (x)g2 (x) = 1≤i≤r

By applying a recursive procedure, we obtain the theorem.




Remark 2.11 If both f1 (x) and f2 (x) are polynomials, this theorem is equivalent to the uniqueness of factorization. Theorem 2.8 (Simultaneous resolution of singularities) Let k be an integer and f0 (x), f1 (x), . . . , fk (x) be real analytic functions on an open set in Rd which contains the origin x = 0. Assume that, for all 0 ≤ i ≤ k, fi (0) = 0 and fi (x) is not a constant function. Then there exists a triple (W, U, g), (a) W is an open set in Rd , (b) U is a real analyic manifold, (c) g : U → W is a real analytic map, which satisfies the following conditions: (1) g is proper. (2) g is an analytic isomorphism of U \ U0 and W \ W0 where U0 = ∪i {x ∈ W ; fi (x) = 0} and W0 = ∪i {u ∈ U ; fi (g(u)) = 0}.

70

Singularity theory

(3) For an arbitrary point P ∈ W0 , there exists a local coordinate u = (u1 , u2 , . . . , ud ) such that f0 (g(u)) = uk101 uk202 · · · ukd0d f1 (g(u)) = a1 (u) uk111 uk212 · · · ukd1d f2 (g(u)) = a2 (u) uk121 uk222 · · · ukd2d .. . fk (g(u)) = ak (u) uk1k1 uk2k2 · · · ukdkd g  (u) = b(w) uh1 1 uh2 2 · · · uhd d , where kij and hi are nonnegative integers, and ai (w) = 0 (1 ≤ i ≤ k) and b(w) = 0 are real analytic functions. Proof of Theorem 2.8 Applying resolution of singularities to f (x) = f0 (x)f1 (x)f2 (x) · · · fk (x), there exists a triple (W, U, g) such that f (g(u)) = uk11 · · · ukdd . By Theorem 2.7, we obtain the theorem.




Remark 2.12 If a compact set K ⊂ Rd is defined by K = {x ∈ Rd ; f1 (x) ≥ 0, f2 (x) ≥ 0, . . . , fk (x) ≥ 0} using real analytic functions f1 (x), f2 (x), . . . , fk (x), then by applying simultaneous resolution of singularities, there exists a triple (W, U, g) by which all functions can be made normal crossing. Then the boundary of the set g −1 (K) is contained in u1 u2 · · · ud = 0 in every local coordinate. Theorem 2.9 (Finite critical values) Let U be an open set in Rd and K be a compact set in U . Assume that f (x) : U → R is a real analytic function. Then the set of critical values of a restricted function f : K → R {y ; y = f (x), ∇f (x) = 0, x ∈ K} has finite elements. Proof of Theorem 2.9 Since f (x) is a continuous function on a compact set, the set f (K) = {f (x) ∈ R1 ; x ∈ K}

2.5 Normal crossing singularities

71

is compact in R. Let y be a critical value in f (K). The closed set Ky ≡ {x ∈ K; f (x) − y = 0} ⊂ K is also compact. For an arbitrary x0 ∈ Ky , by applying the resolution theorem to f (x) − y, there exist an open set W (x0 ) that contains x0 , a manifold M(x0 ), and a real analytic map g : M(x0 ) → W (x0 ) such that f (g(u)) − y is a normal crossing function. Because  ∂f ∂xj ∂ f (g(u)) = , ∂ui ∂xj ∂ui j =1 d

the inverse of the critical points of f (x) − y is contained in the set of critical points of f (g(u)) − y. g −1 ({x ∈ W (x0 ); f (x) − y = 0, ∇f (x) = 0}) ⊂ {u ∈ M(x0 ); f (g(u)) − y = 0, ∇f (g(u)) = 0}. Here f (g(u)) − y is a normal crossing function, hence any critical point of u ∈ M(x0 ) is contained in the set of zero points of f (g(u)) − y = 0. Therefore any critical point of f (x) in W (x0 ) satisfies f (x) − y = 0. In other words, in each x0 ∈ Ky , we can choose W (x0 ) such that all critical points in W (x0 ) are contained in Ky . The set Ky can be covered by the union of neighborhoods Ky ⊂ ∪x0 ∈Ky W (x0 ).

(2.15)

By the compactness, Ky is covered by an open set Wy that is a finite union of W (x0 ). There exists no critical point in Wy \ Ky . The set of all critical points is covered by the union of open sets {Wy }. {x ∈ K; ∇f (x) = 0} ⊂ ∪y Wy .

(2.16)

The set of all critical points is compact because it is a set of all zero points of a continuous function ∇f . Hence ∪y in eq.(2.16) can be chosen to be a finite
union. Therefore the set of critical values has finite elements. Remark 2.13 Let f : Rd → R1 be a real analytic function and t be a real number. In the following sections, we study a Schwartz distribution δ(t − f (x)) of x ∈ Rd . If t is not a critical value of f (x), δ(t − f (x)) is a well-defined function, whereas, if t is a critical value of f (x), such a distribution cannot be defined. By Theorem 2.9, δ(t − f (x))ϕ(x) is well-defined except for finite set of t if a C ∞ class function ϕ(x) is equal to zero outside of a compact set.

72

Singularity theory

2.6 Manifold To study resolution of singularities, we need a mathematical preparation of a manifold. Definition 2.9 (Topological space) Let M be a set. A set U consisting of subsets of M is called a family of open sets if it satisfies the following conditions. (1) Both the set M and the empty set ∅ are contained in U. (2) If {Uλ } are subsets of U, then the union ∪λ Uλ is contained in U. (3) If U1 and U2 are contained in U, then the intersection U1 ∩ U2 is contained in U. The set M is called a topological space if a family of open sets is determined. It is called a Haussdorff space if, for arbitrary x, y ∈ M (x = y), there exist open sets U, V such that x ∈ U , y ∈ V , and U ∩ V = ∅. Definition 2.10 (System of local coordinates) Let M be a Haussdorff space and {Uα } be a family of open sets in M whose union covers M. Assume that, for each Uα , a map φα is defined from Uα to d-dimensional real Euclidean space Rd . The pair {Uα , φα } is said to be a system of local coordinates if, for any α and any open set U in Uα , φα is a continuous and one-to-one map from U to φα (U ) whose inverse is also continuous. A Haussdorff space M that has a system of local coordinates is called a manifold. For each r = 0, 1, 2, . . . , ∞, ω, a manifold M is said to be of class C r if it has a system of local coordinates such that, if U ∗ ≡ U1 ∩ U2 = ∅, both of the maps ! φ1 (U ∗ )  x → φ2 φ1−1 (x) ∈ φ2 (U ∗ ), ! φ2 (U ∗ )  x → φ1 φ2−1 (x) ∈ φ1 (U ∗ ) are of class C r (Figure 2.8). If M is a manifold of class C ω , it is called a real analytic manifold. Example 2.9 Here are examples and counter examples. (1) An open set that is not the empty set in Rd is a real analytic manifold. (2) The d-dimensional sphere {x ∈ Rd+1 ; x2 = 1} is a manifold. (3) Let U be an open set in Rd . If a real analytic function f : U → R1 satisfies ∇f (x) = 0 on the real analytic set U0 = {x ∈ U ; f (x) = 0}, then U0 is a manifold.

2.6 Manifold

73

Fig. 2.8. Definition of manifold

ρj (x)

M

Uj Fig. 2.9. Partition of unity

(4) A projective space, as defined in Chapter 3, is a manifold. (5) Let f (x) be a real analytic function which is represented as an absolutely convergent power series on an open set U1 ⊂ Rd . If it can be analytically continued to U1 ∪ U2 , where U2 is another open set, then U1 ∪ U2 is a manifold and f is a real analytic function on the real analytic manifold U1 ∪ U2 . (6) The set {(x, y) ∈ R2 ; xy = 0} is not a manifold because we cannot define a local coordinate in the neighborhood of the origin. (7) The set {(x, y) ∈ R2 ; xy = 0, (x, y) = (0, 0)} is a manifold. Theorem 2.10 (Partition of a unity) Let K be a compact set in a manifold M. Assume that the finite set of open sets {Uj , j = 1, 2, . . . , J } covers K, K⊂

J "

Uj .

j =1

Then there exists a set of functions {ρj (x)} of class C0∞ which satisfies the following conditions as in Figure 2.9.

74

Singularity theory

(1) For each j , 0 ≤ ρj (x) ≤ 1. (2) For each j , supp ρj ⊂ Uj .  (3) If x ∈ K, Jj=1 ρj (x) = 1. (Explanation of Theorem 2.10) This theorem is well known. The support of a function ρj (x) is defined by suppρj = {x ∈ M; ρj (x) > 0}. By this theorem we obtain the following corollary, which shows that the integration can be calculated as the sum of local integrations. Corollary 2.2 Let K be a compact set of a manifold M and µ be a measure on M. Then an integral of a function f (x) on K can be represented by
J
 f (x)µ(dx) = f (x)ρj (x)µ(dx), K

j =1

K

where {ρj } is the partition of a unity given in Theorem 2.10. Remark 2.14 (Partition of a manifold) In singular learning theory, we study a real analytic function f0 (x) defined on an open set in Rd . To analyze an integration over the parameter space W more precisely, we prepare the division of integration. Let π1 (x), π2 (x), . . . , πk (x) be real analytic functions defined on the open set in Rd . Let us study a compact set W defined by W = {x ∈ Rd ; π1 (x) ≥ 0, π2 (x) ≥ 0, . . . , πk (x) ≥ 0}. By applying the simultaneous resolution theorem, Theorem 2.8, there exist a compact set M of a real analytic manifold and a real analytic map g : M → W such that π0 (g(u)), π1 (g(u)), . . . , πk (g(u)) are simultaneously normal crossing. For each point p ∈ M, an open set of a local coordinate whose origin is p (−b, b)d ≡ {u = (u1 , u2 , . . . , ud ); |ui | < b, i = 1, 2, . . . , d} is denoted by Op (b), where b > 0 is a positive constant. Then M is covered by the union of {Op } M ⊂ ∪p∈M Op (b). Since M is a compact set, it is covered by a finite union of {Op (b)}. Moreover, the set (−b, b)d is covered by 2d sets which are respectively analytically isomorphic to [0, b)d . Each set in M that is analytically isomorphic to [0, b)d is denoted by Mα . Then the set M is covered by a finite union of {Mα }. M ⊂ ∪α Mα .

2.6 Manifold

75

K(g(u)) = 0

[0,b] d [0,b] d

Fig. 2.10. Local coordinate made of [0, b]d

Let us define a function σα (u) whose support is contained in Mα by   1   #d exp − (0 ≤ ui < 1 (∀i)) i=1 σα(0) (u) = 1 − ui 0 otherwise and σα (u) by σ (0) (u) σα (u) =  α (0) . α σα (u) Then σα (u) is a function of class C w in (0, b)d , and σα (u) > 0 (u ∈ [0, b)d ). Moreover, 

σα (u) = 1

(u ∈ M).

α

For arbitrary integrable function H (·),

H (x)dx = H (g(u))|g  (u)|du M

W

=


α

H ((g(u))σα (u)|g  (u)|du.

Mα

Moreover, since the number of elements {Mα } is finite if H (x) ≥ 0,


 H (g(u))|g (u)|du ≤ H (x)dx ≤ C2 H (g(u))|g  (u)|du C1 α

Mα

W

α

Mα

where C1 , C2 > 0 are constants. This method is used in later chapters.

76

Singularity theory

Theorem 2.11 (Partition of an integral) By using the above notation,

H (x)dx = H (g(u))|g  (u)|du M

W

=


α

H (g(u))σα (u)|g  (u)|du. Mα

where each Mα is equal to [0, b] in local coordinate. Moreover, if H (x) ≥ 0,


H (g(u))|g  (u)|du ≤ H (x)dx ≤ C2 H (g(u))|g  (u)|du C1 d

α

Mα

where C1 , C2 > 0 are constants.

W

α

Mα

3 Algebraic geometry

In this chapter, we introduce basic concepts in algebraic geometry: ring and ideal, the relation between algebra and geometry, projective spaces, and blowups. Based on this foundation, the method of how to find a resolution map is introduced. In this book, we mainly study real algebraic geometry.

3.1 Ring and ideal In order to analyze a real algebraic set, we need an algebraic structure of polynomials. Let R = R[x1 , x2 , . . . , xd ] be the set of all polynomials of x1 , x2 , . . . , xd with real coefficients. An element f (x) ∈ R can be written as a finite sum  aα x α , f (x) = α

where α is a multi-index and aα is a real number. For f (x), g(x) ∈ R  f (x) = aα x α ,  g(x) = bα x α , and s ∈ R, we define



f (x) + g(x) =

(aα + bα )x α ,

α



sf (x) =

(saα )x α ,

α

by which R is an infinite-dimensional vector space over a field R. Moreover, R is a ring by defining  aα bβ x α+β . f (x)g(x) = α

The ring R is called a polynomial ring. 77

β

78

Algebraic geometry

Definition 3.1 (Ideal) A subset I in R = R[x1 , x2 , . . . , xd ] is said to be an ideal if 0 ∈ I, f (x), g(x) ∈ I =⇒ f (x) + g(x) ∈ I, f (x) ∈ I, g(x) ∈ R =⇒ f (x)g(x) ∈ I. For a given set of polynomials f1 , f2 , . . . , fk , we define a subset of R by $ k %  f1 , f2 , . . . , fk  = gi (x)fi (x) ; gi ∈ R , i=1

which is the minimum ideal that contains f1 , f2 , . . . , fk . This ideal is called the ideal generated by f1 , f2 , . . . , fk . For a subset S ⊂ R[x1 , x2 , . . . , xd ], the linear hull L.H.(S) is defined by the smallest vector space that contains S, $ %  L.H.(S) = ai fi ; ai ∈ R, fi ∈ S , where

i

 i

shows the finite sum.

Example 3.1 The following are ideals in R[x, y]: x 2  = L.H.({x 2 , x 3 , . . . , x 2 y, x 2 y 2 , . . . , }), x 3 , y + 1 = L.H.({x 3 , x 4 , . . . , x 3 y, x 4 y, . . . , y +1, x(y + 1)2 , . . . , }), x + y, x 2 − y 2  = x + y, x + y 2 , x − y 2 , y 3  = x, y 2 , x 2  ⊂ x, x 2 + y 2  ⊂ x, y. If x m y n is contained in I , then x m+k y n+l ∈ I for arbitrary nonnegative integers k, l. From the definition, the following lemma is immediately derived. Lemma 3.1 (1) An ideal in R is a ring. It is a subring of R. A subring of R is not an ideal in general. (2) If an ideal in R is not 0, then it is an infinite-dimensional vector space. (3) If an ideal contains 1, then it is equal to R.

3.1 Ring and ideal

79

Theorem 3.1 (Hilbert’s basis theorem) For an arbitrary ideal I in R ≡ R[x1 , x2 , . . . , xd ], there exists a finite set f1 , f2 , . . . , fk ∈ R[x1 , x2 , . . . , xd ] such that I = f1 , f2 , . . . , fk . (Explanation of Theorem 3.1) This is the fundamental theorem of ideal theory. For the proof, see [21]. This theorem holds in real and complex polynomial rings. Remark 3.1 (1) A ring is said to be Noetherian if Hilbert’s basis theorem holds for any ideal in the ring. (2) If a ring A is Noetherian, then A[x] is also Noetherian. (3) A ring is Noetherian if and only if, for any nondecreasing sequence of ideals, I1 ⊂ I2 ⊂ · · · ⊂ In ⊂ · · · , there exists N such that for all n ≥ N In = In+1 = In+2 = · · · = . Example 3.2 In statistical learning theory, Hilbert’s basis theorem plays an important role. Let us study a parametric function on x ∈ [0, 1], f (x, a, b, c, d) = a sin(bx) + c sin(dx). Let S be a set of parameters which make f (x, a, b, c, d) ≡ 0, that is to say, S = {(a, b, c, d) ∈ R4 ; f (x, a, b, c, d) = 0 (∀x ∈ [0, 1])}. By using the defintion gk , gk (a, b, c, d) = ab2k+1 + cd 2k+1 (k = 0, 1, 2, . . . , ), the Taylor expansion shows that f (x, a, b, c, d) =

∞  (−1)k x 2k+1 k=0

(2k + 1)!

gk (a, b, c, d).

Since the set of functions {x 2k+1 } is linearly independent on [0, 1], f (x, a, b, c, d) ≡ 0 if and only if gk (a, b, c, d) = 0 (∀k). Hence the set S is represented by S = {(a, b, c, d) ∈ R4 ; g0 (a, b, c, d) = g1 (a, b, c, d) = · · · = 0},

80

Algebraic geometry

which shows S is a common locus defined by infinite polynomials. Let us define an ideal Ik by Ik = g0 , g1 , g2 , . . . , gk . Then it defines a nondecreasing sequence of ideals, I0 ⊂ I1 ⊂ I2 ⊂ I3 ⊂ · · · . By Hilbert’s basis theorem, there exists k such that S = {(a, b, c, d) ∈ R4 ; g0 (a, b, c, d) = · · · = gk (a, b, c, d) = 0}, which shows S is a real algebraic set. To obtain k, we need concrete calculation. In this case, it is easy to show gk+1 (a, b, c, d) = g1 (a, b, c, d)(b2k + d 2k ) − g0 (a, b, c, d)(b2 d 2k + b2k d 2 ) + b2 d 2 gk−1 (a, b, c, d), hence k = 1. Therefore, S = {(a, b, c, d) ∈ R4 ; ab + cd = ab3 + cd 3 = 0}. This relation can be generalized. A polynomial pn (a1 , . . . ad , b1 , . . . , bd ) =

d 

ak bkn ,

k=1

is contained in the ideal pn ∈ p1 , p2 , . . . , pd  for any natural number n.

3.2 Real algebraic set A real algebraic set is a geometric object and an ideal is an algebraic object. There are mathematical relations between geometry and algebra. In algebraic geometry, the set made of d times direct product R × R × · · · × R is called a d-dimensional affine space, which is denoted by Ad . Since this set is equal to the Euclidean space Rd as a set, in this book we identify the affine space with the Euclidean space. Definition 3.2 For an ideal I of R[x1 , x2 , . . . , xd ], a set V(I ) is defined by V(I ) = {x ∈ Rd ; f (x) = 0 (∀f ∈ I )}.

3.2 Real algebraic set

81

A subset V of Rd is said to be a real algebraic set if there exists an ideal I such that V = V(I ). The map I → V(I ) is defined from an ideal to a real algebraic set. Example 3.3 (1) The set of zero locus of polynomials f1 (x), f2 (x), . . . , fk (x) is equal to V(f1 , f2 , . . . , fk ). For example the intersection of x 2 + y 2 + z2 = 10 and xyz = 1 is V(x 2 + y 2 + z2 − 10, xyz − 1). (2) By definition, the map I → V(I ) from the set of all ideals to the set of all real algebraic sets is surjective. (3) The map I → V(I ) is not one-to-one. For example, I1 = x, y,

I2 = x 2 , y 2 , I3 = x 2 + y 2 .

Although three ideals Ii (i = 1, 2, 3) are different from each other, the corresponding algebraic sets are equal to each other, V(I1 ) = V(I2 ) = V(I3 ) = {(0, 0)}. (4) For I = x 2 + 1, V(I ) is the empty set. Definition 3.3 (Defining ideal) For a given real algebraic set V , an ideal I(V ) is defined by the set of all polynomials which are equal to zero on V , I(V ) = {f (x) ∈ R[x1 , x2 , . . . , xd ] ; f (x) = 0 (∀x ∈ V )}. This set is an ideal of R[x1 , x2 , . . . , xd ] because it satisfies the definition for an ideal. The ideal I(V ) is called a defining ideal of a real algebraic set V . Example 3.4 (1) For V = {(x, y); x + y + 1 = 0}, I(V ) = x + y + 1. (2) For V = {(x, y); x 2 y 3 = 0}, I(V ) = xy. (3) For V = {(x, y); x 3 + y 3 = 0}, I(V ) = x + y. (4) For V = {(x, y); x 2 + y 2 = 0}, I(V ) = x, y. (5) For the empty set ∅, I(∅) = 1 = R[x1 , x2 , . . . , xd ]. Theorem 3.2 (Real algebraic set and ideal) Let V1 and V2 be real algebraic sets and I1 and I2 be ideals. (1) V1 ⊂ V2 ⇐⇒ I(V1 ) ⊃ I(V2 ). (2) I1 ⊂ I2 =⇒ V(I1 ) ⊃ V(I2 ). Proof of Theorem 3.2 (1) (=⇒) Let f ∈ I(V2 ). Then f (x) = 0 for all x ∈ V2 . Hence f (x) = 0 for all x ∈ V1 , giving f ∈ I(V1 ). (⇐=) Let x ∈ V1 . Then f (x) = 0 for all f ∈ I(V1 ). Hence f (x) = 0 for all f ∈ I(V2 ), giving x ∈ V2 . (2) (=⇒) Let x ∈ V(I2 ). Then f (x) = 0 for all f ∈ I2 . Hence f (x) = 0 for all
f ∈ I1 , giving x ∈ V(I1 ).

82

Algebraic geometry

Remark 3.2 In Theorem 3.2 (2), ⇐= does not hold. For example, I1 = x, I2 = x 2 + y 2 . Then V(I1 ) ⊃ V(I2 ), but I1 ⊂ I2 . Definition 3.4 (Radical of an ideal). For a given ideal I in R[x1 , x2 , . . . , xd ], the radical of the ideal I is defined by √ I = {f (x) ∈ R[x1 , x2 , . . . , xd ] ; f (x)m ∈ I (∃m : natural number)}, which is also an ideal. √ Remark 3.3 (1) By definition, I ⊂ I . √ √ I , then I is called a radical ideal. Since I= (2) If an ideal I satisfies I = √ √ I , the radical I is a radical ideal. (3) For any real algebraic set V , I(V ) is a radical ideal. (4) The map V → I(V ) can be understood as a function from a real algebraic set to a radical ideal. The map V → I(V ) is not surjective. For example, for a radical ideal I = x 2 + 1, there does not exist a real algebraic set V such that I(V ) = I . (5) The map V → I(V ) is one-to-one by Theorem 3.2 (1). Theorem 3.3 Let V be a real algebraic set and I be an ideal. (1) V(I(V √)) = V . (2) I ⊂ I ⊂ I(V(I )). Proof of Theorem 3.3 (1) Firstly, we prove V ⊂ V(I(V )). Let x ∈ V . Then f (x) = 0 for all f ∈ I(V ), which is equivalent to x ∈ V(I(V )). Secondly, we prove V ⊃ V(I(V )). Since V is a real algebraic set, there exist polynomials f1 , f2 , . . . , fk such that V = V(f1 , f2 , . . . , fk ). Then f1 , f2 , . . . , fk ∈ I(V ), thus f1 , f2 , . . . , fk  ⊂ I(V ). By using Theorem 3.2(2), ⊃ V(I(V )). V = V(f1 , f2 , . . . , fk ) √ √ √ (2) By Remark 3.3, I ⊂ I . Let us prove I ⊂ I(V(I )). If f (x) ∈ I , then there exists m such that f (x)m ∈ I . Therefore f (x) = 0 for any x ∈ V(I ),
which is equivalent to f (x) ∈ I(V(I )). Example 3.5 In R[x, y], ideals I1 = x 2 y 4 , I2 = (x + 1)2 , y 3 z2 , I3 = x 4 + 2x 2 + 1 are not radical ideals. Radicals of them are respectively    I1 = xy, I2 = x + 1, yz, I3 = x 2 + 1.

3.2 Real algebraic set

83

The real algebraic sets are respectively V(I1 ) = V(xy) = V(x) ∪ V(y), V(I2 ) = V(x + 1) ∩ {V(y) ∪ V(z)}, V(I3 ) = ∅, whereas corresponding ideals are respectively given by I(V(I1 )) = xy, I(V(I2 )) = x + 1, yz, I(V(I3 )) = 1 = R[x, y]. In R[x, y, z, w] I = (xy + zw)2 + (xy 3 + zw 3 )2  is a radical ideal. I(V(I )) = xy + zw, xy 3 + zw 3  =

√ I = I.

Remark 3.4 (Hilbert’s Nullstellensatz) (1) In this book, we study mainly the polynomial ring with real coefficients R[x1 , x2 , . . . , xd ] and the real algebraic set contained in Rd . If we have the polynomial ring with complex coefficients and the complex algebraic set contained in Cd , Hilbert’s Nullstellensatz √ I(V(I )) = I holds. This equation is equivalent to the proposition that the map V → I(V ) is surjective onto the set of radical ideals. (2) Hilbert’s Nullstellensatz holds for any polynomial ring with coefficients of an algebraic closed field. If Hilbert’s Nullstellensatz holds, then the correspondence complex algebraic set → radical ideal is one-to-one and onto, giving the result that an algebraic set can be identified with a radical ideal. (3) We study mainly real algebraic geometry, therefore Hilbert’s Nullstellensatz does not hold. However, there are some properties between ideals and real algebraic sets. Definition 3.5 (Prime ideal and irreducible set) (1) An ideal I in R = R[x1 , x2 , . . . , xd ] is called a prime ideal if I = R and if f (x)g(x) ∈ I =⇒ f (x) ∈ I or g(x) ∈ I .

84

Algebraic geometry

(2) A real algebraic set V in Rd is said to be irreducible if V = V1 ∪ V2 =⇒ V = V1 or V = V2 . Remark 3.5 (1) A real algebraic set V is irreducible if and only if I(V ) is a prime ideal. (2) For an arbitrary real algebraic set V , there exist irreducible real algebraic sets V1 , . . . , Vk such that V = V1 ∪ V2 ∪ · · · ∪ Vk . Example 3.6 (1) In R[x, y], x is a prime ideal. x 3 − y 2  is also a prime ideal. xy is not a prime ideal. x 2  is not a prime ideal. x 2 + y 2 , xy is not a prime ideal, because (x + y)2 ∈ x 2 + y 2 , xy. (2) For a real algebraic set V = V((ab + c)2 + c4 ), V1 = V(a, c), V2 = V(b, c). it follows that V = V1 ∪ V2 , hence V is not irreducible. Note that (ab + c)2 + c4  is a prime ideal, whereas I(V ) = ab, c is not a prime ideal. Definition 3.6 (Maximal ideal) An ideal I in R[x1 , x2 , . . . , xd ] is said to be a maximal ideal if I = R and, for any ideal J , I ⊂ J =⇒ I = J or J = R. Example 3.7 In R[x, y, z], the following results hold. (1) x − a, y − b, z − c is a maximal ideal for arbitrary a, b, c. (2) x 2 + y 2 + z2  is not a maximal ideal, because it is contained in x, y, z. (3) x 2 + 1, y, z is a maximal ideal. (4) In the polynomial ring C[x1 , x2 , . . . , xd ] with complex coefficients, an ideal I is a maximal ideal if and only if I = x1 − a1 , x2 − a2 , . . . , xd − ad  for some (a1 , a2 , . . . , ad ) ∈ Cd . Remark 3.6 (1) A maximal ideal is a prime ideal. (2) A prime ideal is a radical ideal.

3.2 Real algebraic set

85

Remark 3.7 A polynomial f is said to be irreducible if f (x) = g(x)h(x) implies f (x) ∝ g(x) or f (x) ∝ h(x). Any polynomial f (x) can be represented by f (x) =

k 

fi (x)ai

i=1

using irreducible polynomials f1 , . . . , fk , where {ai } is a set of natural numbers. For an arbitrary polynomial f (x), fi (x) and ai are determined uniquely without trivial multiplication. By using this representation,  f  = f1 f2 · · · fk  holds. If the coefficient of the ring is an algebraically closed field, then the condition that a polynomial f is irreducible is equivalent to the condition that the real algebraic set V(f ) is irreducible. However, if the coefficient is not an algebraic closed set, then its equivalence does not hold; see Example 3.6 (2). In order to check whether a real algebraic set is irreducible or not, I(V ) should be studied. √ In real algebraic geometry, the geometry of V = V(I ) corresponds not to I but to I(V ). Definition 3.7 (Coordinate ring) Let V be a real algebraic set in Rd . The polynomial ring restricted on V is called a coordinate ring of V , which is denoted by R[V ]. Example 3.8 Let V ⊂ R2 be V = V(x 2 − y 3 ). Then, in R[V ], x 3 + y 3 = x 3 − x 2 = y 3 (x − 1). Remark 3.8 In the polynomial ring R[x1 , x2 , . . . , xd ], we introduce an equivalence relation f ∼ g ⇐⇒ f (x) = g(x)

(x ∈ V ).

Then the quotient set R/∼ is equivalent to the coordinate ring. If R[V ]  f (x)g(x) = 0 =⇒ f (x) = 0 or g(x) = 0 then R[V ] is said to be an integral domain. The ideal I(V ) is a prime ideal if and only if R[V ] is an integral domain.

86

Algebraic geometry

3.3 Singularities and dimension In this section, we study a necessary and sufficient condition of a singularity in a real algebraic set. Definition 3.8 (Dimension of a real algebraic set) Let V be a nonempty real algebraic set in Rd . Assume that polynomials f1 , f2 , . . . , fr satisfy I(V ) = f1 , f2 , . . . , fr . The Jacobian matrix is defined as  ∂f (x) 1

 J (x) =  

∂x1

.. .

∂fr (x) ∂x1

... .. . ...

∂f1 (x) ∂xd

.. .

∂fr (x) ∂xd

   

The maximum value of the rank of the Jacobian matrix, d0 = max rankJ (x), x∈V

is said to be the dimension of the real algebraic set V . Theorem 3.4 Let V be a nonempty real algebraic set in Rd whose dimension is equal to d0 . Then x ∈ V is a nonsingular point of V if and only if rankJ (x) = d0 . In other words, x ∈ V is a singularity if and only if rankJ (x) < d0 . (Explanation of Theorem 3.4) For the proof of this theorem, see [21]. In this book, the definition of a singularity is given in Definition 2.6. Therefore the above statement is a theorem. In some books, the above statement is used as a definition of singularity and Definition 2.6 is a theorem. The condition rankJ (x) < d0 holds if and only if all minor determinants of d0 × d0 are equal to zero. Since the elements of a Jacobian matrix are polynomials, all minor determinants are also polynomials. Therefore the set of singularities Sing(V ) is a real algebraic subset of V , and Sing(V ) is not equal to V . That is to say, Sing(V ) ⊂ V and Sing(V ) = V . Example 3.9 (1) In R3 , let us study a real algebraic set V = V(f, g), where f (x, y, z) = x 2 − y, g(x, y, z) = x 3 − z2 . The Jacobian matrix is given by

2x −1 0 . J (x) = 3x 2 0 −2z

3.4 Real projective space

87

Therefore,  rankJ (x) =

1 (x = (0, 0, 0)) 2 (x = (0, 0, 0)).

Hence the set of singularities is Sing(V ) = {(0, 0, 0)}. (2) In R3 V = V(f, g), where f (x, y, z) = x 2 − 2xy + z and g(x, y, z) = y 2 − z. Then I(V ) = x − y, y 2 − z, where we used f (x, y, z) + g(x, y, z) = (x − y)2 . Therefore

1 −1 0 J (x) = . 0 2y −1 Hence rankJ (x) = 2 for arbitrary x, which means that V does not contain a singularity.

3.4 Real projective space Let Rd+1 be the real Euclidean space and O be the origin. We introduce an equivalence relation ∼ to the set Rd+1 \ O. (x0 , x1 , x2 , . . . , xd ) ∼ (y0 , y1 , y2 , . . . , yd ) if and only if (∃c = 0)

(x0 , x1 , x2 , . . . , xd ) = c (y0 , y1 , y2 , . . . , yd ).

The quotient set is said to be a d-dimensional projective space, Pd = Rd+1 /∼ . The equivalence class that contains (x0 , x1 , x2 , . . . , xd ) ∈ Rd+1 \ O is denoted by (x0 : x1 : x2 : · · · : xd ). By the definition, (x0 ; x1 : x2 : · · · : xd ) = (cx0 : cx1 : cx2 : · · · : cxd )

88

Algebraic geometry

holds for c = 0. The subsets Uk (k = 0, 1, 2, . . . , d) of Pd are defined by U0 = {(1 : x1 : x2 : · · · : xd ); x1 , x2 , . . . , xd , ∈ R},

(3.1)

U1 = {(x0 : 1 : x2 : · · · : xd ); x0 , x2 , . . . , xd ∈ R}, .. .

(3.2)

Ud = {(x0 : x1 : · · · : 1); x0 , x1 , . . . , xd−1 ∈ R}.

(3.4)

(3.3)

Then P d = U0 ∪ U1 ∪ · · · ∪ Ud holds. Since Uk is analytically isomorphic to Euclidean space Rd , P d is a d-dimensional manifold and Uk is a local coordinate. Example 3.10 (1) One-dimensional projective space P1 is P1 = {(x : y) ; (x, y) = (0, 0)}, which can be rewritten as P1 = {(1 : a); a ∈ R} ∪ {(0 : 1)} ∼ R1 ∪ R0 , = where A ∼ = B means that there is a one-to-one and onto map between A and B. (2) Two-dimensional projective space P2 is P2 = {(1 : a : b); (a, b) ∈ R2 } ∪ {(0 : a : b); (a : b) ∈ P1 } ∼ = R2 ∪ P1 ∼ = R2 ∪ R1 ∪ R0 . (3) In the same way, P d = {(1 : x1 : · · · : xd ); (x1 , · · · , xd ) ∈ Rd } ∪{(0 : x1 : · · · : xd ); (x1 : · · · : xd ) ∈ Pd−1 } ∼ = Rd ∪ Rd−1 ∪ · · · ∪ R0 . Remark 3.9 The real projective space P d is orientable if and only if d is an odd number. Definition 3.9 (Homogeneous ideal) For a multi-index α = (α0 , α1 , . . . , αd ), |α| = α0 + α1 + · · · + αd . If a polynomial f (x) ∈ R[x0 , x1 , . . . , xd ] is given by  aα x α , f (x) = |α|=n

3.4 Real projective space

89

then f (x) is said to be a homogeneous polynomial of degree n. If an ideal I ⊂ R[x0 , x1 , . . . , xd ] is generated by homogeneous polynomials f1 (x), f2 (x), . . . , fk (x), so that I = f1 , f2 , . . . , fk , then I is called a homogeneous ideal. Remark 3.10 (1) In R[x, y, z], both x 2 y + z3 and x + y are homogeneous polynomials, hence I = x 2 y + z3 , x + y is a homogeneous ideal. The ideal contains polynomials that are not homogeneous, for example, x 2 y + z3 + x(x + y). (2) For a given polynomial  f (x) = aα x α , α

its homogeneous part of degree n is degn (f )(x) =



aα x α .

|α|=n

An ideal I is a homogenous ideal if and only if, for an arbitrary f ∈ I , its homogeneous part of any degree is contained √ in I . (3) For a homogenous ideal I , its radical I is homogeneous. Definition 3.10 (Real projective variety) If a subset V ⊂ P d is represented by homogeneous polynomials f1 (x), f2 (x), . . . , fk (x) ∈ R[x0 , x2 , . . . , xd ], V = {(x0 : x1 : · · · : xd ) ∈ P d ; f1 (x) = f2 = · · · = fk (x) = 0}, then V is said to be a real projective variety. This set V is written V = V(f1 , f2 , . . . , fk ). Since f1 , . . . , fk are homogeneous polynomials, V is a well-defined subset in the real projective set. Also, for a given homogeneous ideal I , V(I ) = {(x0 : x1 : · · · : xd ) ∈ P d ; degn (f ) = 0, (∀f ∈ I, ∀n ≥ 1)}. Example 3.11 Let V = V(x 3 + xyw + w3 ) ⊂ P3 be a real projective variety. By definition, V = {(x : y : z : w) ∈ P3 ; x 3 + xyw + w3 = 0}. In the local coordinate U0 in eq.(3.1), V ∩ U0 = {(1 : y : z : w); 1 + yw + w3 = 0}.

90

Algebraic geometry

In the same way, V ∩ U1 = {(x : 1 : z : w); x 3 + xw + w3 = 0}. In each local coordinate, a real projective variety is defined by nonhomogeneous polynomials. On the other hand, a real projective variety is determined by one of its local coordinates. For example, if a real projective variety V satisfies V ∩ U0 = {(1 : y : z : w); y 3 + yz + w3 + 2 = 0}, then

 &  y 3 y z  w 3 V = (x : y : z : w); + · + +2=0 x x x x = {(x : y : z : w); y 3 + xyz + w3 + 2x 3 = 0},

which is defined by a homogeneous polynomial. Definition 3.11 For a real projective variety V ⊂ P d , its defining ideal is defined by I(V ) = {f (x) ∈ R[x0 , x1 , . . . , xd ]; f (x) = 0 (∀x ∈ V )}. By definition, I(V ) is a homogeneous ideal. Remark 3.11 Between real projective varieties V and a homogenous ideal I , the following relations hold. Let V1 and V2 be real projective varieties and I1 and I2 be homogeneous ideals. (1) V1 ⊂ V2 ⇐⇒ I(V1 ) ⊃ I(V2 ). (2) I1 ⊂ I2 =⇒ V(I1 ) ⊃ V(I2 ). (3) V √ = V(I(V )) = V . (4) I ⊂ I(V(I )). Remark 3.12 (Zariski topology) In real Euclidean space Rd , the topology is usually determined by the Euclidean norm. Since the real projective space P d is the manifold whose local coordinate is analytically isomorphic to Rd , its topology is determined from Rd . However, in algebraic geometry, the other topology is employed. The Zarisiki topology of Rd is determined by the condition that the family of all real algebraic sets is equal to the family of all closed sets. Also the Zariski topology of P d is determined in the same way. Note that a closed set by Zariski topology is also a closed set by Euclidean topology, but that a closed set by Euclidean topology may not be a closed set by Zariski topology. The closure of a set A by Zariski topology is the smallest real algebraic set that

3.5 Blow-up

91

contains A. An open set in P d is called a quasiprojective variety. For example, local coordinates U0 , U1 , . . . , Ud of P d are quasiprojective varieties.

3.5 Blow-up In this section, we introduce a blow-up. Intuitively speaking, a blow-up reduces the complexity of a singularity. Any singularities can be desingularized by finite recursive blow-ups. Definition 3.12 (Blow-up of Euclidean space) Let Rd be a real Euclidean space, and r be an integer which satisfies 2 ≤ r ≤ d. Let V be a real algebraic set, V = {x = (x1 , x2 , . . . , xd ) ∈ Rd ; x1 = x2 = · · · = xr = 0}. The blow-up of Rd with center V is defined by BV (Rd ) ≡ {(x, (x1 : x2 : · · · : xr )) ∈ Rd × Pr−1 ; x ∈ Rd \ V } where A shows the closure of a set A in Rd × Pr−1 . Here the topology of Rd × Pr−1 is the direct product of both sets. In this definition, closures by both Zariski and Euclidean topologies result in the same set. Remark 3.13 This definition represents the procedure of the blow-up. (1) Firstly, we prepare Rd \ V , in other words, the center V is removed from Rd . (2) Secondly, a point in real projective space (x1 : x2 : · · · : xr ) ∈ Pr−1 can be defined for x = (x1 , x2 , . . . , xd ) ∈ Rd \ V . (3) Thirdly, the pair (x, (x1 : x2 : . . . , xd )) is collected, A = {(x, (x1 : x2 : · · · : xr )) ; x ∈ Rd \ V }. This set is a subset of Rd × Pr−1 . (4) And lastly, the closure of A in Rd × Pr−1 gives the blow-up. By taking the closure, a set {(x, y); x ∈ V , y ∈ Pr−1 } is added to A. Remark 3.14 Let us define a map π : BV (Rd ) → Rd

92

Algebraic geometry

z

z z =+

z=+ y=zx

y x

y

2

R

x

Fig. 3.1. Blow-up of R2

by π((x, y)) = x. Then π : {(x, (x1 : x2 : · · · : xr )); x ∈ Rd \ V } → Rd \ V is a one-to-one and onto map. Moreover π −1 (V ) = Pr−1 holds. Example 3.12 To understand what a blow-up is, the following example is very helpful. The blow-up of R2 with center V = V(x, y) = {(0, 0)} is illustrated / V = {(0, 0)}. in Figure 3.1. Firstly, (x : y) ∈ P1 can be defined for (x, y) ∈ Secondly, the set A = {(x, y, (x : y)) ; (x, y) ∈ / V } ⊂ R 2 × P1 is introduced. By using a notation (x : y) = (1 : z) ∈ P1  y/x (x = 0) z= ∞ (x = 0),

(3.5)

where ∞ = (0 : 1), the set A is rewritten as A = {(x, y, z) ; y = zx (x, y) = (0, 0)} ⊂ R2 × P1 , where, if z = ∞, then y = zx is replaced by x = 0. Lastly the blow-up BV (Rd ) is obtained by its closure, BV (R2 ) = {(x, y, z) ; y = zx} ⊂ R2 × P1 ,

3.5 Blow-up

93

which is a M¨obius’ strip. In this case, taking closure is equivalent to removing the condition (x, y) = (0, 0). The projection map is given by π : (x, y, z) → (x, y), therefore π −1 ({(0, 0)}) = {(0, 0)} × P1 . Definition 3.13 (Blow-up of a real algebraic set) Let Rd be a d-dimensional real Euclidean space. Let r be an integer which satisfies 2 ≤ r ≤ d. V = {x ∈ Rd ; x1 = x2 = · · · = xr = 0}. Let W be a real algebraic set such that V ⊂ W . The blow-up of W with center V is defined by BV (W ) ≡ {(x, (x1 : x2 : · · · : xr )); x ∈ W \ V }. Remark 3.15 (Strict and total transform, exceptional set) Let π be a map defined in Remark 3.14. Then BV (W ) ⊂ π −1 (W ) holds, but BV (W ) = π −1 (W ). The set BV (W ) is said to be a strict transform of W , whereas π −1 (W ) is a total transform. π −1 (W ) \ BV (W ) is called an exceptional set. Example 3.13 Let a real algebraic set in R2 be W = V(x 3 − y 2 ). In Figure 3.2, we illustrate the blow-up of W with center V = V(x, y) = {(0, 0)}. Firstly, V = {(0, 0)} is removed from W . Secondly, the subset A in R2 × P1 is defined by A = {(x, y, (x : y)) ; (x, y) ∈ W \ V } = {(x, y, (x : y)) ; x 3 − y 2 = 0, (x, y) = (0, 0)}. By using the same notation (1 : z) ∈ P1 as in eq.(3.5), the set A is rewritten as A = {(x, y, z) ; x 3 − y 2 = 0, y = zx, (x, y) = (0, 0)}.

94

Algebraic geometry

z

z

z

BV (W ) V y

y

y

x

x

x

W

W-V

W-V Fig. 3.2. Blow-up of a real algebraic set

z

Exceptional set π (V ) ={x = y = 0}

BV (R2) Strict transform BV (W )

y V x Fig. 3.3. Blow-up of V(x 3 − y 2 )

On the set A, x 3 − y 2 = x 2 (x − z2 ). By using x = 0, the set A is equal to A = {(x, y, z) ; x = z2 , y = zx, (x, y) = (0, 0)}. Lastly, its closure is BV (W ): BV (W ) = A = {(x, y, z) ; x = z2 , y = zx}. This set is a strict transform. In this example, W contains singularities, whereas BV (W ) does not contain a singularity. The real algebraic set W which contains a singularity is the image of a nonsingular real algebraic set BV (W ). Such BV (W ) is called a resolution of singularity of W . Figure 3.3 shows the strict

3.5 Blow-up

95

transform, the exceptional set, and the total transform. The projection is defined by π : BV (W ) → W. The exceptional set is π −1 ({(0, 0)}) = {(0, 0)} × P1 , and the total transform is π −1 (W ) = BV (W ) ∪ π −1 ({(0, 0)}). Remark 3.16 Let us study the blow-up from an algebraic point of view. (1) The blow-up of R2 with center V = V(x, y) is equivalent to the substitution x = u = st, y = uv = s. The blow-up BV (R2 ) is made by gluing two coordinates (u, v) and (s, t). (2) In the same way, the blow-up of R3 with center V = V(x, y, z) is equivalent to substitution x = x1 = x2 y2 = x3 z3 , y = y1 x1 = y2 = y3 z3 , z = z1 x1 = z2 y2 = z3 . The blow-up BV (R3 ) is made by gluing three coordinates. (3) The blow-up of R3 with center V = V(x, y) is equivalent to the substitution x = x1 = x2 y2 , y = y1 x1 = y2 , z = z 1 = z2 . The blow-up BV (R3 ) is made by gluing two coordinates. Example 3.14 Figure 3.4 shows a blow-up of a real algebraic set V V = V(x 4 − x 2 y + y 3 ) in R2 . The singularity of V is the origin O. The blow-up g : P 2 → R2 with center O can be represented by using local coordinates x = u = st, y = uw = s.

96

Algebraic geometry

w x=u y = uw

y O

BO(W )

Exceptional line

x

O

u

s x = st y=t

BO(W)

Exceptional line

O

t

Fig. 3.4. Blow-up using local coordinates

The projective variety g −1 (V ) is represented on each local coordinate, u3 (u − w + w 3 ) = 0, s 3 (st 4 − t 2 + 1) = 0, whose singularities are all normal crossing. The blow-up BO (V ) is given by u − w + w 3 = 0, st 4 − t 2 + 1 = 0, which is a nonsingular real algebraic set. Definition 3.14 (General blow-up in Euclidean space) Let Rd be a real Euclidean space and r be an integer which satisfies 2 ≤ r ≤ d. Assume that both V and W (V ⊂ W ) are real algebraic sets in Rd . Let f1 , f2 , . . . , fr be a set of polynomials which satisfy I(V ) = f1 , f2 , . . . , fr . The blow-up of W with center V , BV (W ), is defined by BV (W ) ≡ {(x, (f1 : f2 : · · · : fr )); x ∈ W \ V }. If V does not contain a singularity, then BV (Rd ) is an analytic manifold. Remark 3.17 (1) Let V and W be real projective varieties which satisfy V ⊂ W ⊂ P d . In each local coordinate U0 , U1 , . . . , Ud of P d , real algebraic sets are given by Vi = Ui ∩ V , Wi = Ui ∩ W.

3.5 Blow-up

97

Nonsingular Y

Singularities of X Algebraic set X Fig. 3.5. Resolution of singularities

Then for each 0 ≤ i ≤ d, the blow-up of Wi with center Vi is defined by BVi (Wi ), which is a real projective variety. The blow-up of projective variety W with center V is defined by gluing all real projective varieties {BVi (Wi )}. Such a set made by gluing real projective varieties is called a real algebraic variety. A real algebraic set and a real projective variety are special examples of a real algebraic variety. The blow-up of a real algebraic variety can be defined in the same way, and is also a real algebraic variety. Hence a real algebraic variety is a closed concept by blow-ups. (2) There is a more abstract definition of algebraic variety. In modern mathematics, a property of a set is determined by the family of all functions with the desired property on the set. By using this relation, a set is sometimes defined by the family of all functions on the set. Now, let us introduce two processes of desingularization. The first is the following theorem. Theorem 3.5 (Hironaka’s theorem, I) For an arbitrary real algebraic set V , there is a sequence of real algebraic varieties V0 , V1 , V2 , . . . , Vn which satisfies the following conditions. (1) V = V0 . (2) Vn is nonsingular. (3) For i = 1, 2, . . . , n, Vi = BCi−1 (Vi−1 ), where Vi is a blow-up of Vi−1 with center Ci . (4) Ci is a nonsingular real algebraic variety which is contained in Sing(Vi−1 ). (Explanation of Theorem 3.5) This theorem is proved by Hironaka [40]. By this theorem, any real algebraic set can be understood as an image of a nonsingular real algebraic variety. Figure 3.5 shows the result of this theorem.

98

Algebraic geometry

Recursive blow-ups without exceptional sets Exceptional lines

Recursive blow-ups with exceptional sets

Fig. 3.6. Two processes of desingularization

Theorem 3.6 (Hironaka’s theorem, II) Let f (x) be an arbitrary polynomial in R[x1 , x2 , . . . , xd ]. There is a sequence of pairs of real algebraic varieties (V0 , W0 ), (V1 , W1 ), . . . , (Vn , Wn ) which satisfies the following conditions. (1) Vi ⊂ Wi (i = 1, 2, . . . , n). (2) V0 = V(f ), W0 = Rd . (3) {Wi ; i = 0, 1, 2, . . . , n} are nonsingular algebraic varieties. (4) Vn is defined by a normal crossing polynomial on each local coordinate of Wn . (5) For i = 1, 2, . . . , n, Wi = BCi−1 (Wi−1 ), where Wi is a blow-up of Wi−1 with center Ci . (6) Let πi : Wi → Wi−1 be a projection map defined in the blow-up BCi−1 (Wi−1 ). Then Vi is the total transform of πi , Vi = πi−1 (Vi−1 ). (7) The center Ci of each blow-up is a nonsingular real algebraic variety which is contained in the set of critical points of f ◦ π1 ◦ π2 ◦ · · · ◦ πi . (Explanation of Theorem 3.6) This theorem is also proved by Hironaka [40]. This theorem shows that there exists a recursive procedure by which the resolution of singularities in Theorem 2.3 is attained. Note that Theorem 3.5 does not contain exceptional sets in the blow-ups, whereas this theorem does. In Theorem 3.5, any real algebraic set can be made to be an image of a nonsingular real algebraic variety because exceptional sets are removed. In Theorem 3.6, since exceptional sets are contained, any singularities of a real algebraic set are images of normal crossing singularities. Figure 3.6 shows the difference between two resolutions of singularities. See also [19, 37].

3.6 Examples

99

Remark 3.18 In statistical learning theory, Theorem 2.3 is needed. The map g in Theorem 2.3 can be algorithmically found by Theorem 3.6. An arbitrary polynomial can be made normal crossing by recursive blow-ups with the center of nonsingular sets in a singular locus of a previous algebraic variety. It is not easy to find such a process; however, in some practical statistical models, a concrete resolution map was found, which is introduced in Chapter 7.

3.6 Examples 3.6.1 Simple cases Example 3.15 The blow-up of R2 with center O = {(0, 0)} is M = BO (R2 ) = U1 ∪ U2 , where each local coordinate is given by U1 = {(x1 , y1 )}, U2 = {(x2 , y2 )}, which satisfy the relations x = x1 = x2 y2 , y = x1 y1 = y2 . Therefore the resolution map has the form,  R2 ← M =

U1 U2

The function f (x, y) = x 2 + y 2 is represented on each coordinate, ! f = f (x1 , x1 y1 ) = x12 1 + y12 , ! = f (x2 y2 , y2 ) = y22 x22 + 1 . The function f is not normal crossing on R2 , but it is on M. Example 3.16 Using the same coordinates U1 and U2 as above, the function f (x, y) = x 3 − y 2 is represented on BO (R2 ), ! f = f (x1 , x1 y1 ) = x12 x1 − y12 , hence f is not normal crossing on U1 . On the other hand, on U2 , ! f = f (x2 y2 , y2 ) = y22 x23 y2 − 1

100

Algebraic geometry

is normal crossing because x22 y23 − 1 = 0 does not have singularities. The recursive blow-ups are as follows. For U1 , x1 = x3 = x4 y4 , y1 = x3 y3 = y4 . Then, on U3 f (x, y) = x33 1 − x3 y32

!

is normal crossing. But on U4 , f = x42 y43 (x4 − y4 ) is not normal crossing. One more blow-up is needed: x4 = x5 = x6 y6 , y4 = x5 y5 = y6 . Then f = x56 y54 1 − y52

!

= x62 y66 (x6 − 1), which shows f is normal crossing on both U5 and U6 . The obtained sequence of blow-ups is    U3    ← U U5 2 1 R ← U ←  4 U  6  U2 Note that the manifold M made of local coordinates M = U2 ∪ U3 ∪ U5 ∪ U6 is a real analytic manifold; in other words, it does not have singularities. The map g : M → R2 is defined on each local coordinate, x = x2 y2 = x3 = x52 y52 = x6 y62 , y = y2 = x32 y3 = x53 y52 = x6 y63 , which makes f normal crossing, ! ! ! f = y22 x23 y2 − 1 = x33 1 − x3 y32 = x56 y54 1 − y52 = x62 y66 (x6 − 1),

3.6 Examples

101

z1 z

y1

z2

x1 y

y2 x2

z3

x

y3

x3 Fig. 3.7. Example of blow-up

and the Jacobian determinant is

      |g  | = |y2 | = x32  = 2x44 y43  = x6 y64 .

Example 3.17 In R3 , let us study the function in Figure 3.7: f = x 2 + y 2 − z2 . The blow-up with center O = {(0, 0, 0)} is given by x = x1 = x2 y2 = x3 z3 , y = x1 y1 = y2 = y3 z3 , z = x1 z1 = z2 y2 = z3 . Then the function f on each coordinate is f = x12 1 + y12 − z12 = y22 x22 + 1 − z22

! !

! = z31 x32 + y32 − 1 , which is normal crossing on each coordinate.

3.6.2 A sample of a statistical model Let us study the resolution process of statistical models. Example 3.18 For a Kullback–Leibler distance,
1 K(a, b, c) = (as(bx) + cx)2 q(x)dx, 2

102

Algebraic geometry

where s(t) = t + t 2 and q(x) is the standard normal distribution. Then K(a, b, c) = 12 (ab + c)2 + 32 a 2 b4 . The following resolution of singularities is found by recursive blow-ups. Let U1 , U2 , U3 , U4 be local coordinates and M = U1 ∪ U2 ∪ U3 ∪ U4 . The resolution map g : M → R3 is given by a = a1 c1

b = b1

c = c1 ,

a = a2

b = b2 c2

a = a3

b = b3

c = a3 b3 (b3 c3 − 1),

a = a4

b = b4 c4

c = a4 b4 c4 (c4 − 1).

c = a2 (1 − b2 )c2 ,

Then K(g(u)) is made to be normal crossing,   2K(g(u)) = z12 (a1 b1 + 1)2 + 3a12 b14 ! = a22 c22 1 + 3b22 c22 ! = a32 b34 c32 + 3 ! = a42 b42 c44 1 + 3b42 . The Jacobian determinant g  (u) is also made to be normal crossing. g  (u) = c1 = a2 c2 = a3 b32 = a4 b4 c42 . Therefore, the real log canonical threshold is 3/4. Example 3.19 Let us study a function which is the Kullback–Leibler distance of a layered neural network, Example 7.1, f = f (a, b, c, d) = (ab + cd)2 + (ab2 + cd 2 )2 . (1) Firstly a blow-up with center V(b, d) ⊂ V(f ) is tried. By b = b1 d,   f = d 2 (ab1 + c)2 + d 2 (ab12 + c)2 . Since f is symmetric for (b, d), we need not try d = bd1 . (2) The transform c1 = ab1 + c is an analytic isomorphism and its Jacobian determinant is equal to 1.  !2  f = d 2 c12 + d 2 ab12 + c1 − ab1 .

3.6 Examples

103

(3) The second step is to try the blow-up with center V(c1 , d) ⊂ V(f ). In the first local coordinate, by d = c1 d1 , it follows that  !2  f = c14 d12 1 + d12 ab12 + c1 − ab1 , which is normal crossing. In the second local coordinate, by c1 = c2 d, it follows that  !2  f = d 4 c22 + ab12 + c2 d − ab1 , which is not normal crossing. (4) The third step is the blow-up with center V(a, c2 ) ⊂ V(f ). By a = c2 a1 , f is made normal crossing. By c2 = ac3 , it follows that  !2  f = d 4 a 2 c32 + b12 + c3 d − b1 , which is not yet normal crossing. (5) The fourth step is the blow-up with center V(b1 , c3 ) ⊂ V(f ). By b1 = c3 b2 , f is made normal crossing. By c3 = b1 c4 , it follows that   f = a 2 b12 d 4 c42 + (b1 + c4 d − 1)2 , (3.6) which is not yet normal crossing. (6) The last step is the blow-up with center V(b1 − 1, c4 ) ⊂ V(f ). The relation b1 − 1 = c4 b2 makes f normal crossing. Also c4 = c5 (b1 − 1) results in   f = d 4 a 2 b12 (b1 − 1)2 c52 + (1 + c5 d)2 , which is normal crossing. The last coordinate is given by a = a, b = b1 d, c = a(b1 − 1)b1 c5 d − ab1 , d = d, whose Jacobian determinant is given by |g  | = |ab1 (b1 − 1)d 2 |. The real log canonical threshold is 3/4. Remark 3.19 (1) In this example, the resolution process started from the blowup with center b = d = 0. If one starts from the blow-up with center a = c = 0 or a = d = 0, the other desingularization is found.

104

Algebraic geometry

(2) The concept of blow-up is generalized. A transformation defined by x1 = y1a11 y2a12 · · · yda1d , x2 = y1a21 y2a22 · · · yda2d , .. . xd = y1ad1 y2ad2 · · · ydadd , is called a toric modification if det(A) = 1 where A = (aij ). In this method, a toric variety is introduced, which is the generalized concept from the real algebraic variety. If the Newton diagram of the target function is not degenerate, then the resolution of singularities can be found by the toric modification. (3) In complex algebraic geometry, a log canonical threshold can be defined in the same way, that is to say, it is defined by using resolution of singularities. Even for the same polynomial, the complex resolution is different from the real 2 resolution. For example, f (x, y) = √ x(y + 1)√is normal crossing in real resolution, whereas f (x, y) = x(y + −1)(y − −1) is the complex resolution. Hence the real log canonical threshold is different from the complex one. As is shown in Chapter 6, the real log canonical threshold is equal to the learning coefficient, if the a priori distribution is positive at singularity.

4 Zeta function and singular integral

In singular learning theory, we need an asymptotic expansion of an integral for n → ∞,
Z(n) = exp(−nK(w))ϕ(w)dw, (4.1) where K(w) is a function of w ∈ Rd and ϕ(w) is a probability density function. If the minimum point of K(w) is unique and the Hessian matrix at the minimum point is positive definite, then the saddle point approximation or the Gaussian approximation can be applied. However, to study the case when K(w) = 0 contains singularities, we need a more precise mathematical foundation. An integral such as eq.(4.1) is called a singular integral. To analyze a singular integral, we need the zeta function, Schwartz distribution, Mellin transform, and resolution of singularities. For example, if K(a, b) = (a 3 − b2 )2 , then the function exp(−nK(a, b)) has the form shown in Figure 4.1. Note that the neighborhood of singularities of K(a, b) = 0 occupy almost all parts of the integral when n → ∞.

4.1 Schwartz distribution Definition 4.1 (Function space D) A function ϕ : Rd → C is said to be of class C0∞ if (1) ϕ(w) is a function of class C ∞ . (2) The support of ϕ(w), supp ϕ ≡ {w ∈ Rd ; ϕ(w) = 0}, 105

106

Zeta function and singular integral

1 0

Fig. 4.1. Singular integral

is compact, where A is the closure of a set A with Euclidean topology. The set of all functions of class C0∞ is denoted by D, which is a vector space over the field C. Definition 4.2 (Topology of D) Let {ϕi ; i = 1, 2, . . .} be a sequence of functions in D and ϕ be a function in D. The convergence ϕk → ϕ is defined by the following conditions. (1) There exists a compact set K ⊂ Rd such that supp ϕ ⊂ K, supp ϕk ⊂ K (k = 1, 2, 3, . . .). (2) For each multi-index α, lim max |∂α ϕk (w) − ∂α ϕ(w)| = 0.

k→∞ w∈K

Example 4.1 (1) A nonzero analytic function on Rd does not have a compact support, hence such a function is not contained in D. (2) Let a > 0. A function ρa (w) defined on Rd ,

 1  (w < a) exp − 2 ρa (w) = a − w2  0 (w ≥ a), is contained in D. (3) If f (w) is an analytic function, then f (w)ρa (w) is contained in D. If a → ∞, then f (w)ρa (w) → f (w) for each w. (4) Assume ϕ(w) ∈ D. If ak ∈ Rd satisfies |ak | → ∞ (k → ∞), then for each w ∈ Rd lim ϕ(w − ak ) = 0.

k→∞

4.1 Schwartz distribution

107

However, since the support of ϕ(w − ak ) is not contained in any compact set, it does not converge to ϕ(w) ≡ 0 by the topology of D. Definition 4.3 (Schwartz distribution) A function T : D → C is said to be a Schwartz distribution on Rd if it satisfies the following conditions. (1) T is a linear function from D to C. In other words, for each ϕ ∈ D, T (ϕ) is contained in C and for arbitrary a, b ∈ C and arbitrary ϕ(w), ψ(w) ∈ D T (aϕ + bψ) = aT (ϕ) + bT (ψ). (2) T is a continuous function from D to C. In other words, if a sequence ϕk is such that ϕk → ϕ by the topology of D, then the complex sequence T (ϕk ) converges to T (ϕ). The set of all Schwartz distributions is denoted by D . Example 4.2 (1) A function f : Rd → C is said to be locally integrable if, for an arbitrary compact set K, the integral
f (w)dw K

is well defined and finite, where dw is Lebesgue measure on Rd . If a function f (w) is locally integrable, then
T (ϕ) = f (w)ϕ(w)dw (ϕ ∈ D) satisfies the conditions of Definition 4.3, hence it determines a Schwartz distribution. If a Schwartz distribution T is represented by a locally integrable function in the same way, then T is called a Schwartz distribution with a regular integral. (2) A Schwartz distribution T (ϕ) = ϕ(0) satisfies the conditions of Definition 4.3, but it cannot be represented by any locally integrable function. Usually this Schwartz distribution is denoted by a formal integration,
T (ϕ) = δ(w)ϕ(w)dw. (4.2) Note that, in this equation, δ(w) on the right-hand side is defined by the lefthand side. Here δ(w) is not an ordinary function of w. Therefore, T is a Schwartz distribution but not a Schwartz distribution with a regular integral. This Schwartz distribution T and δ(w) is called a delta function or Dirac’s delta function.

108

Zeta function and singular integral

(3) For ϕ(x, y), a Schwartz distribution defined by
 ∂  T (ϕ) = dy ϕ(x, y) x=0 ∂x is not a Schwartz distribution with a regular integral. A Schwartz distribution
S(ϕ) = ϕ(0, y)dy

=

δ(x)ϕ(x, y)dxdy

is not a Schwartz distribution with a regular integral. Definition 4.4 (Topology of Schwartz distribution) Let T1 , T2 , . . . and T be Schwartz distributions in D . The convergence in D , Tk → T is defined by the condition that, for each ϕ ∈ D, the complex sequence Tk (ϕ) → T (ϕ). Note that, in order to prove the convergence of Schwartz distribution, it is sufficient to prove Tk (ϕ) → T (ϕ) for each ϕ. No uniform convergence for ϕ is needed. Theorem 4.1 (Completeness of D) Let T1 , T2 , . . . , be a sequence of Schwartz distributions. If the sequence Tk (ϕ) converges in C for each ϕ ∈ D, then there exists a Schwartz distribution T ∈ D such that Tk → T . (Explanation of Theorem 4.1) This is a fundamental theorem of Schwartz distribution theory. For the proof, see [32]. If {Tk (ϕ)} is a Cauchy sequence for an arbitrary ϕ ∈ D, then there exists a Schwarz distribution T such that Tk → T . Example 4.3 For a > 0, a function Sa on R1  1/(2a) |w| < a Sa (w) = 0 |w| ≥ a is locally integrable, hence Ta (ϕ) =


Sa (w)ϕ(w)dw

is a Schwartz distribution with a regular integral. When a → 0, a function Sa (w) does not converge to any ordinary function, whereas Ta converges to a delta function as a Schwartz distribution. Theorem 4.2 For any Schwartz distribution T , there exists a sequence of Schwartz distributions with regular integral {Tk } such that Tk → T .

4.1 Schwartz distribution

109

(Explanation of Theorem 4.2) This theorem shows that the set of Schwartz distributions with regular integral is a dense subset in D . For the proof, see [32]. Definition 4.5 (Derivative and integral on D ) Let Tt be a function from the real numbers to a Schwartz distribution, R  t → Tt ∈ D . Based on the topology of D , we can define a derivative and integral of Tt . For each ϕ ∈ D,  d  d Tt (ϕ) = Tt (ϕ) , dt dt and

 Tt dt (ϕ) = Tt (ϕ) dt. Example 4.4 By the identity,
∞

dt dxδ(t − x)ϕ(x) = 0

it follows that

∞

dxϕ(x) (ϕ ∈ D),

0




∞

dtδ(t − x) = θ (x),

0

where θ(x) is a locally integrable function by which a Schwartz distribution is defined with a regular integral,  1 (x > 0) θ (x) = (4.3) 0 otherwise. This function θ (x) is called a Heaviside function or a step function. On the other hand, from the identity
∞ d θ (t − x)ϕ(x)dx = ϕ(t), dt −∞ it follows that d θ (t − x) = δ(t − x). dt Example 4.5 (Definition of δ(t − f (x))) Let f : R1 → R1 be a differentiable function which satisfies df > 0. dx

110

Zeta function and singular integral

We assume that, for arbitrary t, there exists x such that f (x) = t. Let us study how to define a Schwartz distribution δ(t − f (x)) on x ∈ R1 so that it satisfies d δ(t − f (x)) = θ (t − f (x)), (4.4) dt for any parameter t ∈ R1 . By the definition, eq.(4.4) is equivalent to

d δ(t − f (x))ϕ(x)dx = θ (t − f (x))ϕ(x)dx. dt By the definition of eq.(4.3),

f −1 (t) d d θ (t − f (x))ϕ(x)dx. = ϕ(x)dx dt dt −∞ = ϕ(f −1 (t))f −1 (t)
ϕ(x) = δ(x − f −1 (t))  dx. f (x) Therefore, we adopt the definition δ(t − f (x)) =

δ(x − f −1 (t)) . |f  (x)|

Then it satisfies eq.(4.4) as a theorem. As a special case, we obtain the definition δ(f (x)) =

δ(x − x0 ) , |f  (x)|

where x0 = f −1 (0). Note that, if f  (x) = 0, then δ(f (x)) cannot be defined. Remark 4.1 A function of class C0∞ and a Schwartz distribution can be locally defined. (1) The following theorem shows that a C0∞ class function can be localized. By using the partition of unity, Theorem 2.10, a function ϕ ∈ D can be decomposed as ϕ(x) =

J 

ϕ(x) ρj (x).

j =1

Therefore, a Schwartz distribution can also be represented as a sum of localized ones, T (ϕ) =

J  j =1

T (ϕ · ρj ).

(4.5)

4.2 State density function

111

(2) By this decomposition, we can define a Schwartz distribution by its local behavior in any open set. Assume that a compact set K and its open covering K ⊂ ∪j Uj are given. Let C0∞ (Uj ) be the set of all C0∞ class functions whose supports are contained in Uj . If a set of Schwartz distributions {Tj } is given where Tj : C0∞ (Uj ) → R and if, for arbitrary ϕ ∈ C0∞ (Ui ∩ Uj ), the consistent condition Ti (ϕ) = Tj (ϕ) is satisfied, then there exists a unique Schwartz distribution T such that T (ϕ) =

J 

Tj (ϕ · σj ).

(4.6)

j =1

By using this property, a Schwartz distribution can be defined on each local open set. Also a Schwartz distribution on a manifold is defined in the same way on each local coordinate.

4.2 State density function As is shown in Example 4.5, if a real function of class C 1 , f : R1 → R1 satisfies f (x) = 0 ⇐⇒ x = x0 , f  (x0 ) = 0, then the Schwartz distribution δ(f (x)) is defined by δ(f (x)) =

δ(x − x0 ) . |f  (x0 )|

In this section, let us generalize this definition to the case of several variables. Let U be an open set of Rd and f : U → R1 be a real analytic function. Let us define a Schwartz distribution δ(t − f (x)). Assume that ∇f (x) = 0 for f (x) = t. For the definition of the Schwartz distribution we can assume U is suffi∂f ∂f ∂f (x), ∂x (x), . . . , ∂x (x) is ciently small and local such that at least one of ∂x 1 2 d not equal to zero. Therefore, we can assume ∂f (x) = 0 (x ∈ U ) ∂x1

112

Zeta function and singular integral

Tt + ε

f (x) = t + ε

f (x) = t Fig. 4.2. Definition of δ(t − f (x))

without loss of generality. Let us choose a coordinate (f, u2 , u3 , . . . , ud ) instead of (x1 , x2 , x3 , . . . , xd ), where x2 = u2 , x3 = u3 , . . . , xd = ud . Then dx1 dx2 · · · dxd = =

∂(x1 , x2 , x3 , . . . , xd ) df du2 du3 · · · dud ∂(f, u2 , u3 , . . . , ud ) df du2 du3 · · · dud . |df/dx1 |

Let ϕ(f, x2 , x3 , . . . , xd ) be a function of class C0∞ whose support is contained in U . A Schwartz distribution Dt on Rd is defined by
ϕ(t, u2 , u3 , . . . , ud ) Dt (ϕ) = (4.7)  ∂f  du2 du3 · · · dud .   (t, u , u , . . . , u )  2 3 d  ∂x1 Also we define another Schwartz distribution Tt in Figure 4.2 by
ϕ(x)dx Tt (ϕ) = f (x) 0 be positive real numbers. A function  λ−1 t (0 < t < a) f (t) = 0 otherwise has bounded variation at an arbitrary point.
a F (z) = t λ−1 t z dt 0

converges absolutely in −λ < Re(z) < ∞, and is equal to F (z) =

a z+λ . z+λ

Hence F (z) can be analytically continued to the meromorphic function on the entire complex plane, which is equal to a z+λ /(z + λ). The inverse transform of F (z) is given by
c+i∞ z+λ a 1 s −z−1 dz, f (s) = 2π i c−i∞ z + λ which is equal to

by the residue theorem.

  f (s) = a z+λ s −z−1 

z=−λ

= s λ−1

118

Zeta function and singular integral

Example 4.7 This example is very important in this book. Let λ > 0 and a > 0 be positive real numbers, and m be a natural number. The Mellin transform of   −λ  a m−1  a (0 < t < a) t λ−1 log t f (t) = (m − 1)!  0 otherwise is given by a −λ F (z) = (m − 1)!




a

t λ−1

 log

0

a m−1 z t dt. t

By using partial integration, F (z) =



a −λ a m−1 a t z+λ  log 0 (m − 1)! (z + λ) t
  a z+λ−1 t a −λ a m−2 + dt log (m − 2)! 0 (z + λ) t .. .

a −λ = (z + λ)m−1




a

t z+λ−1 dt. 0

This integral converges absolutely in −λ < Re(z) < ∞, and is equal to F (z) =

az . (z + λ)m

The inverse transform is given by f (s) =

  d m−1  a z 1  a −λ a m−1  = . s λ−1 log  dz s s z=−λ (m − 1)! s

4.4 Evaluation of singular integral Based on the resolution theorem, any singularities are images of normal crossing singularities. Any singular integral is equal to the sum of integrals of normal crossing singularities. However, evaluation of a singular integral contains a rather complicated calculation, hence we introduce an example before the general theory. Example 4.8 Let us consider a normal crossing function K(a, b) = a 2 b2 .

4.4 Evaluation of singular integral

119

Fig. 4.3. Normal crossing singular integral

Then the function exp(−nK(a, b)) in a singular integral is illustrated in Figure 4.3,
Z(n) = exp(−nK(a, b)) da db. [0,1]2

The zeta function


ζ (z) =

K(a, b)z da db

(4.12)

[0,1]2

is the Mellin transform of the state density function,
v(t) = δ(t − K(a, b)) da db.

(4.13)

[0,1]2

Moreover, the singular integral is the Laplace transform of the state density function.
1 exp(−nt) v(t) dt. (4.14) Z(n) = 0

Among the three integrals, eqs.(4.12), (4.13), (4.14), the zeta function can be calculated explicitly for Re(z) > −1/2, ζ (z) =

1 . 4(z + 1/2)2

For Re(z) > −1/2, this function is holomorphic. It can be analytically continued to the meromorphic function on the entire complex plane. By using the inverse Mellin transform, the state density function is given by v(t) = 14 t −1/2 (−log t),

120

Zeta function and singular integral

if 0 < t < 1, or v(t) = 0 otherwise. Therefore
1 Z(n) = v(t)e−nt dt


0 n

=

v 0

 t  dt e−t n n

log n 1 = C1 √ − C2 + C3 (n), 4n 4 n where C1 and C2 are positive constants defined by
∞ C1 = t −1/2 e−t dt,


0 ∞

C2 =

t −1/2 e−t log t dt,

0

and C3 (n)/n converges to zero when n → ∞. Therefore, the largest order of √ the singular integral Z(n) is equal to log n/ n. Any singular integral can be asymptotically expanded in the same way as this example. We need to generalize the previous example. The following equations give constructive evaluation of the singular integral. Let r be a natural number. For a vector x = (x1 , x2 , . . . , xr ) ∈ Rr and multi-indices, k = (k1 , k2 , . . . , kr ), h = (h1 , h2 , . . . , hr ), x 2k and x h are monomials defined by x 2k = x12k1 x22k2 · · · xr2kr , x h = x1h1 x2h2 · · · xrhr . Also we define |k| = k1 + k2 + · · · + kr . For a given function f : Rr → C, the integral in [0, b]r is denoted by

b
b f (x)dx = dx1 · · · dxr f (x1 , . . . , xr ). [0,b]r

0

0

4.4 Evaluation of singular integral

121

Theorem 4.6 Let k1 , . . . , kr be natural numbers and h1 , . . . , hr be nonnegative integers. Assume that there exists a rational number λ > 0 such that h2 + 1 hr + 1 h1 + 1 = = ··· = = λ. 2k1 2k2 2kr Then for arbitrary real number a, b > 0, the state density function
δ(t − a x 2k ) x h dx v(t) = [0,b]r

is equal to v(t) =

  

γb

t λ−1  ab2|k| r−1 log λ a t 0

(0 < t < ab2|k| )

(4.15)

otherwise,

where γb > 0 is a constant γb =

2r

b|h|+r−2|k|λ . (r − 1)! k1 k2 · · · kr

(4.16)

Proof of Theorem 4.6 The Mellin transform of v(t) for the case b = 1 is given by
ζ (z) = (a x 2k )z x h dx [0,1]r

=

az 1 . (2k1 )(2k2 ) · · · (2kr ) (z + λ)r

By using the result of Example 4.7, if 0 < t < a,
t λ−1  a r−1 δ(t − a x 2k ) x h dx = γ1 λ ; log a t [0,1]r otherwise v(t) = 0. Then by putting x  = bx and a  = ab−2|k| , we obtain the
theorem. Corollary 4.3 Let θ (t) be a step function defined by θ (t) = 1 (t ≥ 0) or θ(t) = 0 (t < 0). Assume the same condition as Theorem 4.6. Then
r  γb (r − 1)! λ  b2|k| r−m θ (t − x 2k ) x h dx = . (4.17) t log λm (r − m)! t [0,b]r m=1 Proof of Corollary 4.3 Let V (t) be the left-hand side of eq.(4.17), then V (t) is equal to v(t) in Theorem 4.6 with a = 1,

r−1 b2|k|  λ−1 V (t) = γb t . log t

122

Zeta function and singular integral

Since V (0) = 0,


T

V (T ) = 0

r−1 b2|k| γb t λ−1 log dt. t

By using recursive partial integration such as

tλ V (T ) = γb λ



b2|k| log t

r−1 

T 0

γb (r − 1) + λ






T

t

λ−1

0

b2|k| log t

r−2 dt,




we complete the corollary.

Theorem 4.7 Let r and s be natural numbers. Assume that four multi-indices k, k  , h, h satisfy h2 + 1 hr + 1 h1 + 1 = = ··· = =λ 2k1 2k2 2kr and that hj + 1 2kj

>λ

(j = 1, 2, . . . , s).

A singular integral Z p (n) is defined by
Z (n) = p




[0,b]r



dy K(x, y)p exp(−nβK(x, y)2 ) x h y h ,

dx

(4.18)

[0,b]s

where p ≥ 0, β > 0 and 

K(x, y) = x k y k . Then there exist constants a1 , a2 > 0 such that, for arbitrary natural number n > 1, a1

(log n)r−1 (log n)r−1 ≤ Z p (n) ≤ a2 . λ+p n nλ+p

4.4 Evaluation of singular integral

123 

Proof of Theorem 4.7. Firstly we prove the case b = 1. Since y 2k ≤ 1 in [0, 1]s ,

  Z p (n) ≥ dx dy exp(−nβx 2k ) x h+kp y h +k p [0,1]r

= a1 = a1 = a1 ≥ a1 = a1

[0,1]s




dx exp(−nβx 2k ) x h+kp [0,1]r
∞




dt [0,1]r

0





dt

t λ−1  n r−1 p −βt t e log nλ+p t

dt

t λ−1 (log n)r−1 t p e−βt nλ+p

n 0 1 0

dx δ(t − nx 2k ) t p e−βt x h

(log n)r−1 , nλ+p

where we used Theorem 4.6, − log t > 0 for 0 < t < 1, and a1 and a1 are constants. On the other hand, by using Theorem 4.6,

∞
  dt e−βt t p dx dy δ(t − nx 2k y 2k ) x h y h Z p (n) = 0




[0,1]r




∞

[0,1]s

λ−1

c0 t λ+p n (y 2k )λ [0,1]s 0


r−1  r − 1 (log n)m ∞ ≤ dt nλ+p m 0 m=0 ≤

dt

dy

  ny 2k r−1 h p −βt  y t e log  t
dy

[0,1]s

  y 2k r−1−m h −2λk  × c0 t λ−1 e−βt log y .  t

The last integration is finite because λ > 0 and h − 2λk  > −1. Therefore we have proved the theorem for the case b = 1. Secondly, we study the case for general b > 0. By putting bx  = x, by  = y, in eq.(4.18), we have

 dx  dy  K p exp(−nβB2 K 2 ) (x  )h (y  )h , Z p (n) = B1 [0,1]r

[0,1]s







where B1 = br+s+|h|+|h |+|k|p+|k |p > 0, B2 = b2|k|+2|k | > 0, K(x  , y  ). Therefore we obtain the theorem.

and

K =




Definition 4.10 (Partition function) Let ξ and ϕ be functions of C 1 class from [0, b]r+s to R. Assume that ϕ(x, y) > 0, (x, y) ∈ [0, b)r+s . The partition

124

Zeta function and singular integral

function of ξ , ϕ, n > 1, and p ≥ 0, is defined by

 dx dy K(x, y)p x h y h ϕ(x, y) Z p (n, ξ, ϕ) = [0,b]r

[0,b]s

× exp(−nβ K(x, y)2 +

√ nβ K(x, y) ξ (x, y)),

where 

K(x, y) = x k y k . We use the definitions ξ  = and

max

(x,y)∈[0,b]r+s

|ξ (x, y)|

 ∂ξ     . 1≤j ≤r (x,y)∈[0,b]r+s ∂xj

∇ξ  = max

max

Theorem 4.8 Assume the same condition as Theorem 4.7 for k, k  , h, h . Then there exist constants a1 , a2 > 0 such that, for arbitrary ξ and ϕ (ϕ(x) > 0 ∈ [0, b]d ) and an arbitrary natural number n > 1, a1

(log n)r−1 −βξ 2 /2 (log n)r−1 βξ 2 /2 e min ϕ ≤ Z p (n, ξ, ϕ) ≤ a2 e ϕ λ+p n nλ+p

holds, where min ϕ = min ϕ(x). x∈[0,b]d

Proof of Theorem 4.8. By using the Cauchy–Schwarz inequality, √ 1 | n K(x, y) ξ (x, y)| ≤ {n K(x, y)2 + ξ 2 }, 2 hence Zp

 3n  2



βξ 2 exp − min ϕ ≤ Z p (n, ξ, ϕ) 2

n βξ 2 ≤ Zp exp ϕ, 2 2

where Z p (n) is given by eq.(4.18). By appying Theorem 4.7, we obtain the
theorem. Theorem 4.9 Assume the same condition as Theorem 4.7 for k, k  , h, h . Let ξ and ϕ be functions of class C 1 . The central part of the partition function is

4.4 Evaluation of singular integral

125

defined by Y p (n, ξ, ϕ) ≡




γb (log n)r−1 nλ+p




∞

dy t λ+p−1 y µ e−βt+β

dt

√

tξ0 (y)

ϕ0 (y),

[0,b]s

0

(4.19) where we use the notation, ξ0 (y) = ξ (0, y), ϕ0 (y) = ϕ(0, y), µ = h − 2λk  . Then there exist a constant c1 > 0 such that, for arbitrary n > 1, ξ , ϕ, and p ≥ 0, |Z p (n, ξ, ϕ) − Y p (n, ξ, ϕ)| ≤ c1

(log n)r−2 βξ 2 /2 {β∇ξ ϕ + ∇ϕ + ϕ}. e nλ+p

Proof of Theorem 4.9 Firstly, we prove that |Z p (n, ξ0 , ϕ0 ) − Y p (n, ξ, ϕ)| ≤ c1 By the definition of Z p (n, ξ0 , ϕ0 ),
∞
Z p (n, ξ0 , ϕ0 ) = dt 0

× tp e

(log n)r−2 βξ0 2 /2 e ϕ0 . nλ+p

(4.20)


dy δ(t − nK(x, y))

dx

[0,b]r [0,b]s √ −βt+β t ξ0 (y) h h

x y ϕ0 (y).

By using Theorem 4.6, γb Z (n, ξ0 , ϕ0 ) = λ+p n




p

dt dy t λ+p−1 e−βt+β

√

t ξ0 (y)

[0,b]∗

  ny 2k b2|k| r−1 × y µ log ϕ0 (y), t

where the integrated region is 

[0, b]∗ = {(t, y) ; 0 < t < ny 2k b2|k| , y ∈ [0, b]s }. 

By expanding {log n + log(y 2 b2|k| /t)}r−1 , Z p (n, ξ0 , ϕ0 ) is decomposed as Z p (n, ξ0 , ϕ0 ) = Y p (n, ξ, ϕ) + Z1 + Z2

126

Zeta function and singular integral

where Z1 = −γb

(log n)r−1 nλ+p




dtdy t λ+p−1 y µ e−βt+β

√

tξ0 (y)

[0,b]∗∗

ϕ0 (y)

with 

[0, b]∗∗ = {(t, y) ; ny 2k b2|k| ≤ t, y ∈ [0, b]s }, and Z2 =


r−2  r − 1 c0 (log n)m dt dy t λ+p−1 λ+p ∗ n m [0,b] m=0 × e−βt+β

√

t ξ0 (y)

  y 2k r−1−m y µ log ϕ0 (y). t

Therefore Z1 is evaluated as follows: (log n)r−1 βξ0 2 /2 e ϕ0  nλ+p
∞
t   × t λ+p−1 e−βt dt dy y µ θ − y 2k b2|k| n [0,b]s 0

|Z1 | ≤ γb

By using eq.(4.17), there exist c1 > 0, δ > 0 such that
t   t δ  nb2|k| s−1  dy y µ θ , − y 2k b2|k| ≤ c1 log n n t [0,b]s it follows that |Z1 | ≤ c1

(log n)r−1 βξ0 2 /2 e ϕ0 . nλ+p+δ

Also the term Z2 can be evaluated |Z2 | ≤ c1

(log n)r−2 βξ0 2 /2 e ϕ0 . nλ

Hence we obtain eq.(4.20). Secondly, let us prove (log n)r−2 βξ 2 /2 e (β∇ξ ϕ + ∇ϕ). nλ+p (4.21) A function H = H (z, y) is defined by |Z p (n, ξ, ϕ) − Z p (n, ξ0 , ϕ0 )| ≤ a2

H (z, y) = eβ

√

n K(x,y) ξ (z,y)

ϕ(z, y).

4.4 Evaluation of singular integral

127

Let Z be the left-hand side of eq.(4.21) and K = K(x, y). Then

2  Z ≤ K p e−nβK x h y h |H (x, y) − H (0, y)| dxdy. There exists z∗ ∈ [0, b]r such that |H (x, y) − H (0, y)| ≤

≤

 ∂    xj  H (z∗ , y) ∂z j j =1

r 

r 

√ xj {β nK∇ξ ϕ + ∇ϕ}.

j =1

Therefore, by using Theorem 4.7, eq.(4.21) is obtained. Finally, by combining eq.(4.20) with eq.(4.21), the proof of Theorem 4.9 is completed.
Theorem 4.10 Assume the same condition as Theorem 4.9. Let Y p (n, ξ, ϕ) be the central part of the partition function in eq.(4.19) in Theorem 4.9. Then there exist constants a3 , a4 > 0 such that, for arbitrary ξ , ϕ, n > 1, and p ≥ 0, a3

(log n)r−1 −βξ 2 /2 (log n)r−1 βξ 2 /2 p e min |ϕ| ≤ Y (n, ξ, ϕ) ≤ a e ϕ 4 nλ+p nλ+p

holds. Proof of Theorem 4.10. This theorem is proved in the same way as Theorem
4.8. Remark 4.4 In statistical learning theory, we need to evaluate the asymptotic behavior of a general singular integral
 Z = exp(−nβK(w) + β nK(w)ξ (w))ϕ(w)dw, when n → ∞. By applying resolution of singularities to K(w), which makes K(gα (u)) normal crossing on each local coordinate Uα , 

K(gα ) = x 2k y 2k , the term Z can be written as a finite sum of the partition functions,  Z= Z(n, ξ ◦ gα , ϕ ◦ gα |gα |), α

to which theorems in this section can be applied.

128

Zeta function and singular integral

4.5 Asymptotic expansion and b-function In the previous section, the singular integral was studied from the viewpoint of resolution of singularities. In this section, its asymptotic expansion is analyzed without resolution of singularities. Let K(w) ≥ 0 be a real analytic function on an open set U ⊂ Rd and ϕ(w) be a function of class C0∞ whose support is contained in U . The zeta function of K(w) and ϕ(w) is defined by
(4.22) ζ (z) = K(w)z ϕ(w)dw. For Re(z) > 0, ζ (z) is a holomorphic function. In fact, for z0 = a + bi (a > 0), ζ (z) is differentiable as a complex function because, if z → z0 , there exists z∗ such that  ζ (z) − ζ (z ) 
∗ 0    ≤ |K(w)z log K(w)|ϕ(w)dw.  z − z0 The analytic continuation of ζ (z) can be obtained from the following theorem. Theorem 4.11 (b-function, Bernstein–Sato polynomial) Let K(w) be a real analytic function on an open set U in Rd . Then there exist a differential operator D(z, w, ∂w ) and a polynomial b(z) of z such that D(z, w, ∂w )K(w)z+1 = b(z)K(w)z

(w ∈ U, z ∈ C),

(4.23)

where D(z, w, ∂w ) and b(z) satisfies the following conditions. (1) D(z, w, ∂w ) is a finite sum  ∂α aα (w)bα (z) α D(z, w, ∂w ) = ∂w α using real analytic functions aα (w) and polynomials bα (z). (2) The smallest-order function among all functions b(z) that satisfy eq.(4.23) is uniquely determined if the coefficient of the largest order is 1. This polynomial is called the b-function or Bernstein-Sato polynomial. (3) The roots of b(z) = 0 are real, negative, and rational numbers. (Explanation of Theorem 4.11) This theorem was proved by Bernstein [16], Sato [80], Kashiwara [46], and Bj¨ork [17]. In the proof that all roots of the b-function are rational numbers, resolution of singularities is employed. The algorithm and software for calculating the b-function for a given polynomial were realized by Oaku [67, 68] and Takayama [89]. In the analysis of the b-function, D-module theory plays the central role. From a historical point of view, in 1954, Gel’fand conjectured that the Schwartz distribution K(w)z can

4.5 Asymptotic expansion and b-function

129

Im(z) ζ(z) by analytic continuation O

Pole k order mk

Pole 1 order m1

Re(z)

ζ(z) by integral

Fig. 4.4. Analytic continuation of zeta function

be analytically continued to the entire complex plane. The first answer to the conjecture was given by Atiyah using resolution of singularities [14]. Without resolution theorem, Bernstein and Sato created the b-function. Let D ∗ (z, w, ∂w ) be the adjoint operator of D(z, w, ∂w ),  ∂α D ∗ (z, w, ∂w )ϕ(w) = (−1)|α| bα (z) α {aα (w)ϕ(w)}. ∂w α Then





ψ(w)(Dϕ(w))dw =

(D ∗ ψ(w))ϕ(w)dw

for arbitrary ψ(w), ϕ(w) ∈ C0∞ . By using the existence of the b-function,
ζ (z) = K(w)z ϕ(w)dw =

1 b(z)

1 = b(z)


D(z, w, ∂w )K(w)z+1 ϕ(w)dw


K(w)z+1 D ∗ (z, w, ∂w )ϕ(w)dw.

(4.24)

Therefore the zeta function ζ (z) can be analytically continued to Re(z) > −1 by eq.(4.24). By recursively using this procedure illustrated in Figure 4.4, the zeta function can be analytically continued to the meromorphic function on the entire complex plane. All poles of the zeta function are real, negative, and rational numbers. Example 4.9 (1) For x 2 , d 2 2 z+1 (x ) = (2z + 2)(2z + 1)(x 2 )z dx 2

130

Zeta function and singular integral

shows that its b-function is (z + 1)(z + 1/2). The largest root of the b-function is −1/2. (2) For x j y k , where j and k are positive integers, the largest root of the bfunction is − min(1/j, 1/k). (3) For x 2 + y 2 + z2 , the b-function is (z + 1)(z + 3/2), since  ∂2 ∂2 ∂2  2 + + (x + y 2 + w 2 )z+1 = (z + 1)(4z + 6)(x 2 + y 2 + w2 )z . ∂x 2 ∂y 2 ∂w 2 Remark 4.5 There are some remarks on zeta functions. (1) A holomorphic function
1 x z dx (Re(z) > −1) ζ (z) = 0

is analytically continued to the meromorphic function 1/(z + 1) on the entire complex plane; however, the integration
1 x z dx 0

is not a finite value if Re(z) ≤ −1. In other words,
1 x z dx (Re(z) ≤ −1). ζ (z) = 0

(2) Assume that a function ϕ(x) is of class C ∞ . The analytic continuation of a Schwartz distribution Tz defined by
1 Tz (ϕ) = x z ϕ(x)dx 0

can be found:


ζ (z) =

1

x z (ϕ(0) + ϕ  (0)x +

0

ϕ  (0)x 2 + · · · )dx 2

ϕ  (0) ϕ  (0) ϕ(0) + + + ··· . = z + 1 z + 2 2(z + 3) This relation shows that the asymptotic expansion of a Schwartz distribution is Tz ∼ =

∞  k=0

δ (k) (x) . k!(z + k + 1)

By the existence of the b-function, the zeta function defined by eq.(4.22) can be understood as the meromorphic function on the entire complex plane. Let its poles be 0 > −λ1 > −λ2 > · · ·

4.5 Asymptotic expansion and b-function

131

and the order of (−λk ) be mk . Then
ζ (z) = K(w)z ϕ(w)dw has Laurent series expansion ζ (z) =

∞  k=1

Dk (ϕ) , (z + λk )mk

where {Dk } is a set of Schwartz distributions. By using an inverse Mellin transform, the state density function
v(t) = δ(t − K(w))ϕ(w)dw has an asymptotic expansion, v(t) =

mk ∞  

Dkm (ϕ)t λk −1 (− log t)m−1 ,

k=1 m=1

where {Dkm } is a set of Schwartz distributions. Hence
Z(n) = exp(−nK(w))ϕ(w)dw


M

=

exp(−nt)v(t)dt, 0

where M = maxw K(w), has also an asymptotic expansion
Mn mk ∞    1 n m−1 λk −1 −t Z(n) = D (ϕ) t e dt. log km nλk t 0 k=1 m=1 Remark 4.6 In this chapter we have shown the asymptotic expansion of
Z(n) = exp(−nK(w))ϕ(w)dw. The largest term of Z(n) is (log n)m1 −1 /nλ−1 . In statistical learning theory, we need the probability distribution of
Zn = exp(−nKn (w))ϕ(w)dw. In the following chapter, we prove that (log n)m1 −1 ∗ Zn nλ where Zn∗ is a random variable which converges in law to a random variable. Zn =

132

Zeta function and singular integral

Remark 4.7 (Riemann zeta function) In modern mathematics, a lot of zeta functions play important roles in many mathematical research fields. The original concept of many zeta functions is the Riemann zeta function ζ (z) = which is equal to ζ (z) =

1 (z)

∞  1 , nz n=1


0

∞

x z−1 dx. ex − 1

This function is holomorphic in Re(z) > 1 and analytically continued to the meromorphic function of the entire complex plane by the relation, z 1 − z 1 1   ζ (z) = ζ (1 − z), π z/2 2 π (1−z)/2 2 where (z) is the gamma function that satisfies (z + 1) = z(z). The Riemann hypothesis states that all roots of ζ (z) in 0 < Re(z) < 1 are on the line Re(z) = 1/2. The Riemann hypothesis is one of the most important conjectures in mathematics. In this book we show that the zeta function of a statistical model determines the asymptotic learning efficiency. The Riemann zeta function determines the asymptotic distribution of the prime numbers.

5 Empirical processes

In singular statistical models, the set of true parameters is not one point but a real analytic set with singularities. In conventional statistical learning theory, asymptotic normality is proved by applying central limit theorem to the log likelihood function in the neighborhood of the true parameter, whereas in singular learning theory, the main formulas are proved by applying empirical process theory to that in the neighborhood of the true analytic set with singularities.

5.1 Convergence in law Definition 5.1 Let (
, B) be a measurable space. (1) Let {Pn } and P respectively be a sequence of probability distributions and a probability distribution on (
, B). It is said that {Pn } converges to P in law, or {Pn } weakly converges to P , if, for any bounded and continuous function f :
→ R1 ,

f (x)P (dx) = lim f (x)Pn (dx). (5.1) n→∞

(2) Let {Xn } be a sequence of random variables and X be a random variable which take values on the measurable space (
, B). It is said that {Xn } converges to X in law, or {Xn } weakly converges to X, if PXn converges to PX in law, or equivalently, if, for any bounded and continuous function f , EX [f (X)] = lim EXn [f (Xn )]. n→∞

Remark 5.1 (1) It is well known that if, for any bounded and uniformly continuous function f , EXn [f (Xn )] = EX [f (X)], 133

134

Empirical processes

then Xn converges to X in law. In other words, to prove the convergence of {Xn } in law, we can restrict f as a uniformly bounded function. Here a function f is said to be uniformly continuous if, for a given  > 0, there exists δ > 0 such that |x − y| < δ =⇒ |f (x) − f (y)| < .

(5.2)

Note that δ does not depend on x, y. (2) Even if Xn and Yn converge in law to X and Y respectively, Xn + Yn may not converge in law in general. For example, let X be an R1 -valued random variable which is subject to the normal distribution with mean 0 and variance 1. Then Y = −X is subject to the same probability distribution as X and both Xn = X and Yn = (−1)n X converge to X in law. However, Xn + Yn does not converge in law. (3) In the definition of convergence in law, random variables X1 , X2 , . . . , Xn , . . . and X have the same image space (
, B). They may be defined on different probability spaces. In other words, in the definition, Xn :
n →
, the space
n may depend on n. Even if {Xn } are defined on different spaces, the probability distributions of {Xn } are defined on the common measurable set
. (4) If all random variables X1 , X2 , . . . , Xn , . . . , and X are defined on the same probability space, and if Xn → X in probability, then Xn → X in law. In fact, for any bounded and uniformly continuous function f which satisfies eq.(5.2) |EX [f (X)] − EXn [f (Xn )]| ≤ |E[f (X)] − E[f (Xn )]|{|X−Xn | 0 and then taking n sufficiently large, we obtain convergence in law. Example 5.1 Let Xn : R1 → R1 be a random variable which is subject to a probability distribution  1  1 δ(x − i) dx, Cn i=1 2i n

pn (x)dx =

5.1 Convergence in law

135

where Cn is a normalizing constant. Then Xn converges in law because
f (x)pn (x)dx =

∞ n  f (i) 1  f (i) → . Cn i=1 2i 2i i=1

On the other hand, a random variable which is subject to pn (x)dx = δ(x − n)dx does not converge in law, because
sin(x)pn (x)dx = sin(n) does not converge for a bounded and continuous function sin(x). Note that convergence in law is a mathematically different concept from the topology of a Schwartz distribution. In fact, sin(x) is not contained in C0∞ . As a Schwartz distribution, pn (x) → 0 holds. There are several elemental theorems for convergence in law. We use Thereoms 5.1 and 5.2 in Chapter 6. Theorem 5.1 Let (
1 , F1 , P ) be a probability space. Also let (
2 , F2 ) and (
3 , F3 ) be measurable spaces. Assume that {Xn :
1 →
2 } is a sequence of random variables which converges to X in law. If g :
2 →
3 is a continuous function, then {g(Xn ) :
1 →
3 } converges to g(X) in law. Proof of Theorem 5.1 If f :
2 →
3 is a bounded and continuous function, then f (g( )) is also a bounded and continuous function from
1 to
3 . Therefore lim EXn [f (g(Xn ))] = EX [f (g(X))],

n→∞

which shows that g(Xn ) converges in law to g(X).




Theorem 5.2 Let {Xn } and {Yn } be sequences of random variables which take values on the Euclidean space RN . (1) If Xn converges to 0 in law, then Xn converges to 0 in probability, where the probability distribution of 0 is defined by δ(x). (2) If Xn and Yn respectively converge to X and 0 in law, then Xn + Yn converges to X in law. Proof of Theorem 5.2 Let pn (dx) be a probability distribution of Xn . For an arbitrary  > 0, there exists a bounded and continuous function ρ(x) (0 ≤

136

Empirical processes

ρ(x) ≤ 1) which satisfies

 ρ(x) =

1 (x ≤ /2) 0 (x ≥ ).

By the convergence in law of Xn → 0,




1 − P (Xn  > ) =

x≤

pn (dx)


≥

ρ(x)pn (dx)


→

ρ(x)δ(x)dx = 1,

which shows P (Xn  > ) → 0. (2) Let f : RN → R be a bounded and uniformly continuous function. For an arbitrary  > 0, there exists δ > 0 such that |x − y| < δ =⇒ |f (x) − f (y)| < , where δ does not depend on x, y. Then A(n) ≡ |E[f (X)] − E[f (Xn + Yn )]| ≤ |E[f (X)] − E[f (Xn )]| + |E[f (Xn )] − E[f (Xn + Yn )]| = |E[f (X)] − E[f (Xn )]| + B(n), where B(n) is defined by B(n) = E[|f (Xn ) − f (Xn + Yn )|]. Also B(n) = E[|f (Xn ) − f (Xn + Yn )|]{|Yn | M)  −M (f (x) < −M). Then |fM (X)| ≤ |f (X)| and fM (x) is a continuous and bounded function. For each x, |fM (x)| is a non-decreasing function of M and lim |fM (x)| = |f (x)|.

M→∞

(5.4)

Then E[|fM (X)|] → E[|f (X)|] by the monotone lemma of Lebesgue measure theory. By the convergence Xn → X in law, E[|fM (X)|] = lim E[|fM (Xn )|] n→∞

≤ lim sup E[|f (Xn )|] < C n→∞




holds for each M.

Example 5.2 If a function f is unbounded, then three conditions, (1) Xn → X in law, (2) E[|f (Xn )|] < ∞, and (3) E[|f (X)|] < ∞, do not ensure E[f (Xn )] → E[f (X)] in general. For example, a sequence of probability distributions on R1 Pn (dx) =

1 { n δ(x) + δ(x − n) } dx n+1

and a probability distribution P (dx) = δ(x) dx satisfies the convergence in law, Pn → P . Let Xn and X be random variables which are subject to Pn and P respectively. For a continuous and unbounded

138

Empirical processes

function f (x) = x, E[f (Xn )] =

n → 1, n+1

whereas E[f (X)] = 0. When we need convergence of the expectation value of a sequence of weak convergent random variables, the following definition is important. Definition 5.2 (Asymptotically uniformly integrable, AUI) A sequence of realvalued random variables {Xn } is said to be asymptotically uniformly integrable if it satisfies lim lim sup E[|XN |]{|XN |≥M} = 0.

(5.5)

M→∞ n→∞ N≥n

Theorem 5.4 (1) If two sequences of real-valued random variables {Xn } and {Yn } satisfy |Xn | ≤ Yn and {Yn } is asymptotically uniformly integrable, then {Xn } is asymptotically uniformly integrable. (2) Let {Xn } be a sequence of real-valued random variables. If there exists a random variable Y such that |Xn | ≤ Y,

E[Y ] < ∞,

then {Xn } is asymptotically uniformly integrable. (3) Let 0 < δ < s be positive constants. If a sequence of real-valued random variables {Xn } satisfies E[|Xn |s ] < C where C does not depend on n, then Xns−δ is asymptotically uniformly integrable. Proof of Theorem 5.4 (1) Since |Xn (w)| ≤ Yn (w) for any w ∈
, for any M, {w ∈
; |Xn (w)| ≥ M} ⊂ {w ∈
; Yn (w) ≥ M}. It follows that E[|Xn |]{|Xn |≥M} ≤ E[|Xn |]{Yn ≥M} ≤ E[Yn ]{Yn ≥M} which shows (1) of the theorem. (2) This is a special case of (1) by putting Yn = Y . (3) By the assumption, it follows that E[|Xn |s−δ ]{|Xn |≥M} ≤ E[|Xn |s /M δ ]{|Xn |≥M} ≤ which completes (3) of the theorem.

C Mδ




5.1 Convergence in law

139

Theorem 5.5 (Convergence of expectation value) Assume that a sequence of
-valued random variables {Xn } converges to X in law, and that f :
→ R1 is a continuous function satisfying E[|f (Xn )|] < C. If f (Xn ) is asymptotically uniformly integrable, then lim E[f (Xn )] = E[f (X)]

n→∞

holds. Proof of Theorem 5.5 By Theorem 5.3, E[f (X)] is well defined and finite. Let fM (x) be the function defined in eq.(5.3). Then |E[f (X)] − E[f (Xn )]| ≤ |E[f (X)] − E[fM (X)]| + |E[fM (X)] − E[fM (Xn )]| + |E[fM (Xn )] − E[f (Xn )]|. Let the last term be A(n). A(n) ≤ E[|fM (Xn ) − f (Xn )|] ≤ E[|f (Xn )|]{|f (Xn )|≥M} . Therefore, |E[f (X)] − E[f (Xn )]| ≤ |E[f (X)] − E[fM (X)]| + |E[fM (X)] − E[fM (Xn )]| + E[|f (Xn )|]{|f (Xn )|≥M} . By using Theorem 5.3 and the definition of AUI, eq.(5.5), for arbitrary  > 0, there exists M which satisfies both |E[f (X)] − E[fM (X)]| <  and lim sup E[|f (XN )|]{|f (XN )|≥M} < .

n→∞ N≥n

For such an M, we can choose n which satisfies both |E[fM (X)] − E[fM (Xn )]| < , and sup E[|f (XN )|]{|f (XN )|≥M} < ,

N≥n

which completes the theorem.




140

Empirical processes

Remark 5.2 In statistical learning theory, we often use this theorem in the case Zn = f (ξn ) + an Xn . We prove the following conditions. (1) The sequence of random variables {Zn } is asymptotically uniformly integrable. (2) The sequence of random variables {ξn } converges to ξ in law. (3) The function f is continuous. (4) The real sequence an converges to zero. (5) The sequence of random variables Xn converges in law. Then, by Theorem 5.1, we have the convergence in law, Zn → f (ξ ), and by Theorem 5.5, E[Zn ] → E[f (ξ )].

5.2 Function-valued analytic functions In statistical learning theory, a statistical model or a learning machine is a probability density function-valued analytic function. In this section, we introduce a function-valued analytic function. Let RN be an N -dimensional Euclidean space and q(x) ≥ 0 be a probability density function on RN ,
q(x)dx = 1. RN

Hereafter, a real number s ≥ 1 is fixed. The set of all measurable functions f from RN to C1 which satisfy
|f (x)|s q(x)dx < ∞ is denoted by Ls (q). The set Ls (q) is a vector space over C. For an element f ∈ Ls (q), we define the norm  · s by 
1/s |f (x)|s q(x)dx , f s ≡ which satisfies the following conditions. (1) For any f ∈ Ls (q), f s ≥ 0. (2) f s = 0 ⇐⇒ f = 0. (3) For a ∈ C, af s = |a|f s . (4) For any f, g ∈ Ls (q), f + gs ≤ f s + gs . By this norm, Ls (q) is a Banach space.

5.2 Function-valued analytic functions

141

Remark 5.3 In Ls (q), the following hold. (1) Let α > 0 and β > 0 be constants which satisfy 1/α + 1/β = 1. For arbitrary f, g ∈ Ls (q), 
    f (x)g(x)q(x)dx  ≤ f α gβ . This is called H¨older’s inequality. The special case α = β = 1/2 is called the Cauchy–Schwarz inequality. (2) From H¨older’s inequality, for arbitrary 1 < s < s  , f s ≤ f s  holds. Therefore, 

Ls (q) ⊂ Ls (q). (3) From a mathematical point of view, Ls (q) is not a set of functions but the quotient set of the equivalent relation, f ∼ g ⇐⇒ f (x) = g(x)

(a.s. q(x)dx).

Although f represents the equivalence class, the value f (x) is well defined almost surely for x with q(x)dx. Definition 5.3 (Function-valued analytic function) Let s ≥ 1 be a real constant. A function f : RN × Rd  (x, w) → f (x, w) ∈ R1 is said to be Ls (q)-valued real analytic if there exists an open set W ⊂ Rd such that W  w → f ( , w) ∈ Ls (q) is analytic. Here ‘analytic’ means that, for arbitrary w∗ ∈ W , there exists {aα (x) ∈ Ls (q)} such that  f (x, w) = aα (x)(w − w ∗ )α (5.6) α∈Nd

is a convergent power series in a Banach space; in other words,  α∈Nd

aα s

d 

|wj − wj∗ |αj < ∞

j =1

in some open set |wj − wj∗ | < δj

(j = 1, 2, . . . , N).

(5.7)

142

Empirical processes

By the completeness of Ls (q), if eq.(5.7) holds, then the sum in eq.(5.6) absolutely converges. Remark 5.4 (Analytic function of several complex variables) Let f (z1 , z2 , . . . , zd ) be a function from Cd to C1 which is given by the Taylor expansion,  aα (z − b)α , (5.8) f (z) = α∈Nd

where (b1 , b2 , . . . , bd ) ∈ Cd . If the power series in eq.(5.8) absolutely converges in the complex region {(z1 , z2 , . . . , zd ) ∈ Cd ; |zi − bi | < ri ; i = 1, 2, . . . , d},

(5.9)

and it does not in {(z1 , z2 , . . . , zd ) ∈ Cd ; |zi − bi | > ri ; i = 1, 2, . . . , d}, then (r1 , . . . , rd ) are said to be associated convergence radii of f . If d = 1, then r1 is called the convergence radius that is uniquely determined. However, if d ≥ 2, then there are another (r1 , r2 , . . . , rd ) which are also associated convergence radii. If real numbers r = (r1 , r2 , . . . , rd ) are associated convergence radii, then lim sup |aα |r α = 1, |α|→∞

where, for a multi-index α, |α| is defined by |α| = α1 + α2 + · · · + αd . If C1 , C2 , . . . , Cd are closed and continuous lines in the region defined by eq.(5.9), then Cauchy’s integral formula

f (z1 , . . . , zd ) 1 ··· dz1 · · · dzd f (b) = (2π i)d C1 (z − b1 ) · · · (zd − bd ) 1 Cd √ holds, where i = −1. Remark 5.5 (Function-valued analytic function of several complex variables) Let f (x, z) be a function from Cd to Ls (q) defined by  aα (x)(z − b)α , f (x, z) = α

where aα ∈ L (q). Then, by using the completeness of Ls (q), the same associated convergence radii of a function-valued analytic function can be defined as above. For the associated convergence radii, s

lim sup aα s r α = 1. |α|→∞

5.2 Function-valued analytic functions

143

Cauchy’s integral formula also holds,

1 f (x, z) f (x, b) = · · · dz1 · · · dzd . d (2π i) C1 Cd (z1 − b1 ) · · · (zd − bd ) Moreover, by 1 aα (x) = (2π i)d it follows that |aα (x)| ≤

1 (2π)d





···

C1

Cd





··· C1

Cd

f (x, z) dz1 · · · dzd , (z1 − b1 )α1 +1 · · · (zd − bd )αd +1 supz |f (x, z)| d|z1 | · · · d|zd |. |z1 − b1 |α1 +1 · · · |zd − bd |αd +1

Then, if the radius of integrated path C = (C1 , C2 , . . . , Cd ) is given as R α < r α , we obtain an inequality, |aα (x)| ≤

supz∈C |f (x, z)| . Rα

(5.10)

Since the Taylor expansion of f (x, z) at z = b converges absolutely, completeness of Ls (q) ensures that sup |f (x, z)| z∈C s

is contained in L (q). Therefore, aα s ≤

 supz∈C |f (·, z)| s 0, and Fn (y) > 0. Moreover, s−2   (s − 1)s

s 1 Fn ((s − 1)s ) = 1+ 1+ √ 1 + s/2n 2n(s − 1) n (s − 1)  s

se (s − 1) 1+ ≤ (s − 1)s , ≤ 1 + s/2n 2n(s − 1) where we used (1 + 1/t)t < e < s − 1 (t ≥ 0). Therefore, for arbitrary n, 0 < yn ≤ (s − 1)s ,




which completes the theorem.

Theorem 5.8 Let s ≥ 2 be a positive and even integer. Assume that the function f (x, w) defines an Ls (q)-valued real analytic function on the open set W ⊂ Rd . For the pre-empirical process ψn (w) in eq.(5.12), there exists a constant C > 0 such that E[sup |ψn (w)|s ] ≤ C, w∈K

where K ⊂ W is a compact set. Proof of Theorem 5.8 Since ψn (w) is defined so that E[ψn (w)] = 0, we can assume EX [f (X, w)] = 0 without loss of generality. The Taylor expansion of f (x, w) for w ∗ ∈ K absolutely converges in a region, B(w∗ , r(w ∗ )) ≡ {w ∈ W ; |wj − wj∗ | < rj },

148

Empirical processes

where {rj } are associated convergence radii. The set K can be covered by the union of all open sets, K ⊂ ∪w∗ ∈K B(w ∗ , r(w ∗ )). Since K is compact, K is covered by a finite union of B(w ∗ , r(w ∗ )). Therefore, in order to prove the theorem, it is sufficient to prove that, in each B = B(w∗ , r(w ∗ )), E[sup |ψn (w)|s ] ≤ C w∈B

holds. By the Taylor expansion of f ∈ Ls (q) for w ∗ , 1  ψn (w) ≡ √ f (Xi , w) n i=1 n

1  = √ aα (Xi )(w − w ∗ )α n i=1 α n

=

n  1  aα (Xi ) (w − w∗ )α , √ n i=1 α

where aα ∈ Ls (q) is a coefficient of Taylor expansion. Since sup |ψn (w)| ≤ w∈B

n    1  α  aα (Xi ) rj j , √ n i=1 α j

we can apply Theorem 5.7 n   s 1/s

1/s   1  α   E sup |ψn (w)|s ≤ E √ aα (Xi ) rj j n w∈B α j i=1 n s 1/s    1  α  ≤ E √ aα (Xi ) rj j n α j i=1

≤ (s − 1)

 α

aα s



α

rj j

j

Here the {rj } were taken to be smaller than the associated convergence radii,
and the last term is a finite constant. Definition 5.5 (Separable and complete metric space of continuous functions) Let K ⊂ Rd be a compact set and C(K) be a set of all the continuous functions

5.3 Empirical process

149

from K to C. A metric of f1 , f2 ∈ C(K) is defined by d(f1 , f2 ) = max |f1 (x) − f2 (x)|. x∈K

Then C(K) is a complete and separable metric space. In fact, it is well known in the Weierstrass approximation theorem that, for any function f1 ∈ C(K) and any  > 0, there exists a polynomial f2 which satisfies d(f1 , f2 ) < . Let B be the Borel set of C(K), or equivalently, the minimal sigma algebra that contains all open subsets in C(K). Then (C(K), B) is a measurable set. Definition 5.6 (Empirical process) Let (
, B, P ) be a probability space and X be an RN -valued random variable. Assume that {Xn } is a set of RN -valued random variables which are independently subject to the same probability distribution as X. The probability distribution of X is denoted by q(x)dx. Let f : RN × W → R1 define an Ls (q)-valued analytic function w → f (·, w) where W ⊂ Rd is an open set. Assume that E[f (X, w)2 ] < ∞ for any w ∈ W and that the compact set K is a subset of W . The empirical process ψn is a C(K)-valued random variable defined by 1  ψn (w) = √ {f (Xi , w) − EX [f (X, w)]}. n i=1 n

(5.21)

Remark 5.7 In the above definition, ψn is contained in the functional space C(K) and it is measurable, therefore ψn is a C(K)-valued random variable. The empirical process can be defined even if f (·, w) is not an Ls (q)-valued analytic function; however, in this book, we adopt this restricted definition. For the more general definition, see [33, 92]. Definition 5.7 (Tight random variable) Let (C(K), B) be a measurable space as defined in Definition 5.5. (1) A probability measure µ on (C(K), B) is said to be tight if, for any  > 0, there exists a compact set C ⊂ C(K) such that µ(C) > 1 − . (2) A sequence of measures {µn } is said to be uniformly tight if, for any  > 0, there exists a compact set C ⊂ C(K) such that inf n µn (C) > 1 − . (3) A C(K)-valued random variable X is said to be tight if the probability measure to which X is subject is tight. A sequence of C(K)-valued random variables {Xn } is said to be uniformly tight if the sequence of probability distributions to which {Xn } are subject is uniformly tight. Example 5.3 The empirical process {ψn (w)} in Definition 5.6 is uniformly tight if, for arbitrary  > 0, there exists a compact set C ⊂ C(K) such that P (ψn ∈ C) > 1 − 

150

Empirical processes

for any n. If E[sup |ψn (w)|s ] = C  < ∞, w∈K

then C  ≥ E[sup |ψn (w)|s ]{supw |ψn (w)|≥M} w∈K

≥ M P (sup |ψn (w)| ≥ M). s

w∈K

Therefore P (sup |ψn (w)| < M) ≥ 1 − w∈K

C , Ms

which shows {ψn } is uniformly tight. Theorem 5.9 (Convergence in law of empirical process) Let s ≥ 2 be a positive and even constant. Assume that f (x, w) is an Ls (q)-valued analytic function defined on an open set W ⊂ RN . If K is a compact subset in W , then the empirical process ψn ∈ C(K) defined in eq.(5.21) satisfies the following conditions. (1) The set of C(K)-valued random variables {ψn } is uniformly tight. (2) There exists a C(K)-valued random variable ψ such that {ψn } converges to ψ in law. In other words, for an arbitrary bounded and continuous function g from C(K) to R1 , lim Eψn [g(ψn )] = Eψ [g(ψ)].

n→∞

(3) The random process ψ is the normal distribution that has expectation eq.(5.13) and covariance eq.(5.14). Proof of Theorem 5.9 Firstly, by Example 5.3 and Theorem 5.8, the sequence {ψn (w)} is uniformly tight. Prohorov’s theorem shows that, in a uniformly tight sequence of measures on a complete and separable metric space, there exists a subsequence which converges in law to a tight measure. Since C(K) ⊂ RK , B ⊂ B K and the limiting process is unique in (RK , B K ) by Remark 5.6, we
obtain the theorem. Remark 5.8 Even if f (x, w) is not an Ls (q)-valued analytic function, the same conclusion as Theorem 5.9 can be derived on the weaker condition [33, 92]. Theorem 5.10 Let s be a positive even integer. Assume that a function f (x, w) is an Ls+2 (q)-valued analytic function of w ∈ W and K ⊂ W is a compact set. The empirical process ψn defined by eq.(5.21) satisfies lim Eψn [sup |ψn (w)|s ] = Eψ [sup |ψ(w)|s ].

n→∞

w∈K

w∈K

5.3 Empirical process

151

Proof of Theorem 5.10 Let h : C(K) → R1 be a function defined by C(K)  ϕ → h(ϕ) ≡ sup |ϕ(w)| ∈ R1 . w∈K

Then h is a continuous function on C(K) because |h(ϕ1 ) − h(ϕ2 )| = | sup |ϕ1 (w)| − sup |ϕ2 (w)|| w∈K

w∈K

≤ sup |ϕ1 (w) − ϕ2 (w)|. w∈K

By the convergence in law of ψn → ψ in Theorem 5.9 and continuity of h, the convergence in law sup |ψn (w)|s → sup |ψ(w)|s w∈K

w∈K

holds. On the other hand, by Theorem 5.8, Eψn [sup |ψn (w)|s+2 ] ≤ C w∈K s

which implies that h(ψn ) is asymptotically uniformly integrable. By applying
Theorem 5.5, we obtain the theorem. Example 5.4 Let X and Y be real-valued independent random variables which are respectively subject to the uniform distribution on [−1, 1] and the standard normal distribution. Let q(x) and q(y) denote the probability density functions of X and Y respectively. When we use a statistical model, 1 p(y|x, a, b) = √ exp(− 12 (y − a tanh(bx))2 ), 2π then q(y) = p(y|x, 0, 0). If {Xi , Yi } are independently taken from the distribution q(x)q(y), then the Kullback–Leibler distance and the log likelihood ratio function are respectively given by
a2 tanh(bx)2 q(x)dx, K(a, b) = 2 1  2 {a tanh(bXi )2 − 2aYi tanh(bXi )}. 2n i=1 n

Kn (a, b) = If we define

[K(a, b)]

1/2

 √ + K(a, b) √ = − K(a, b)

(ab ≥ 0) (ab < 0),

152

Empirical processes

then the origin a = b = 0 is a removable singularity of ψn (a, b) =

√ K(a, b) − Kn (a, b) n , [K(a, b)]1/2

which is an empirical process, and the log likelihood ratio function can be written as the standard form, Kn (a, b) = K(a, b) +

[K(a, b)]1/2 ψn (a, b). √ n

In Chapter 6, we show that any log likelihood ratio function can be written in the same standard form. Example 5.5 Let X and Y be random variables which are independently subject to the standard normal distribution. We study a function f (a, b, x, y) = ax + by. Then the expectation and variance are respectively given by E[f (a, b, X, Y )] = 0, E[f (a, b, X, Y )2 ] = a 2 + b2 . If {Xi } and {Yi } are independently subject to the same probability distributions as X and Y respectively, 1  f (Xi , Yi , a, b) fn (a, b) = √ √ n i=1 a 2 + b2 n

n n  1    1   b = √ Xi + √ Yi √ √ n i=1 n i=1 a 2 + b2 a 2 + b2

a

defines an empirical process on a set W = {(a, b); |a|, |b| ≤ 1, a 2 + b2 = 0}. Although E[fn (a, b)2 ] = 1 for an arbitrary (a, b) ∈ W , lim

(a,b)→(0,0)

fn (a, b)

does not exist. The origin is a singularity of this empirical process. Let us apply the blow-up, a = a1 = a2 b2 , b = a1 b1 = b2 .

5.3 Empirical process

153

ψn (w)

Kn (w)

W W

W0

W0

Fig. 5.1. Empirical process

Then (a 2 + b2 )1/2 = a12 = b22

!1/2 !1/2

1 + b12

!1/2 !1/2

a22 + 1

.

Therefore, instead of fn (a, b), we can consider fˆn on the real projective space represented on each coordinate by n n  1   1 b1  1   fˆn = ( Xi + ( Yi √ √ n i=1 n i=1 1 + b12 1 + b12

 1   a2  1   1 = ( Xi + ( Yi , √ √ n i=1 n i=1 a22 + 1 a22 + 1 n

n

which is an empirical process without singularities. Example 5.6 In singular learning theory, the log likelihood ratio function Kn (w) is illustrated in the left of Figure 5.1. If w ∈ W0 , in other words, K(w) = 0, then Kn (w) = 0. To analyze Kn (w), it is represented as 1  Kn (w) = K(w) − √ K(w) ψn (w), n but ψn (w) is ill-defined at a singularity of K(w) = 0 as in the right of Figure 5.1. In Chapter 6, we show that the same procedure as in Example 5.4 and 5.5, where ψn (w) becomes well-defined, is realized by resolution of singularities.

154

Empirical processes

5.4 Fluctuation of Gaussian processes Definition 5.8 (Fluctuation function) Let β, λ > 0 be real numbers. A fluctuation function is defined by
∞ √ t λ−1 e−βt+β a t dt. Sλ (a) = 0

The reason why this definition is introduced is clarified in Chapter 6. For a given real-valued random variable X, Sλ (X) shows the fluctuation of X under the singular dimension λ with inverse temperature β. For example, if X is subject to the normal distribution with mean 0 and variance 2, and 0 < β < 1, then E[Sλ (X)] = where




∞

(λ) =

(λ) , (β − β 2 )λ

t λ−1 e−t dt

0

is the gamma function. It is easy to show that Sλ (0) = (λ), Sλ (a) = β Sλ+1/2 (a), Sλ (a) = β 2 Sλ+1 (a). By using partial integration, it follows that
∞  −βt  √ e Sλ+1 (a) = (t λ eβ a t ) dt − β 0
∞ −βt √ e (t λ eβ a t ) dt = β 0 =

a λ Sλ+1/2 (a) + Sλ (a). 2 β

Therefore Sλ (a) =

aβ  S (a) + λβSλ (a). 2 λ

(5.22)

Let M be a compact subset of some manifold, and a(x, u) be a function such that RN × M  (x, u) → a(x, u) ∈ R1 .

5.4 Fluctuation of Gaussian processes

155

Assume that ξ (u) is a Gaussian process on M which satisfies E[ξ (u)] = 0, E[ξ (u)ξ (v)] = EX [a(X, u)a(X, v)] ≡ ρ(u, v). Definition 5.9 (Singular fluctuation) Let µ(du) be a measure on M. Assume ρ(u, u) = 2 for an arbitrary u ∈ M. The singular fluctuation of the Gaussian process ξ (u) on M is defined by
 1 1 µ(du) ξ (u) Sλ (ξ (u)) , E ν(β) = 2β Z(ξ ) M where


Z(ξ ) =

M

µ(du) Sλ (ξ (u)).

From eq.(5.22) we obtain the following theorem. Theorem 5.11 The following equations hold.

 µ(du) S  (ξ (u))  λ (5.23) = λβ + β 2 ν(β), E Z(ξ )

  µ(du) a(X, u) S  (ξ (u)) 2  λ 1 EEX = λβ + (β 2 − β)ν(β). (5.24) 2 Z(ξ ) Proof of Theorem 5.11 The first relation eq.(5.23) is derived directly from eq.(5.22). Let us prove eq.(5.24). Let {gi }∞ i=1 be independent Gaussian random variables on R which satisfy E[gi ] = 0, E[gi gj ] = δij . For such random variables,

∂  F (gi ) E[gi F (gi )] = E ∂gi holds for a differentiable function of F (·). Since L2 (q) is a separable Hilbert space, there exists a complete orthonormal system {ek (x)}∞ k=1 . By defining
bk (u) = a(x, u) ek (u) q(x) dx, it follows that a(x, u) =

∞ 

bk (u)ek (x),

k=1

EX [a(X, u)a(X, v)] =

∞  k=1

bk (u)bk (v).

156

Empirical processes

A Gaussian process defined by ξ ∗ (u) =

∞ 

bk (u) gk

k=1

has the same expectations and covariance matrices as ξ (u), therefore it is subject to the same probability distribution as ξ (u). Thus we can identify ξ (u) = ξ ∗ (u) in the calculation of expectation values. E[ξ (u)ξ (v)] =

∞ 

bi (u)bi (v).

i=1

Then

 µ(du) ξ (u)S  (ξ (u))  λ A≡E Z(ξ ) ∞ 


S  (ξ (u))  µ(du) =E bi (u)gi λ Z(ξ ) i=1 =E




∞ 

µ(du)

i=1

bi (u)

∂ Sλ (ξ (u))  . ∂gi Z(ξ )

By using S  (ξ (u))bi (u) ∂  Sλ (ξ (u))  = λ ∂gi Z(ξ ) Z(ξ )
 Sλ (ξ (u)) µ(dv) Sλ (ξ (v))bi (v), − Z(ξ )2 we obtain  2 Sλ (ξ (u)) ∞ i=1 bi (u) Z(ξ ) 



Sλ (ξ (u))Sλ (ξ (v)) i bi (u)bi (v)  µ(du) µ(dv) −E Z(ξ )2


S  (ξ (u))ρ(u, u)  µ(du) λ =E Z(ξ )


S  (ξ (u))Sλ (ξ (v)) ρ(u, v)  µ(du) µ(dv) λ −E . Z(ξ )2

A=E




µ(du)

(5.25)

5.4 Fluctuation of Gaussian processes Therefore, by ρ(u, u) = 2,

 µ(du) S  (ξ (u)) 

 µ(du) ξ (u) S  (ξ (u))  λ λ = 2E E Z(ξ ) Z(ξ )

  µ(du) a(X, u) S  (ξ (u)) 2  λ −EEX , Z(ξ )

157

(5.26)




which completes the proof.

Remark 5.9 (1) A singular fluctuation expresses the fluctuation on M. It is represented by
 1 ∂ 

µ(du) Sλ (aξ (u)) . E log ν(β) = − a=1 2β ∂a M Moreover, by using ∂ Sλ (a) = −Sλ+1 (a) + aSλ+1/2 (a) ∂β = −(λ/β)Sλ (a) + (a/(2β))Sλ (a), the singular fluctuation and F = −E[log Z(ξ )] satisfy λ ∂F = − ν(β). ∂β β (2) Define an expectation value, √  ∞ µ(du) 0 f (t, u) t λ−1 e−βt+βξ (u) t dt √ f (t, u) ≡ .  ∞ µ(du) 0 t λ−1 e−βt+βξ (u) t dt Then, by Theorem 5.11 and EX [a(X, u)2 ] = 2, ν(β) can be rewritten as * )

) √ *2  β ν(β) = Eξ EX a(X, u)2 t − a(X, u) t . 2 √ This is the variance of ta(X, u). For the meaning of this term, see Remark 6.13.

6 Singular learning theory

In this chapter we prove the four main formulas in singular learning theory. The formulas which clarify the singular learning process are not only mathematically beautiful but also statistically useful. Firstly, we introduce the standard form of the log likelihood ratio function. A new foundation is established on which singular learning theory is constructed without the positive definite Fisher information matrix. By using resolution of singularities, there exists a map w = g(u) such that the log likelihood ratio function of any statistical model is represented by 1 Kn (g(u)) = u2k − √ uk ξn (u), n where uk is a normal crossing function and ξn (u) is an empirical process which converges to a Gaussian process in law. Secondly, we prove that, under a natural condition, the stochastic complexity of an arbitrary statistical model can be asymptotically expanded as
 n Fn = −log p(Xi |w)β ϕ(w)dw i=1

= nβSn + λ log n − (m − 1) log log n + FnR , where Sn is the empirical entropy of the true distribution, FnR is a random variable which converges to a random variable in law, and (−λ) and m are respectively equal to the largest pole and its order of the zeta function of a statistical model,
ζ (z) = K(w)z ϕ(w)dw. In regular statistical models λ = d/2 and m = 1 where d is the dimension of the parameter space, whereas in singular learning machines λ = d/2 and m ≥ 1 158

Singular learning theory

159

in general. The constant λ, the learning coefficient, is equal to the real log canonical threshold of the set of true parameters if ϕ(w) > 0 on singularities. Thirdly, we prove that the means of Bayes generalization error Bg , Bayes training error Bt , Gibbs generalization error Gg , and Gibbs training error Gt are respectively given by  λ + νβ − ν  1 1 E[Bg ] = +o , (6.1) β n n  λ − νβ − ν  1 1 +o , (6.2) E[Bt ] = β n n  λ + νβ  1 1 +o , (6.3) E[Gg ] = β n n  λ − νβ  1 1 +o , (6.4) E[Gt ] = β n n where n is the number of random samples, β > 0 is the inverse temperature of the a posteriori distribution, and ν = ν(β) > 0 is the singular fluctuation. From these equations, the equations of states in statistical estimation are derived: 1 , E[Bg ] − E[Bt ] = 2β(E[Gt ] − E[Bt ]) + o n 1 . E[Gg ] − E[Gt ] = 2β(E[Gt ] − E[Bt ]) + o n Although both λ and ν(β) strongly depend on the set of a true distribution, a statistical model, and an a priori distribution, the equations of states always hold independent of them. Therefore, based on the equations of states, we can predict Bayes and Gibbs generalization errors from Bayes and Gibbs training errors without any knowledge of the true distribution. Based on random samples, we can evaluate how appropriate a model and an a priori distribution are. And, lastly, we show the symmetry of the generalization and training errors in the maximum likelihood or the maximum a posteriori estimation, 1 C +o , E[Rg ] = n n 1 C , E[Rt ] = − + o n n where Rg and Rt are the generalization error and the training error of the maximum likelihood or a posteriori estimator respectively. In singular statistical models, the constant C > 0 is given by the maximum values of a Gaussian

160

Singular learning theory

process on the set of true parameters, hence C is far larger than d/2 in general. In order to make C small, we need a strong improvement by some penalty term, which is not appropriate in singular statistical estimation. From a historical point of view, the concepts and proofs of this chapter were found by the author of this book.

6.1 Standard form of likelihood ratio function To establish singular learning theory we need some fundamental conditions. Definition 6.1 (Fundamental condition (I)) Let q(x) and p(x|w) be probability density functions on RN which have the same support. Here p(x|w) is a parametric probability density function for a given parameter w ∈ W ⊂ Rd . The set of all parameters W is a compact set in Rd . The Kullback–Leibler distance is defined by
q(x) dx. K(w) = q(x) log p(x|w) We assume W0 = {w ∈ W ; K(w) = 0} is not the empty set. It is said that q(x) and p(x|w) satisfy the fundamental condition (I) with index s (s ≥ 2) if the following conditions are satisfied. (1) For f (x, w) = log(q(x)/p(x|w)), there exists an open set W (C) ⊂ Cd such that: (1-a) W ⊂ W (C) , (1-b) W (C)  w → f (·, w) is an Ls (q)-valued complex analytic function, (1-c) M(x) ≡ supw∈W (C) |f (x, w)| is contained in Ls (q). (2) There exists  > 0 such that, for Q(x) ≡ sup p(x|w), K(w)≤

the following integral is finite,
M(x)2 Q(x)dx < ∞. Remark 6.1 These are remarks about the fundamental condition (I). (1) By definition, there exists a real open set W (R) ⊂ Rd such that W ⊂ W (R) ⊂ W (C) .

6.1 Standard form of likelihood ratio function

161

The log density ratio function f (x, w) is an Ls (q)-valued real analytic function on W (R) and an Ls (q)-valued complex analytic function on W (C) . For a sufficiently small constant  > 0, we define W = {w ∈ W ; K(w) ≤ }. Based on the resolution theorem in Chapters 2 and 3, there exist open sets W(R) ⊂ W (R) , W(C) ⊂ W (C) and subsets of manifolds M, M(R) , and M(C) such that W ⊂ W(R) ⊂ W(C) , M ⊂ M(R) ⊂ M(C) , and that M ≡ g −1 (W ),

! M(R) ≡ g −1 W(R) , ! M(C) ≡ g −1 W(C) , where w = g(u) is the resolution map and its complexification. In the following, we use this notation. (2) In the fundamental condition (I), we mainly study the case in which W0 ≡ {w ∈ W ; K(w) = 0} is not one point but a set with singularities. In other words, the Fisher information matrix of p(x|w) is degenerate on W0 in general. However, this condition contains the regular statistical model as a special case. (3) From conditions (1-b) and (1-c), f (x, w) is represented by the absolutely convergent power series in the neighborhood of arbitrary w0 ∈ W :  aα (x)(w − w0 )α . f (x, w) = α

The function aα (x) ∈ L (q) is bounded by s

|aα (x)| ≤

M(x) , Rα

where R is the associated convergence radii. (4) If M(x) satisfies
M(x)2 eM(x) q(x)dx < ∞,

162

Singular learning theory

then condition (2) is satisfied, because



M(x)2 Q(x)dx =

M(x)2 sup p(x|w)dx


=
≤

K(w)≤

M(x)2 sup e−f (x,w) q(x)dx K(w)≤

M(x)2 eM(x) q(x)dx.

(5) Let w = g(u) be a real proper analytic function which makes K(g(u)) normal crossing. If q(x) and p(x|w) satisfy the fundamental condition (I) with index s, then q(x) and p(x|g(u)) also satisfy the same condition, where Rd and Cd are replaced by real and complex manifolds respectively. (6) The condition that W is compact is necessary because, even if the log density ratio function is a real analytic function of the parameter, |w| = ∞ is an analytic singularity in general. For this reason, if W is not compact and W0 contains |w| = ∞, the maximum likelihood estimator does not exist in general. For example, if x = (x1 , x2 ), w = (a, b), and p(x2 |x1 , w) ∝ exp(−(x2 − a sin(bx1 ))2 /2), then the maximum likelihood estimator never exists. On the other hand, if |w| = ∞ is not a singularity, Rd ∪ {|w| = ∞} can be understood as a compact set and the same theorems as this chapter hold. From the mathematical point of view, it is still difficult to construct singular learning theory of a general statistical model in the case when W0 contains analytic singularities. From the viewpoint of practical applications, we can select W as a sufficiently large compact set. (7) If a model satisfies the fundamental condition (I) with index s + t, (s ≥ 2, t > 0), then it automatically satisfies the fundamental condition (I) with index s. The condition with index s = 6 is needed to ensure the existence of the asymptotic expansion of the Bayes generalization error in the proof. (See the proof of Theorem 6.8.) (8) Some non-analytic statistical models can be made analytic. For example, in a simple mixture model p(x|a) = ap1 (x) + (1 − a)p2 (x) with some probability densities p1 (x) and p2 (x), the log density ratio function f (x, a) is not analytic at a = 0, but it can be made analytic by the representation p(x|θ ) = α 2 p1 (x) + β 2 p2 (x), on the manifold θ ∈ {α 2 + β 2 = 1}. As is shown in the proofs, if W is contained in a real analytic manifold, then the same theorems as this chapter hold. (9) The same results as this chapter can be proven based on the weaker conditions. However, to describe clearly the mathematical structure of singular

6.1 Standard form of likelihood ratio function

163

learning theory, we adopt the condition (I). For the case of the weaker condition, see Section 7.8. Theorem 6.1 Assume that q(x) and p(x|w) satisfy the fundamental condition (I) with index s = 2. There exist a constant  > 0, a real analytic manifold M(R) , and a real analytic function g : M(R) → W(R) such that, in every local coordinate U of M(R) , 2kd 1 2k2 K(g(u)) = u2k = u2k 1 u2 · · · ud ,

where k1 , k2 , . . . , kd are nonnegative integers. Moreover, there exists an Ls (q)valued real analytic function a(x, u) such that f (x, g(u)) = a(x, u) uk and

(u ∈ U ),

(6.5)

(u ∈ U ).

(6.6)


a(x, u)q(x)dx = uk

Remark 6.2 (1) In this chapter,  > 0 is taken so that Theorem 6.1 holds. The set of parameters W is represented as the union of two subsets. The former is W which includes {w; K(w) = 0} and the latter is W \ W in which K(w) > . To W , we can apply resolution of singularities. (2) A typical example of K(w), w = g(u), f (x, g(u)), and a(x, u) is shown in Example 7.1. Proof of Theorem 6.1 Existence of  > 0, M(R) , and g is shown by resolution of singularities, Theorem 2.3. Let us prove eq.(6.5). For arbitrary u ∈ U ,
K(g(u)) = f (x, g(u))q(x)dx
=

(e−f (x,g(u)) + f (x, g(u))) − 1)q(x)dx


=

f (x, g(u))2 −t ∗ f (x,g(u)) q(x)dx, e 2

where 0 < t ∗ < 1. Let U  ⊂ U be a neighborhood of u = 0. For arbitrary L > 0 the set DL is defined by  DL ≡ x ∈ RN ; sup |f (x, g(u))| ≤ L . u∈U 

Then for any u ∈ U  ,


u

2k

≥ DL

f (x, g(u))2 −L e q(x)dx, 2

164

Singular learning theory

giving the result that, for any uk = 0 (u ∈ U  ),
f (x, g(u))2 q(x)dx. 1 ≥ e−L 2u2k DL

(6.7)

Since f (x, g(u)) is an Ls (q)-valued real analytic function, it is given by an absolutely convergent power series,  aα (x)uα f (x, g(u)) = α

= a(x, u)uk + b(x, u)uk , where a(x, u) =



aα (x)uα−k ,

α≥k

b(x, u) =



aα (x)uα−k ,

α } and ξn (u) on M converge in law to the Gaussian processes ψ(w) and ξ (u), respectively. (2) Assume that the fundamental condition (I) holds with s = 4. Then the empirical processes satisfy



 lim E sup |ψn (w)|s−2 = E sup |ψ(w)|s−2 < ∞, n→∞

K(w)>



u∈M



K(w)>

 lim E sup |ξn (u)|s−2 = E sup |ξ (u)|s−2 < ∞.

n→∞



u∈M

Proof of Theorem 6.2 For ψn (w), this theorem is immediately derived from Theorems 5.9 and 5.10. Let us prove the theorem for ξn (u). The subset M is compact because W is compact and the resolution map g : M(R) → W(R) is proper. Therefore M can be covered by a finite union of local coordinates.
From Theorems 5.9 and 5.10, we immediately obtain the theorem. Remark 6.4 By the above theorem, the limiting process ξ (u) is a Gaussian process on M which satisfies E[ξ (u)] = 0 and E[ξ (u)ξ (v)] = EX [a(X, u)a(X, v)] − uk v k .

6.1 Standard form of likelihood ratio function

167

In particular, if K(g(u)) = K(g(v)) = 0, then E[ξ (u)ξ (v)] = EX [a(X, u)a(X, v)]. In other words, the Gaussian process ξ (u) has the same mean and covariance function as ξn (u). Note that the tight Gaussian process is uniquely determined by its mean and covariance. Theorem 6.3 Assume that q(x) and p(x|w) satisfy the fundamental condition (I) with index s = 4. If K(g(u)) = 0, then EX [a(x, u)2 ] = E[|ξn (u)|2 ] = E[|ξ (u)|2 ] = 2. Proof of Theorem 6.3 It is sufficient to prove EX [a(X, u)2 ] = 2 when K(g(u)) = 0. Let the Taylor expansion of f (x, g(u)) be  aα (x)uα . f (x, g(u)) = α

Then M(x) , Rα

|aα (x)| ≤

where R are associated convergence radii and  a(x, u) = aα (x)uα−k . α≥k

Hence |a(x, u)| ≤

 M(x) α≥k

= c1

Rα

r α−k

M(x) , Rk

where c1 > 0 is a constant. In the same way as in the proof of Theorem 6.1 and with f (x, g(u)) = a(x, u)uk , for arbitrary u (uk = 0), we have
a(x, u)2 −t ∗ a(x,u)uk q(x)dx, e 1= 2 where 0 < t ∗ < 1. Put S(x, u) =

a(x, u)2 −t ∗ a(x,u)uk q(x). e 2

168

Singular learning theory

Then S(x, u) ≤ c1

 M(x)2 k max 1, e−a(x,u)u q(x) 2k u R

= c1

M(x)2 max{q(x), p(x|w)} w R 2k

≤ c1

M(x)2 Q(x). R 2k

By the fundamental condition (I), M(x)2 Q(x) is an integrable function, hence S(x, u) is bounded by the integrable function. By using Lebesgue’s convergence theorem for uk → 0, we obtain
a(x, u)2 q(x)dx 1= 2




for any u that satisfies u2k = 0.

6.2 Evidence and stochastic complexity Definition 6.2 (Evidence) Let q(x) and p(x|w) be probability distributions which satisfy the fundamental condition (I) with s = 2. The set Dn = {X1 , X2 , . . . , Xn } consists of random variables which are independently subject to q(x)dx. Let ϕ(w) be a probability density function on Rd . The evidence of a pair p(x|w) and ϕ(w) for Dn is defined by
 n  p(Xi |w)β ϕ(w)dw. Zn = i=1

Also the stochastic complexity is defined by Fn = −log Zn . The normalized evidence and the normalized stochastic complexity are respectively defined by Zn0 =

Zn n 

q(xi )β

i=1


=

exp(−nβKn (w))ϕ(w)dw,

Fn0 = −log Zn0 .

(6.12) (6.13)

6.2 Evidence and stochastic complexity

169

Theorem 6.4 For arbitrary natural number n, the normalized stochastic complexity satisfies
  E Fn0 ≤ −log exp(−nβK(w))ϕ(w)dw. Proof of Theorem 6.4
Fn0 = −log exp(−nβKn (w))ϕ(w)dw
= −log

exp(−nβ(Kn (w) − K(w)) − nβK(w))ϕ(w)dw
exp(−nβ(Kn (w) − K(w)))ρ(w)dw

= −log
−log

exp(−nβK(w))ϕ(w)dw,

where ρ(w) = 

exp(−nβK(w))ϕ(w) . exp(−nβK(w  ))ϕ(w  )dw 

By Jensen’s inequality and a definition K ∗ (w) ≡ Kn (w) − K(w),

 exp(−nβK ∗ (w))ρ(w)dw ≥ exp − nβK ∗ (w)ρ(w)dw . Using E[K ∗ (w)] = 0, we obtain the theorem.




Remark 6.5 In the proof of Theorem 6.4, the convergence in law of ξn (u) → ξ (u) is not needed. Therefore, the upper bound of the stochastic complexity can be shown by the weaker condition. Definition 6.3 (Fundamental condition (II)) Assume that the set of parameters W is a compact set defined by W = {w ∈ Rd ; π1 (w) ≥ 0, π2 (w) ≥ 0, . . . , πk (w) ≥ 0}, where π1 (w), π2 (w), . . . , πk (w) are real analytic functions on some real open set W (R) ⊂ Rd . The a priori probability density function ϕ(w) is given by ϕ(w) = ϕ1 (w)ϕ2 (w) where ϕ1 (w) > 0 is a function of class C ∞ and ϕ2 (w) ≥ 0 is a real analytic function. Remark 6.6 In singular statistical models, the set of parameters and the a priori distribution should be carefully prepared from the theoretical point of view. In particular, their behavior in the neighborhood of K(w) = 0 and the boundary of W have to be set naturally. The condition that π1 (w), π2 (w), . . . , πk (w)

170

Singular learning theory

and ϕ1 (w) are real analytic functions is necessary because, if at least one of them is a function of class C ∞ , there is a pathological example. In fact, if ϕ1 (w) = exp(−1/w2 ) (w ∈ R1 ) and K(w) = w 2 in a neighborhood of the origin, then (d/dw)k ϕ1 (0) = 0 for an arbitrary k ≥ 0, and


1 1 (w 2 )z exp − 2 dw w −1 has no pole. Theorem 6.5 (Partition of parameter space) Assume the fundamental conditions (I) and (II) with index s = 2. Let  > 0 be a constant. By applying Hironaka’s resolution theorem (Theorem 2.3) to a real analytic function, K(w)( − K(w))ϕ2 (w)

k 

πj (w),

j =1

we can find a real analytic manifold M(R) and a proper and real analytic map g : M(R) → W(R) such that all functions K(g(u)),

 − K(g(u)),

ϕ2 (g(u)),

π1 (g(u)), . . . , πk (g(u))

have only normal crossing singularities. By using Remark 2.14 and Theorem 2.11, we can divide the set W = {w ∈ W ; K(w) ≤ } such that the following conditions (1), (2), (3), and (4) are satisfied. (1) The set of parameters M = g −1 (W ) is covered by a finite set M = ∪α Mα , where Mα is given by a local coordinate, Mα = [0, b]d = {(u1 , u2 , . . . , ud ) ; 0 ≤ u1 , u2 , . . . , ud ≤ b}. (2) In each Mα , 2kd 1 2k2 K(g(u)) = u2k = u2k 1 u2 · · · ud ,

where k1 , k2 , . . . , kd are nonnegative integers. (3) There exists a function φ(u) of class C ∞ such that ϕ(g(u))|g  (u)| = φ(u)uh = φ(u)uh1 1 uh2 2 · · · hhd d , where |g  (u)| is the absolute value of the Jacobian determinant and φ(u) > c > 0 (u ∈ [0, b]d ) is a function of class C ∞ , where c > 0 is a positive constant.

6.2 Evidence and stochastic complexity

171

(4) There exists a set of functions {σα (u)} of class C ∞ which satisfy 

σα (u) ≥ 0, σα (u) = 1,

α

σα (u) > 0 (u ∈ [0, b)d ), supp σα (u) = [0, b]d , such that, for an arbitrary integrable function H (w),

H (w)ϕ(w)dw = H (g(u))ϕ(g(u))|g  (u)|du M

W

=


α

H (g(u))φ ∗ (u)uh du,

Mα

∗

where we defined φ (u) by omitting local coordinate α, φ ∗ (u) ≡ σα (u)φ(u). Moreover there exist constants C1 > 0 such that

C1 H (g(u))φ(u)uh du ≤ H (w)ϕ(w)dw α

Mα

W

≤


α

H (g(u))φ(u)uh du.

(6.14)

Mα

Proof of Theorem 6.5 This theorem is obtained by the resolution theorem
(Theorem 2.3), Theorem 2.11, and Remark 2.14. Theorem 6.6 Assume the fundamental conditions (I) and (II) with index s = 2. The holomorphic function of z ∈ C,
ζ (z) = K(w)z ϕ(w)dw (Re(z) > 0), (6.15) can be analytically continued to the unique meromorphic function on the entire complex plane whose poles are all real, negative, and rational numbers. Proof of Theorem 6.6 Let us define
ζ1 (z) =


K(w)z ϕ(w)dw,

K(w) 0,
[0,b]d

u2kz+h+j du =

d 

b2kp z+hp +jp +1 . (2kp z + hp + jp + 1) p=1

Hence the function ζ1 (z) can be analytically continued to the unique meromor
phic function and all poles are real, negative, and rational numbers. Definition 6.4 (Zeta function and learning coefficient) The meromorphic function ζ (z) that is analytically continued from eq.(6.15) is called the zeta function of a statistical model. The largest pole and its order are denoted by (−λ) and m, respectively, where λ and m are respectively called the learning coefficient and its order. If the Kullback–Leibler distance and the a priori distribution are represented as in Theorem 6.5, then the learning coefficient is given by h + 1 j , (6.16) λ = min min α 1≤j ≤d 2kj and its order m is m = max {j ; λ = (hj + 1)/(2kj )}, α

(6.17)

where  shows the number of elements of the set S. Let {α ∗ } be a set of all local coordinates in which both the minimization in eq.(6.16) and the maximization in eq.(6.17) are attained. Such a set of local coordinates {α ∗ } is said to be the essential family of local coordinates. For each local coordinate α ∗ in the essential family of local coordinates, we can assume without loss of generality that u is represented as u = (x, y) such that x = (u1 , u2 , . . . , um ), y = (um+1 , um+2 , . . . , ud ),

6.2 Evidence and stochastic complexity

173

and that λ=

hj + 1 (1 ≤ j ≤ m), 2kj

λ
0 is a constant defined by eq.(4.16). Proof of Theorem 6.7 The normalized evidence can be divided as Zn0 = Zn(1) + Zn(2) , where


Zn(1) =
Zn(2) =

e−nβKn (w) ϕ(w)dw, K(w)≤

e−nβKn (w) ϕ(w)dw. K(w)>

Firstly, let us study Zn(2) . If K(w) > , by using the Cauchy–Schwarz inequality,  nKn (w) = nK(w) − K(w)ψn (w) nK(w) − ψn (w)2 2  1 ≥ n − sup |ψn (w)|2 . 2 K(w)>

≥

Since ψn (w) is an empirical process which converges to a Gaussian process with supremum norm in law, supK(w)> |ψn (w)|2 converges in law, and therefore 0≤

β  nλ e−nβ/2 nλ (2) 2 Z ≤ exp |ψ (w)| sup n n (log n)m−1 (log n)m−1 2 K(w)>

(6.18)

174

Singular learning theory

converges to zero in probability by Theorem 5.2. Secondly, Zn(1) is given by 
∞
√ Zn(1) = dt exp(−nβu2k + β nuk ξn (u))uh φ ∗ (u)du. α

0

Let us define Y (1) (ξn ) ≡ γb

Mα


α∗




∞

dt

dy t λ−1 y µ e−βt+β

0

√

tξn,0 (y) ∗ φ0 (y).

Then Y (1) (ξ ) is a continuous function of ξ with respect to the norm  · , hence the convergence in law Y (1) (ξn ) → Y (1) (ξ ) holds. Let us apply Theorem 4.9 to the coordinates α ∗ with p = 0, r = m, f = ξn . Also we apply Theorem 4.8 to the other coordinates with r = m − 1 and f = ξn . Then there exists a constant C1 > 0 such that  nλ Z (1)  C1  βξn 2 /2   n (1) − Y (ξ ) e {βξn φ ∗  + ∇φ ∗  + φ ∗ }.  n ≤ m−1 (log n) log n α Since ξn  converges in law, the right-hand side of this equation converges to zero in probability. Therefore, the convergence in law nλ Z (1) → Y (1) (ξ ) (log n)m−1 n holds, which completes the theorem.




Main Theorem 6.2 (Convergence of stochastic complexity) (1) Assume that q(x), p(x|w) and ϕ(w) satisfy the fundamental conditions (I) and (II) with index s = 2. Then the following convergence in law holds: Fn0 − λ log n + (m − 1) log log n √ 
∞
γb dt t λ−1 e−βt+β tξ0 (y) φ0∗ (y)dy. → − log α∗

0

(2) Assume that q(x), p(x|w) and ϕ(w) satisfy the fundamental conditions (I) and (II) with index s = 4. Then the following convergence of expectation holds:   E Fn0 − λ log n + (m − 1) log log n


∞
 √ → −E log γb dt t λ−1 e−βt+β tξ0 (y) φ0∗ (y)dy . α∗

0

Proof of Main Theorem 6.2 (1) From Theorem 6.7 and the fact that −log(·) is a continuous function, the first part is proved by Theorem 5.1.

6.2 Evidence and stochastic complexity

175

(2) For the second part, it is sufficient to prove that An ≡ −log

Zn0 nλ (log n)m−1

is asymptotically uniformly integrable. By using the same decomposition of Zn0 as in the proof of Theorem 6.7, An = −log

 Z (1) nλ Zn(2) nλ  n + . (log n)m−1 (log n)m−1

By eq.(6.18) and Theorem 4.8 with p = 0 and r = m,  β  An ≥ −log exp(βξn 2 /2)ϕ + exp sup |ψn (w)|2 2 K(w)>  ≥ −(β/2) max ξn 2 , sup |ψn (w)|2 + C2 , K(w)>

where C2 is a constant and we used the fact that log(ep + eq ) ≤ max{p, q} + log 2 for arbitrary p, q. On the other hand, by φ(u) > 0 and Theorem 6.5 (4), 
∞
√ dt du exp(−nβu2k + nβuk ξn )uh du. Zn(2) ≥ C1 α

0

Hence, by Theorem 4.8, and An ≤ −log(Zn(2) nλ /(log n)m−1 ), An ≤ βξn 2 /2 − log min |φ| + C3 , where C3 is a constant. By Theorem 5.8, E[ξn 4 ] < ∞, E[ψn 4 ] < ∞. Hence An is asymptotically uniformly integrable. By Theorem 5.5, we obtain
the theorem. Corollary 6.1 (1) Assume that q(x), p(x|w) and ϕ(w) satisfy the fundamental conditions (I) and (II) with index s = 2. Then the following asymptotic expansion holds, Fn = nβSn + λ log n − (m − 1) log log n + FnR , where FnR is a random variable which converges to a random variable in law. (2) Assume that q(x), p(x|w) and ϕ(w) satisfy the fundamental conditions (I) and (II) with index s = 4. Then the following asymptotic expansion of the expectation holds,   E[Fn ] = nβS + λ log n − (m − 1) log log n + E FnR , where E[FnR ] converges to a constant.

176

Singular learning theory

F 0n Regular case

Singular case n Fig. 6.1. Stochastic complexity

Proof of Corollary 6.1 By the definition FnR

= −log

 α∗


γb




∞

dt

du t λ−1 e−βt+β

√

0

tξn,0 (y)

φ0∗ (y)dy,

this corollary is immediately derived from Main Theorem 6.2.




Remark 6.7 (1) Figure 6.1 shows the behavior of the normalized stochastic complexity. If the a priori distribution is positive on K(w) = 0, then the learning coefficient is equal to the real log canonical threshold. Note that the learning coefficient is invariant under a transform p(x|w) → p(x|g(u)), ϕ(w)dw → ϕ(g(u))|g  (u)|du. (2) If a model is regular, λ = d/2 and m = 1, where d is the dimension of the parameter space. Examples of λ and m in several models are shown in Chapter 7. (3) Assume that β = 1. By Main Theorem 6.2 and Theorem 1.2, if the mean of Bayes generalization error Bg has an asymptotic expansion, then    0  − E Fn0 , E[Bg ] = E Fn+1 1 λ . = +o n n However, in general, even if a function f (n) has an asymptotic expansion, f (n + 1) − f (n) may not have an asymptotic expansion. To prove the Bayes generalization error has an asymptotic expansion, we need to show a more precise result as in the following section.

6.3 Bayes and Gibbs estimation

177

6.3 Bayes and Gibbs estimation In the previous section, we studied the asymptotic expansion of the stochastic complexity. In this section, mathematical relations in the Bayes quartet are proved, which are called equations of states in learning. Throughout this section, we assume the fundamental conditions (I) and (II) with index s = 6. Firstly, the main theorems are introduced without proof. Secondly, basic lemmas are prepared. Finally, the main theorems are proved.

6.3.1 Equations of states Assume that random samples X1 , X2 , . . . , Xn are independently taken from the probability distribution q(x)dx. For a given set of random samples Dn = {X1 , X2 , . . . , Xn }, the generalized a posteriori distribution is defined by p(w|Dn ) =

n  1 ϕ(w) p(Xi |w)β , Zn i=1

which can be rewritten as p(w|Dn ) =

1 exp(−nβKn (w))ϕ(w), Zn0

where β > 0 is the inverse temperature. Definition 6.5 (Bayes quartet) Let Ew [·] be the expectation value using p(w|Dn ). Four errors are defined. (1) Bayes generalization error, 

q(X) Bg = EX log . Ew [p(X|w)] (2) Bayes training error, 1 q(Xi ) log . n i=1 Ew [p(Xi |w)] n

Bt =

(3) Gibbs generalization error,



q(X)  Gg = Ew EX log . p(X|w) (4) Gibbs training error, Gt = Ew

n

1 

n

i=1

log

q(Xi )  . p(Xi |w)

This set of four errors is called the Bayes quartet.

178

Singular learning theory

The most important variable in practical applications among them is the Bayes generalization error because it determines the accuracy of the estimation. However, we prove that there are mathematical relations between them. It is shown in Theorem 1.3 that, by using the log density ratio function f (x, w) = log

q(x) , p(x|w)

and the log likelihood ratio function 1 f (Xi , w), Kn (w) = n i=1 n

the Bayes quartet can be rewritten as 

Bg = EX − log Ew [e−f (X,w) ] , 1 − log Ew [e−f (Xi ,w) ], Bt = n i=1 n

Gg = Ew [K(w)], Gt = Ew [Kn (w)]. If the true distribution q(x) is contained in the statistical model p(x|w), then the four errors in the Bayes quartet converge to zero in probability when n tends to infinity. In this section, we show how fast random variables in the Bayes quartet tend to zero. Theorem 6.8 Assume the fundamental conditions (I) and (II) with s = 6. (1) There exist random variables Bg∗ , Bt∗ , G∗g , and G∗t such that, when n → ∞, the following convergences in law hold: nBg → Bg∗ , nBt → Bt∗ ,

nGg → G∗g , nGt → G∗t .

(2) When n → ∞, the following convergence in probability holds: n(Bg − Bt − Gg + Gt ) → 0. (3) Expectation values of the Bayes quartet converge: E[nBg ] → E[Bg∗ ], E[nBt ] → E[Bt∗ ], E[nGg ] → E[G∗g ], E[nGt ] → E[G∗t ].

6.3 Bayes and Gibbs estimation

179

Main Theorem 6.3 (Equations of states in statistical estimation) Assume the fundamental conditions (I) and (II) with s = 6. For arbitrary q(x), p(x|w), and ϕ(w), the following equations hold. E[Bg∗ ] − E[Bt∗ ] = 2β(E[G∗t ] − E[Bt∗ ]),

(6.19)

E[G∗g ] − E[G∗t ] = 2β(E[G∗t ] − E[Bt∗ ]).

(6.20)

Remark 6.8 (1) Main Theorem 6.3 shows that the increases of errors from training to prediction are in proportion to the differences between the Bayes and Gibbs training. We give Main Theorem 6.3 the title Equations of states in statistical estimation, because these hold for any true distribution, any statistical model, any a priori distribution, and any singularities. If two of these errors are measured by observation, then the other two errors can be estimated without any knowledge of the true distribution. (2) From the equations of states, widely applicable information criteria (WAIC) are obtained. See Section 8.3. (3) Although the equations of states hold universally, the four errors themselves strongly depend on a true distribution, a statistical model, an a priori distribution, and singularities. Corollary 6.2 The two generalization errors can be estimated by the two training errors,





E[Bg∗ ] 1 − 2β 2β E[Bt∗ ] = . (6.21) −2β 1 + 2β E[G∗t ] E[G∗g ] Proof of Corollary 6.2 This corollary is directly derived from Main Theorem
6.3. Remark 6.9 (1) From eq.(6.21), it follows that





E[G∗g ] 1 − 2β 2β E[G∗t ] = , −2β 1 + 2β E[Bt∗ ] E[Bg∗ ] which shows that there is symmetry in the Bayes quartet. Theorem 6.9 Assume the fundamental conditions (I) and (II) with index s = 6. When n → ∞, the convergence in probability nGg + nGt −

2λ →0 β

holds, where λ is the learning coefficient. Moreover, E[G∗g ] + E[G∗t ] =

2λ . β

(6.22)

180

Singular learning theory

Corollary 6.3 Assume the fundamental conditions (I) and (II) with s = 6. The following convergence in probability holds, 2λ → 0, β

nBg − nBt + 2nGt − where λ is the learning coefficient. Moreover,

E[Bg∗ ] − E[Bt∗ ] + 2E[G∗t ] =

2λ . β

In particular, if β = 1, E[Bg∗ ] = λ. Proof of Corollary 6.3 This corollary is derived from Theorem 6.8 (1) and 6.9.
Definition 6.6 (Empirical variance) The empirical variance V of the log likelihood function is defined by V =

n  

Ew [(log p(Xi |w))2 ] − ( Ew [log p(Xi |w)] )2 .

(6.23)

i=1

By using the log density ratio function f (x, w) = log(q(x)/p(x|w)), this can be rewritten as n   Ew [ f (Xi |w)2 ] − Ew [f (Xi |w)]2 . (6.24) V = i=1

Theorem 6.10 The following convergences in probability hold: V − 2(nGt − nBt ) → 0,

(6.25)

V − 2(nGg − nBg ) → 0.

(6.26)

There exists a constant ν = ν(β) > 0 such that lim E[V ] =

n→∞

and

2ν(β) β

λ  + 1− β λ  E[Bt∗ ] = − 1 + β

E[Bg∗ ] =

1 ν(β), β 1 ν(β), β

(6.27) (6.28)

E[G∗g ] =

λ + ν(β), β

(6.29)

E[G∗t ] =

λ − ν(β). β

(6.30)

6.3 Bayes and Gibbs estimation

181

Error

Gg Bg O

n

Gt Bt Fig. 6.2. Bayes and Gibbs errors

The constant ν(β) > 0 is a singular fluctuation defined in eq.(6.48), which satisfies an inequality, 0 ≤ ν(β) ≤

Eξ [ξ 2 ] Eξ [ξ 2 ]1/2 ( Eξ [ξ 2 ] + 16λ/β, + 8 8

where ξ  is the maximum value of the random process ξ (u).

Remark 6.10 The behavior of the Bayes quartet is shown in Figure 6.2. From Theorem 6.10, the singular fluctuation ν(β) can be represented in several ways: ν(β) = (1/2)(E[Bg∗ ] − E[Bt∗ ]) = (1/2)(E[G∗g ] − E[G∗t ]) = β (E[G∗g ] − E[Bg∗ ]) = β (E[G∗t ] − E[Bt∗ ]) = lim

n→∞

β E[V ]. 2

From the equations of states, β E[V ] + o(1/n), n β E[Gg ] = E[Gt ] + E[V ] + o(1/n). n E[Bg ] = E[Bt ] +

(6.31) (6.32)

182

Singular learning theory

Using these relations, ν(β) can be estimated from numerical experiments. By definition, ν(β) is invariant under a birational transform p(x|w) → p(x|g(u)), ϕ(w)dw → ϕ(g(u))|g  (u)|du. Remark 6.11 (Regular model case) In singular learning machines, λ is not equal to d/2 in general, and ν(β) depends on β. In a regular statistical model, λ = ν(β) = d/2, which means that, d , 2 d E[Bt∗ ] = − , 2  1 d E[G∗g ] = 1 + , β 2  1 d , E[G∗t ] = − 1 + β 2 E[Bg∗ ] =

which is a special case of Main Theorem 6.3. This result is obtained by the classical asymptotic theory of the maximum likelihood estimator with positive definite Fisher information matrix. Assume that examples are independently taken from q(x) = p(x|w0 ). Let In (w) and I (w) be the empirical and mean Fisher information matrices respectively. The expectation of a function f (w) under the a posteriori distribution is asymptotically given by  2 !  In (w) ˆ 1/2 (w − w) ˆ  dw f (w) exp − nβ 2 ∼ Ew [f (w)] = ,  2 !  In (w) ˆ 1/2 (w − w) ˆ  dw exp − nβ 2 where, for a given symmetric matrix I and a vector v, |I 1/2 v|2 = (v, I v). The √ random variable n(wˆ − w0 ) converges in law to the normal distribution with mean zero and covariance matrix I (w0 )−1 , and the Taylor expansions for w0 and wˆ are given by 

n E[nK(w)] ˆ = E nK(w0 ) + |I (w0 )1/2 (wˆ − w0 )|2 2 d = + o(1) 2

 n ˆ + |In (w) ˆ 1/2 (w0 − w)| ˆ 2 E[nKn (w0 )] = E nKn (w) 2 = 0.

6.3 Bayes and Gibbs estimation

183

Therefore the Gibbs generalization and training errors are given by E[nGg ] = E[Ew [nK(w)]]

n  = E Ew [|I (w) ˆ 1/2 (w − w)| ˆ 2 ] + nK(w) ˆ + o(1) 2 d d + + o(1), = 2β 2 E[nGt ] = E[Ew [nKn (w)]]

n  = E Ew [|I (w) ˆ 1/2 (w − w)| ˆ 2 ] + nKn (w) ˆ + o(1) 2 d d − + o(1). = 2β 2 Consequently, in regular statistical models, λ = ν(β) = d/2.

6.3.2 Basic lemmas In this subsection, some basic lemmas are prepared which are used in the proofs of the above theorems. Note that there are three different expectations. The first is the expectation over the parameter space of the a posteriori distribution. The second is the expectation over random samples Dn = {X1 , X2 , . . . , Xn }. It is denoted by E[ ], which is also used for the expectation over the limit process ξ . The last is the expectation over the test sample X. It is denoted by EX [ ]. For a given constant a > 0, we define an expectation value in the restricted set {w; K(w) ≤ a} by
f (w)e−βnKn (w) ϕ(w)dw K(w)≤a Ew [f (w)|K(w)≤a ] =
. −βnKn (w) e ϕ(w)dw K(w)≤a

The four errors of the Bayes quartet in the restricted region are given by 

Bg (a) = EX − log Ew [e−f (X,w) |K(w)≤a ] , 1 − log Ew [e−f (Xj ,w) |K(w)≤a ], Bt (a) = n j =1 n

Gg (a) = Ew [K(w)|K(w)≤a ], Gt (a) = Ew [Kn (w)|K(w)≤a ].

184

Singular learning theory

Since W is compact and K(w) is a real analytic function, K = sup K(w) w∈W

is finite, therefore Bg (K) = Bg , Bt (K) = Bt , Gg (K) = Gg , Gt (K) = Gt . Also we define ηn (w) for w such that K(w) > 0 by ηn (w) =

K(w) − Kn (w) , √ K(w)

(6.33)

and Ht (a) =

sup

|ηn (w)|2 .

0 0, the following inequalities hold. Bt (a) ≤ Gt (a) ≤ 32 Gg (a) + 12 Ht (a), 0 ≤ Bg (a) ≤ Gg (a), − 14 Ht (a) ≤ Gt (a). (2) In particular, by putting a = K, Bt ≤ Gt ≤ 32 Gg + 12 Ht , 0 ≤ Bg ≤ Gg , − 14 Ht ≤ Gt . Proof of Lemma 6.1 (1) Because Bg (a) is the Kullback–Leibler distance from q(x) to the Bayes predictive distribution using the restricted a priori distribution on K(w) ≤ a, it follows that Bg (a) ≥ 0. Using Jensen’s inequality, Ew [e−f (x,w) |K(w)≤a ] ≥ exp(−Ew [f (x, w)|K(w)≤a ])

(∀x),

6.3 Bayes and Gibbs estimation

185

hence Bg (a) ≤ Gg (a) and Bt (a) ≤ Gt (a). By using the Cauchy–Schwarz inequality,  Kn (w) = K(w) − K(w) ηn (w) ≤ K(w) + 12 {K(w) + ηn (w)2 }.

(6.34)

Therefore Gt (a) ≤ (3Gg (a) + Ht (a))/2. Also  Kn (w) = K(w) − K(w)ηn (w)  ηn (w) 2 ηn (w)2 ≥ K(w) − − 2 4 ≥−

ηn (w)2 . 4

(6.35)

Hence we have Gt (a) ≥ −Ht (a)/4. (2) is immediately derived from (1).




Remark 6.12 (1) By Lemma 6.1, if nHt (a), nGg (a), and nBt (a) are asymptotically uniformly integrable (AUI), then nGt (a) and nBg (a) are also AUI, for an arbitrary a > 0. (2) In Lemma 6.4, we prove nHt () is AUI. In Lemma 6.5, we prove nGg () and nBt () are AUI. In Lemma 6.2, we show nHt is AUI using Lemma 6.4. In Lemma 6.3, we show nGg and nBt are AUI using Lemma 6.5. Then the four errors in the Bayes quartet are all AUI. (3) Based on Theorem 5.4 (3), if E[|Xn |s ] < ∞, then Xns−δ (δ > 0) is AUI. Lemma 6.2 (1) There exists a constant CH > 0 such that E[(nHt )3 ] = CH < ∞. (2) For an arbitrary δ > 0, P (nHt > nδ ) ≤

CH . n3δ

(6.36)

(3) nHt is asymptotically uniformly integrable. Proof of Lemma 6.2 (1) For any  > 0 and a > 0, √

1  1 ·√ n ηn (w) = √ (EX [f (X, w)] − f (Xj , w)) n j =1 K(w) n

is an empirical process and f (x, w) is a real analytic function of w. Therefore 

√ E sup | n ηn (w)|6 < const.  nδ ) and CH = E[(nHt )3 ] ≥ E[(nHt )3 S] ≥ E[S] n3δ .




(3) is immediately derived from (1).

Lemma 6.3 (1) For an arbitrary  > 0, the following convergences in probability hold: n(Bg − Bg ()) → 0, n(Bt − Bt ()) → 0, n(Gg − Gg ()) → 0, n(Gt − Gt ()) → 0. (2) The four errors of the Bayes quartet nBg , nBt , nGg , and nGt are all asymptotically uniformly integrable. Proof of Lemma 6.3 We use the notation,
S0 (f (w)) = f (w) e−nβKn (w) ϕ(w)dw, K(w) 0, β  ! S1 (f (w)) ≤ sup f (w) e−nβ/2 exp nHt , 2 w   ! β S0 (g(w)) ≥ c0 inf g(w) n−d/2 exp − nHt , w 2

6.3 Bayes and Gibbs estimation

187

where c0 > 0 is a constant and d is the dimension of the parameter space. Hence supw f (w) S1 (f (w)) ≤ s(n), S0 (g(w)) inf w g(w) where, by using Theorem 7.2, s(n) =

nd/2 −nβ/2+nβHt e , c0

 Then | log s(n)| ≤ nβ/2 + nβHt + o(n). Let Mn ≡ nj=1 M(Xj )/n. Then E[Mn3 ] ≤ EX [M(X)3 ], EX [M(X)k ]{M(X)>n} ≤ EX [M(X)3 ]/n3−k . (1) Firstly, we study the Bayes generalization error,  Ew [e−f (X,w) ] Ew [e−f (X,w) |K(w)≤ ] 

 S1 (1)  S1 (e−f (X,w) )  = nEX − log 1 + + log 1 + . S0 (e−f (X,w) ) S0 (1)

n(Bg − Bg ()) = nEX − log

Thus 

 S1 (1)  S1 (e−f (X,w) )  + log 1 + n|Bg − Bg ()| ≤ nEX log 1 + S0 (e−f (X,w) ) S0 (1) ≤ nEX [log(1 + s(n) e2M(X) )] + ns(n) = ns(n) + nEX [log(1 + s(n) e2M(X) )]{2M(X)≤nβ/4} + nEX [log(1 + s(n) e2M(X) )]{2M(X)>nβ/4} ≤ ns(n) + ns(n) exp(nβ/4) + nEX [|2M(X)| + | log s(n)|]{2M(X)>nβ/4} . It follows that n(Bg − Bg ()) → 0. Secondly, in the same way, the Bayes training error satisfies n|Bt − Bt ()| ≤

n 

log(1 + s(n) e2M(Xj ) ) + n log(1 + s(n)) ≡ Ln . (6.37)

j =1

We can prove the convergence in mean E[Ln ] → 0 because E[Ln ] = E[Ln ]{Ht ≤β/4} + E[Ln ]{Ht >β/4} ≤ nEX [log(1 + (nd /c0 ) e2M(X)−nβ/4 )] +

nd+1 exp(−nβ/4) + 2nE[Mn + | log s(n)|]{Ht >β/4} . c0

188

Singular learning theory

Thus we obtain n(Bg − Bg ()) → 0. Thirdly, the Gibbs generalization error can be estimated as  S (K(w)) + S (K(w)) nS (K(w))  1 0  0  − n|Gg − Gg ()| ≤ n  S0 (1) + S1 (1) S0 (1) ≤

nS1 (K(w)) nS0 (K(w))S1 (1) + S0 (1) S0 (1)2

≤ 2n K s(n),

(6.38)

which converges to zero in probability. Lastly, in the same way, the Gibbs training error satisfies n|Gt − Gt ()| ≤ 2n s(n) sup |Kn (w)| w

≤ 2n s(n) Mn , which converges to zero in probability. (2) Firstly, from Lemma 6.2, nHt is AUI. Secondly, let us prove nBt is AUI. From eq.(6.37), |nBt | ≤ |nBt ()| + Ln . Moreover, by employing a function 1 b(s) = − log Ew [e−sf (Xj ,w) ], n j =1 n

there exists 0 < s ∗ < 1 such that nBt = nb(1) =

∗ n  Ew [f (Xj , w)e−s f (Xj ,w) ]

j =1

Ew [e−s

∗ f (X

j ,w)

]

Hence the following always holds: |nBt | ≤

n  j =1

sup |f (Xj , w)| ≤ nMn . w

Therefore |nBt | ≤ |nBt ()| + B ∗ , where ∗

B ≡



nMn Ln

(nHt > βn/4) (nHt ≤ βn/4).

.

6.3 Bayes and Gibbs estimation

189

By summing up the above equations, E[|nBt |3/2 ] ≤ E[2|nBt ()|3/2 ] + E[2(B ∗ )3/2 ]. In Lemma 6.5, we prove E[|nBt ()|3/2 ] < ∞. By Lemma 6.2 (2) with δ such that nδ = βn/4, we have P (Ht > β/4) ≤ CH /n3 , hence E[(B ∗ )3/2 ] ≤ E[(B ∗ )3/2 ]{Ht >β/4} + E[(B ∗ )3/2 ]{Ht ≤β/4} 1/2

≤ E[(nMn )3 ]1/2 E[1]{Ht >β/4} 1/2

+ E[(Ln )3 ]{Ht ≤β/4} < ∞. Hence |nBt | is AUI. Lastly, we show nGg is AUI. From eq.(6.38), 0 ≤ nGg ≤ nGg () + 2n s(n) K. Moreover, nGg ≤ nK always, by definition. Therefore nGg ≤ nGg () + K ∗ where 

∗

K ≡  ≤

nK K n s(n)

(nHt > n2/3 ) (nHt ≤ n2/3 ) (nHt > n2/3 ) (nHt ≤ n2/3 ).

nK K e−nβ/3

Then 0 ≤ E[(nGg )3/2 ] ≤ E[2(nGg ())3/2 ] + E[2(K ∗ )3/2 ]. It is proven in Lemma 6.5 that E[(nGg ())3/2 ] < ∞. By Lemma 6.2 (2) with δ = 2/3, we have P (nHt > n2/3 ) ≤ CH /n2 , hence E[(K ∗ )3/2 ] ≤ n3/2 K

3/2 CH

n2

+ Ke−nβ/2 < ∞.

Hence nGg is AUI. Since E[(nHt )3 ] < ∞, E[(nBt )3/2 ] < ∞, and
E[(nGg )3/2 ] < ∞, all four errors are also AUI by Lemma 6.1. Based on Lemma 6.3, Bg (), Bt (), Gg (), and Gt () are the major parts of the four errors when n → ∞. The region in the parameter set to be studied is W = {w ∈ W ; K(w) ≤ }

190

Singular learning theory

for a sufficiently small  > 0. Since W contains singularities of K(w) = 0, we need Theorems 6.1 and 6.5. Let us define the supremum norm by f  = sup |f (u)|. u∈M

There exists an Ls (q)-valued analytic function M  u → a(x, u) ∈ Ls (q) such that, in each local coordinate, f (x, g(u)) = a(x, u) uk , EX [a(X, u)] = uk , K(g(u)) = 0 ⇒ EX [a(X, u)2 ] = 2, EX [a(X)s ] < ∞. We define a(X) = supu∈M |a(X, u)|. An empirical process ξn (u) is defined by eq.(6.11). Then the empirical process satisfies the following lemma. Lemma 6.4 (1) Let s = 6. The empirical process ξn (u) satisfies E[ξn s ] < const. < ∞ E[∇ξn s ] < const. < ∞

 where the constant does not depend on n, and ∇ξn  = dj=1 ∂j ξn . (2) The random variable nHt () is asymptotically uniformly integrable. Proof of Lemma 6.4 (1) This lemma is derived from Theorem 5.8 and fundamental condition (I). (2) is immediately derived from (1).
Let the Banach space of uniformly bounded and continuous functions on M be B(M) = {f (u) ; f  < ∞}. Since M is compact, B(M) is a separable norm space. The empirical process ξn (u) defined on B(M) weakly converges to the tight Gaussian process ξ (u). Definition 6.7 (Integral over parameters) Let ξ (u) be an arbitrary function on M of class C 1 . We define the mean of f (u) over M for a given ξ (u) by 
f (u) Z(u, ξ ) du Euσ [f (u)|ξ ] =

α

[0,b]d


α

[0,b]d

, Z(u, ξ ) du

6.3 Bayes and Gibbs estimation

where

 α

191

is the summation over all coordinates of M, 0 ≤ σ ≤ 1, and Z(u, ξ ) = uh φ ∗ (u) e−βnu

2k

√ +β nuk ξ (u)−σ uk a(X,u)

.

Based on this definition of Euσ [ |ξ ] and the standard form of the log likelihood ratio function, the major parts of the four errors are given by the case σ = 0, 

k (6.39) Bg () = EX − log Eu0 [e−a(X,u)u |ξn ] , 1 k − log Eu0 [e−a(Xj ,u)u |ξn ], n j =1 n

Bt () =

Gg () = Eu0 [u2k |ξn ],  

1  Gt () = Eu0 u2k − √ uk ξn (u)ξn . n

(6.40) (6.41) (6.42)

Lemma 6.5 Assume that k1 > 0, where k1 is the first coefficient of the multiindex k = (k1 , k2 , . . . , kd ), and that 0 ≤ σ ≤ 1. (1) For an arbitrary real analytic function ξ (u) and a(x, u), c1 Euσ [u2k |ξ ] ≤ {1 + ξ 2 + ∂1 ξ 2 n + a(X) + ∂1 a(X)}, c2 σ 3k Eu [u |ξ ] ≤ 3/2 {1 + ξ 3 + ∂1 ξ 3 n + a(X)3/2 + ∂1 a(X)3/2 }, where ∂1 = (∂/∂u1 ), c1 , c2 > 0 are constants. (2) For the empirical process ξn (u),   E Euσ [n u2k |ξn ] < ∞,   E Euσ [n3/2 u3k |ξn ] < ∞. (3) Random variables nGg () ard nBt () are asymptotically uniformly integrable. Proof of Lemma 6.5 (1) Let 0 ≤ p ≤ 3. We use the notation g(u) = uk22 · · · ukdd and h(u) = uh2 2 · · · uhd d , which do not depend on u1 . Then uk = uk11 g(u), uh = uh1 1 h(u), 
2k (uk )p uh e−βnu +f (u) du, Np = α

√

[0,b]d

f (u) = β nuk ξ (u) − σ uk a(X, u).

192

Singular learning theory

By eq.(6.14) and given φ(u) > 0, for each 0 ≤ p ≤ 3, there exists a constant cp > 0 such that 0 ≤ Euσ [upk |ξ ] ≤ cp

Np . N0

By applying partial integration to Np and using q = (p − 2)k1 + h1 + 1, 
2k −1+q −βnu2k +f (u) Np = g(u)p h(u)u1 1 e du α

=−

[0,b]d


α

≤


α

=

[0,b]d

[0,b]d

g(u)p−2 h(u) 2k q ∂1 (u1 ef (u) ) e−βnu du 2βnk1

[0,b]d

(uk )p−2 uh −βnu2k +f (u) e (q + u1 ∂1 f (u)) du. 2βnk1


α

g(u)p−2 h(u) q f (u) 2k u1 e ∂1 (e−βnu ) du 2βnk1

By the relation √ u1 ∂1 f (u) = β nuk (k1 ξ (u) + u1 ∂1 ξ (u)) − σ uk (k1 a(X, u) + u1 ∂1 a(X, u)), and the Cauchy–Schwarz inequality, since u ∈ [0, b]d , there exists B > 0 such that |u1 ∂1 f (u)| ≤

βnk1 u2k + B(ξ 2 + ∂1 ξ 2 2 + a + ∂1 a).

Hence, by B  = max{B, q}, Np Np ≤ + B  (1 + ξ 2 + ∂1 ξ 2 N0 4N0 +a + ∂1 a)

Np−2 . N0

(6.43)

The case p = 2 shows the first half of (1). For the latter half, By eq.(6.43) with p = 3, using B  = 4B  /3, N3 N1 ≤ B  (1 + ξ 2 + ∂1 ξ 2 + a + ∂1 a) . N0 N0

6.3 Bayes and Gibbs estimation

193

Since N1 /N0 ≤ (N2 /N0 )1/2 by the Cauchy–Schwarz inequality, there exists B  > 0 such that N3 ≤ B  (1 + ξ 2 + ∂1 ξ 2 + a + ∂1 a)3/2 . N0   In general ( ni=1 |ai |2 /n)1/2 ≤ ( ni=1 |ai |3 /n)1/3 , so the latter half of (1) is obtained. (2) By Lemma 6.4 and the result of (1) of this lemma, part (2) is immediately derived. (3) By the definition, nGg () = Eu0 [n u2k |ξn ]. Then from (2) of this lemma, nGg () is asymptotically uniformly integrable (AUI). Let us prove nBt () is AUI. By using the notation b(s) = −

n 

log Eu0 [e−s a(Xj ,u)u ], k

j =1

there exists 0 < s ∗ < 1 such that nBt () = b(1) = b(0) + b (0) + 12 b (s ∗ ) = B1 + B2 , where B1 =

n 

Eu0 [a(Xj , u)uk |ξn ],

j =1 n  1  s∗  B2 = Eu [a(Xj , u)2 n u2k |ξn ] . X=Xj 2n j =1

The first term B1 = nGt (). From Lemma 6.1, − 14 nHt () ≤ nGt () ≤ 12 (3nGg () + nHt ()). Therefore E[|B1 |3/2 ] < ∞, because E[(nGg ())3/2 ] < ∞ and E[(nHt ())3 ] < ∞. Moreover, |B2 |3/2 ≤

n   ∗ 3/2 1  a(Xj )3  Eus [n u2k |ξn ] . X=Xj n j =1

By the statements (1) and (2) of this lemma, E[|B2 |3/2 ] < ∞, therefore nBt ()
is AUI.

194

Singular learning theory

Without loss of generality, for each local coordinate, we can assume u = (x, y)  x ∈ Rr , y ∈ Rr (r  = d − r), k = (k, k  ), h = (h, h ), and h1 + 1 hr + 1 h + 1 = ··· = = λα < 1  ≤ · · · . 2k1 2kr 2k1 

We define µ = h − 2k  λα ∈ Rr ; then hi + 1 = −1, 2ki

µi > hi − 2ki 

hence y µ is integrable in [0, b]r . Both λα and r depend on the local coordinate. Let λ be the smallest λα and m be the largest r among the coordinates in which λ = λα . Then (−λ) and m are respectively equal to the largest pole and its order of the zeta function, as is shown in Definition 6.4. Let α ∗ be the index of the set of coordinates which satisfy λα = λ and r = m. As is shown in Lemma  6.6, only coordinates Mα∗ affect the four errors. Let α∗ be the sum of such coordinates. For a given function f (u), we use the notation f0 (y) = f (0, y). Also a0 (X, y) = a(X, 0, y). Definition 6.8 The expectation of a function f (y, t) for a given function ξ (u) on the essential family of local coordinates is defined by 
∞
dt dy f (y, t) Z0 (y, t, ξ ) Ey,t [f (y, t)|ξ ] =

where



dy stands for

α∗

0


α∗

 [0,b]d−m




∞

dt

, dy Z0 (y, t, ξ )

0

dy and

Z0 (y, t, ξ ) = γb y µ t λ−1 e−βt+β

√

t ξ0 (y) ∗ φ0 (y).

Here γb > 0 is a constant defined by eq.(4.16). Lemma 6.6 Let p ≥ 0 be a constant. There exists c1 > 0 such that, for an arbitrary C 1 -class function f (u) and analytic function ξ (u), the following inequality holds: ∗   0 E [(n u2k )p f (u)|ξ ] − Ey,t [t p f0 (y)|ξ ] ≤ D(ξ, f, φ ) , u log n

where c1 e2βξ  φ ∗  {β∇ξ f φ ∗  + ∇(f φ ∗ ) + f φ ∗ } (min φ ∗ )2 2

D(ξ, f, φ ∗ ) ≡ and ∇f  =

 j

∂j f .

6.3 Bayes and Gibbs estimation

195

Proof of Lemma 6.6 Using Z p and Y p in Definition 4.10 and eq.(4.19), we define A, B, and C by  p p ∗ α n Z (n, ξ, f φ ) 0 2k p , A ≡ Eu [(n u ) f (u)|ξ ] =  0 ∗ α Z (n, ξ, f φ )  p p ∗ α ∗ n Y (n, ξ, f φ ) p , B ≡ Ey,t [t f0 (y)|ξ ] =  0 ∗ α ∗ Y (n, ξ, f φ )  p p ∗ α n Y (n, ξ, f φ ) , C≡  0 ∗ α Y (n, ξ, f φ )   where α and α∗ denote the sum of all local coordinates and the sum of coordinates in the essential family respectively. To prove the lemma, it is sufficient to show |A − B| ≤ D(ξ, f, φ ∗ )/ log n. Since |A − B| ≤ |A − C| + |C − B|, we show the inequalities for |A − C| and |C − B| respectively. The set (n, ξ, f φ ∗ ) is omitted for simplicity. Firstly, |A − C| is bounded by  p   Zp Y  α p  − α 0  |A − C| = n  0 Z α αY  p  0  0     p p Y { Y − Z } αZ − αY p  0 + α  α0  0 α =n   Z Z Y α α α    p p p |Y 0 − Z 0 | α |Z − Y | αY p p  0 + n  0 × α 0 . ≤n αZ αY αZ For general ai , bi > 0,

  ai ai  ≤ . bi bi

Therefore, |A − C| ≤

 np |Z p − Y p | Z0

α

+

 np Y p α

Y0

×

 |Y 0 − Z 0 | α

Z0

.

Then by using Theorems 4.8, 4.9, 4.10, there exist constants C1 , C2 > 0 such that C1 eβξ  {β∇ξ f φ ∗  + ∇(f φ ∗ ) + f φ ∗ } log n min φ ∗ 2

|A − C| ≤

C2 e2βξ  φ ∗ {β∇ξ f φ ∗  + ∇(f φ ∗ ) + f φ ∗ } . log n (min φ ∗ )2 2

+

196

Singular learning theory

Secondly,

  Yp p ∗ Y   |C − B| = np  α 0 − α 0 . Y Y ∗ α α

Let us use the simplified notation, Tp =



Y p,

α∗



Up = Then, by

 α

=

 α∗

+



Y p.

α\α ∗ α\α ∗ ,

T p + Up T p   |C − B| = np  0 − 0 0 T +U T ≤

np U p np U 0 T p + . T0 (T 0 )2

By Theorem 4.10, there exists C3 > 0 such that C3 eβξ  φ ∗  C4 e2βξ  φ ∗ 2 + . min φ ∗ (min φ ∗ )2 2

|C − B| ≤

2

By combining two results, we obtain the lemma.




6.3.3 Proof of the theorems In this subsection, we prove the theorems. Definition 6.9 (Explicit representation of the Bayes quartet) Four functionals of a given function ξ (u) are defined by Bg∗ (ξ ) ≡ 12 EX [ Ey,t [a0 (X, y)t 1/2 |ξ ]2 ],

(6.44)

Bt∗ (ξ ) ≡ G∗t (ξ ) − G∗g (ξ ) + Bg∗ (ξ ),

(6.45)

G∗g (ξ ) ≡ Ey,t [t|ξ ],

(6.46)

G∗t (ξ ) ≡ Ey,t [t − t 1/2 ξ0 (y)|ξ ].

(6.47)

Note that these four functionals do not depend on n. If ξ (u) is a random process, then the four functionals are random variables. The singular fluctuation is defined by 

ν(β) = 12 Eξ Ey,t [t 1/2 ξ0 (y)|ξ ] . (6.48)

6.3 Bayes and Gibbs estimation

197

Proof of Theorems 6.8 In this proof, we use the simplified notation Euσ [f (u)] = Euσ [f (u)|ξn ], Ey,t [f (y, t)] = Ey,t [f (y, t)|ξn ], in other words, ‘|ξn ’ is omitted. Firstly we prove the following convergences in probability. nBg () − Bg∗ (ξn ) → 0,

(6.49)

nBt () − Bt∗ (ξn ) → 0,

(6.50)

nGg () − G∗g (ξn ) → 0,

(6.51)

nGt () −

G∗t (ξn )

→ 0.

(6.52)

Based on eq.(6.41), eq.(6.46), and Lemma 6.6 with p = 1,   |nGg () − G∗g (ξn )| = Eu0 [nu2k ] − Ey,t [t] ≤

D(ξn , 1, φ ∗ ) . log n

Because the convergence in law ξn → ξ holds, eq.(6.51) is obtained. Also, based on eq.(6.42), eq.(6.47), and Lemma 6.6 with p = 1, 12 ,   √ |nGt () − G∗t (ξn )| = Eu0 [nu2k − nuk ξn ] − Ey,t [t − t 1/2 ξ0 ] ≤

D(ξn , 1, φ ∗ ) D(ξn , ξn , φ ∗ ) + . log n log n

Because the convergence in law ξn → ξ holds, eq.(6.52) is obtained. Let us prove eq.(6.49); we define 

k bg (σ ) ≡ EX −log Eu0 [e−σ a(X,u)u ] , then it follows that nBg () = nbg (1) and there exists 0 < σ ∗ < 1 such that nBg () = nbg (0) + nbg (0) + = nEu0 [u2k ] −

n  n bg (0) + bg(3) (σ ∗ ) 2 6

(6.53)

n EX Eu0 [a(X, u)2 u2k ] 2

n + EX Eu0 [a(X, u)uk ]2 + 16 nbg(3) (σ ∗ ), 2

(6.54)

198

Singular learning theory

where we used bg (0) = 0, and EX [a(X, u)] = uk hence bg (0) = Eu0 [u2k ]. The first term on the right-hand side of eq.(6.54) is equal to nGg (). By Lemma 6.6, the following convergence in probability   nEX E 0 [a(X, u)2 u2k ] − EX Ey,t [a0 (X, y)2 t] u

≤

EX [D(ξ, a(X, u)2 , φ ∗ )] →0 log n

(6.55)

holds, where D(β, ξ, a(X, u), φ ∗ ) is defined as in Lemma 6.6. Since EX [a0 (X, y)2 ] = 2, the sum of the first two terms on the right-hand side of eq.(6.54) converges to zero in probability. For the third term, by using the notation ρ(u, v) = EX [a(X, u)a(X, v)], ρ0 (u, y) = ρ(u, (0, y)), ρ00 (y  , y) = ρ((0, y  ), (0, y)), and applying Lemma 6.6,   nEX E 0 [a(X, u)uk ]2 − Ey,t [a0 (X, y)t 1/2 ]2  u √

√ !  ≤  nEu0 uk nEv0 [ρ(u, v)v k ] − Ey,t [ρ0 (u, y)t 1/2 ]  

! √  + Ey,t t 1/2 nEu0 [ρ0 (u, y)uk ] − Ey  ,t  [ρ00 (y  , y)(t  t)1/2 ]  √ c1 n 0 k E [u ] D(ξn , ρ(·, ·), φ ∗ ) ≤ log n u c1 (6.56) Ey,t [t 1/2 D(ξn , ρ(·, y), φ ∗ )]. + log n Equation (6.56) converges to zero in probability by Lemma 6.5. Therefore the difference between the third term and Bg∗ (ξn ) converges to zero in probability. For the last term, we have    (3) ∗  nb (σ ) = nEX E σ ∗ [a(X, u)3 u3k ] + 2E σ ∗ [a(X, u)uk ]3 g u u  ∗ ∗  −3Euσ [a(X, u)2 u2k ]Euσ [a(X, u)u]  

∗ ≤ 6nEX a(X)3 Euσ [u3k ] ,

6.3 Bayes and Gibbs estimation

199

where we used H¨older’s inequality. By applying Lemma 6.5,

 (3) ∗  nb (σ ) ≤ 6c2 EX a(X)3 {1 + ξn 3 + ∂ξn 3 g n1/2  + a(X)3/2 + ∂a(X)3/2 } ,

(6.57)

which shows nbg(3) (σ ∗ ) converges to zero in probability, because the fundamental condition (I) with index s = 6 is assumed. Hence eq.(6.49) is proved. We proceed to the proof of eq.(6.50). An empirical expectation Ej∗ [ ] is simply denoted by 1 f (Xj ). n j =1 n

Ej∗ [f (Xj )] = By defining

bt (σ ) = Ej∗ [− log Eu0 [e−σ a(Xj ,u)u ]], k

it follows that nBt () = nbt (1) and there exists 0 < σ ∗ < 1 such that nBt () = nGt () −

n ∗ 0 E E [a(Xj , u)2 u2k ] 2 j u

n + Ej∗ Eu0 [a(Xj , u)uk ]2 + 16 nbt(3) (σ ∗ ). 2

(6.58)

Then, by applying Lemma 6.5, nbt(3) (σ ∗ ) converges to zero in probability in the same way as eq.(6.57). In fact,  (3) ∗   ∗  σ ∗ nbt (σ ) = E E [a(Xj , u)3 u3k ] + 2E σ ∗ [a(Xj , u)uk ]3 j u u  ∗ ∗  −3Euσ [a(Xj , u)2 u2k ]Euσ [a(Xj , u)u]  

∗ ≤ 6nEj∗ a(Xj )3 Euσ [u3k ] . By applying Lemma 6.5, using the fundamental condition (7) with 5 = 6,

 (3) ∗  nbt (σ ) ≤ 6c2 E ∗ a(Xj )3 {1 + ξn 3 + ∂ξn 3 j 1/2 n  + a(Xj )3/2 + ∂a(Xj )3/2 } ,

(6.59)

which converges to zero in probability. By the same methods as eq.(6.55) and eq.(6.56), replacing respectively EX [a(X)2 ] with Ej∗ a(Xj )2  and ρ(u, v)

200

Singular learning theory

with ρn (u, v) = Ej∗ a(Xj , u)a(Xj , v), n     Ej∗ Eu0 [a(Xj , u)2 u2k ] − G∗g (ξn ) 2  n   ≤ Ej∗ Eu0 [a(Xj , u)2 u2k ] − EX Eu0 [a(X, u)2 u2k ] 2  n   +  EX Eu0 [a(X, u)2 u2k ] − G∗g (ξn ) 2 n  Eu0 [u2k ] ≤ sup |Ej∗ a(Xj , u) − EX a(X, u)| 2 u n    +  EX Eu0 [a(X, u)2 u2k ] − G∗g (ξn ), 2 which converges to zero by Lemma 6.5 and eq.(6.55). In the same way, the following convergence in probability holds, 1 ∗ 0 E E [a(Xj , u)uk ]2 2 j u

− Bg∗ (ξn ) → 0,

and therefore the following convergence in probability also holds: nBt () − nGt () + nGg () − nBg () → 0.

(6.60)

Therefore eq.(6.50) is obtained. By combining eq.(6.49)–eq.(6.52) with Lemma 6.3 (2), we obtain the following convergences in probability: nBg − Bg∗ (ξn ) → 0,

(6.61)

nBt − Bt∗ (ξn ) nGg − G∗g (ξn ) nGt − G∗t (ξn )

→ 0,

(6.62)

→ 0,

(6.63)

→ 0.

(6.64)

The four functionals Bg∗ (ξ ), Bt∗ (ξ ), G∗g (ξ ), and G∗t (ξ ) are continuous functions of ξ ∈ B(M). From the convergence in law of the empirical process ξn → ξ , these convergences in law Bg∗ (ξn ) → Bg∗ (ξ ), Bt∗ (ξn ) → Bt∗ (ξ ), G∗g (ξn ) → G∗g (ξ ), G∗t (ξn ) → G∗t (ξ ), are derived. Therefore Theorem 6.8 (1) and (2) are obtained. Theorem 6.8 (3) is shown because the four errors are asymptotically uniformly integrable by Lemma 6.3.


6.3 Bayes and Gibbs estimation

201

Proof of Main Theorem 6.3 Before proving the theorem, we introduce a property of a Gaussian process. We use the notation


∞

Sλ (a) =


0

∗

du =



√

dt t λ−1 e−βt+aβ t ,


γb

dx dy δ(x) y µ ,

α∗


Z(ξ ) =

du∗ Sλ (ξ (u)),

where u = (x, y). Then, by Definition 6.9,  1  du∗ a(X, u)Sλ (ξ (u)) 2  E E , E[Bg∗ ] = X 2β 2 Z(ξ ) E[Bt∗ ] = E[Bg∗ ] + E[G∗t ] − E[G∗g ],  1 du∗ Sλ (ξ (u))  ∗ , E[Gg ] = 2 E β Z(ξ )   1 du∗ Sλ (ξ (u))  1 du∗ ξ (u)Sλ (ξ (u))  ∗ − E . E[Gt ] = 2 E β Z(ξ ) β Z(ξ ) Let ν = ν(β) be the singular fluctuation in eq.(6.48) and A be a constant,  1 du∗ ξ (u)Sλ (ξ (u))  E , (6.65) ν= 2β Z(ξ )  1 du∗ Sλ (ξ (u))  A = 2E . (6.66) β Z(ξ ) By eq.(5.24) in Theorem 5.11 and ρ(u, u) = EX [a(X, u)2 ] = 2 with u = (0, y), we have 1 ν, β  1 ν, E[Bt∗ ] = A − 2 + β E[Bg∗ ] = A −

(6.67) (6.68)

E[G∗g ] = A,

(6.69)

E[G∗t ] = A − 2ν.

(6.70)

By combining these equations to eliminate A and ν, we obtain two equations
which do not contain either A or ν, giving Main Theorem 6.3.

202

Singular learning theory

Proof of Theorem 6.9 By eq.(5.22) with a = ξn (u) 2 Sλ (ξn (u)) 1 ξn (u)Sλ (ξn (u)) 2λ − − = 0. β 2 Sλ (ξn (u)) β Sλ (ξn (u)) β By the definitions of G∗g (ξn ) and G∗t (ξn ), G∗g (ξn ) + G∗t (ξn ) −

2λ = 0. β

By the convergences in law nGg → G∗g (ξ ) and nGt → G∗t (ξ ), Theorem 6.9 is
obtained. Proof of Theorem 6.10 Let ν = ν(β) be the singular fluctuation in eq.(6.48). By eq.(5.23) in Theorem 5.11, eq.(6.66), and eq.(6.65), A=

λ + ν(β). β

Hence from eqs.(6.67)–(6.70), we obtain eqs.(6.27)–(6.30). From the definition in eq.(6.24), V = nEj∗ Eu0 [a(Xj , u)2 u2k ] − nEj∗ Eu0 [a(Xj , u)uk ]2 + op (1), where op (1) is a random variable which converges to zero in probability. Then, based on eq.(6.58) in the proof of Theorem 6.8, the following convergence in probability holds, V − 2(G∗t (ξn ) − Bt∗ (ξn )) → 0, which gives eqs.(6.25) and (6.26). Therefore V converges in law and is asymptotically uniformly integrable, so, when n → ∞, E[V ] →

2ν(β) . β

Let us introduce an expectation   defined by √  ∗ du f (u, t) t λ−1 e−βt+β tξ (u) √ f (u, t) = .  du∗ t λ−1 e−βt+β tξ (u) Then A = Eξ [t], √ ν(β) = 12 Eξ [ tξ (u)].

(6.71)

6.4 Maximum likelihood and a posteriori

203

By using the Cauchy–Schwarz inequality, √ Eξ [ tξ (u)] ≤ Eξ [t]1/2 Eξ [ξ (u)2 ]1/2 √ ≤ A Eξ [ξ 2 ]1/2 . By combining this inequality with

√ A λ λ Eξ [ξ 2 ]1/2 , A = + ν(β) ≤ + β β 2

we obtain √ Eξ [ξ 2 ]1/2 ( A≤ + Eξ [ξ 2 ]/16 + λ/β, 4




which completes the proof.

Remark 6.13 (Singular fluctuation) By using the expectation notation defined in eq.(6.71) we can represent the variance of a0 (X, y), √ ν(β) = 12 Eξ [ tξ0 (y)] 2 * )√ *2 

) √ β ta0 (X, y) − ta0 (X, y) . = Eξ EX 2 Note that a(x, w) =

log(q(x)/p(x|w)) − K(w) . √ K(w)

Although a(x, w) is not well defined at a singularity in the original parameter space, it can be made well-defined on the manifold by resolution of singularities. Both the real log canonical threshold λ and the singular fluctuation ν(β) determine the asymptotic behavior of a statistical model. In regular statistical models, λ = ν(β) = d/2, where d is the dimension of the parameter space, whereas, in singular statistical models, λ and ν(β) are different from d/2 in general.

6.4 Maximum likelihood and a posteriori In this section, we study the estimator wˆ which minimizes −

n  i

log p(Xi |w) + an σ (w)

204

Singular learning theory

in a compact set W , which is equal to the parameter that minimizes Rn0 (w) = nKn (w) + an σ (w). We assume that σ (w) is a C 2 -class function of w in an open set which contains W , and that {an ≥ 0} is a nondecreasing sequence. We can assume σ (w) ≥ 0 without loss of generality. If an = 0 for arbitrary n, then wˆ is called the maximum likelihood estimator (MLE) and if an = 1 for arbitrary n and σ (w) = − log ϕ(w), where ϕ(w) is an a priori probability density function, then wˆ is called the maximum a posteriori estimator (MAP). The generalization and training errors are respectively defined by Rg = K(w), ˆ ˆ Rt = Kn (w). Although the MAP employs an a priori distribution, its generalization error is quite different from that of Bayes estimation. To study the ML or MAP method, we have to analyze the geometry of the parameter space. Let us assume the fundamental conditions (I) and (II) with index s = 4. We prove that, for arbitrary  > 0, P (K(w) ˆ > ) is sufficiently small that it does not affect the asymptotic generalization and training errors. To study the event K(w) ˆ ≤ , we use the resolution of singularities and the standard form of the log likelihood ratio function. Then the Kulback–Leibler distance becomes a normal crossing function defined on a local coordinate [0, b]d of a manifold 2kr 1 2k2 u2k = u2k 1 u2 · · · ur ,

where r is an integer which satisfies 1 ≤ r ≤ d and k1 , k2 , . . . , kr > 0 are natural numbers. Without loss of generality, we can assume that b = 1. For a given u ∈ [0, 1]d , the integer a is defined by the number (1 ≤ a ≤ r) which satisfies u2 u2a ≤ i ka ki

(i = 1, 2, . . . , r).

(6.72)

Intuitively, a is the number on the axis which is farthest from the given point u. A map [0, 1]d  u → (t, v), where t ∈ R1 , v = (v1 , v2 , . . . , vd ) ∈ Rd , is defined by t = u2k , $( u2i − (ki /ka )u2a vi = ui

(1 ≤ i ≤ r) (r < i ≤ d).

6.4 Maximum likelihood and a posteriori

u2

205

u12 – (u*1)2 u22 – (u2* ) 2 = 2k1 2k 2 Perpendicular

(u , u ) * 1

* 2

u12k u22k = t 1

2

u1

O v1 = ((u 1* ) 2

2

1

* 2 1/2 2

))

Fig. 6.3. Normal crossing coordinate

Then, by definition, va = 0. The set V is defined by V ≡ {v = (v1 , v2 , . . . , vd ) ∈ [0, 1]d ; v1 v2 · · · vr = 0}. Then [0, 1]d  u → (t, v) ∈ T × V is a one-to-one map as in Figure 6.3. Under this correspondence u is identified with (t, v). Remark 6.14 Note that, if a surface parameterized by t, 2kr 1 2k2 u2k 1 u2 · · · ur = t,

(6.73)

and the curve parameterized by (u∗1 , u∗2 , . . . , u∗r ), u21 − (u∗1 )2 u2 − (u∗2 )2 u2 − (u∗r )2 = 2 = ··· = r , 2k1 2k2 2kr

(6.74)

have a crossing point, then they are perpendicular to each other in the restricted space Rr at the crossing point. In fact, the perpendicular vector of eq.(6.73) and the tangent vector at (u1 , u2 , . . . , ur ) of eq.(6.74) are equal to each other, k

1

u1

,

k1 kr  ,..., . u2 ur

The map u → (t, v) can be understood as a function from a point (u∗1 , u∗2 , . . . , u∗r ) to the crossing point of eq.(6.74) and u2k = 0. It is equal to the limit point of steepest descent dynamics, eq.(1.35).

206

Singular learning theory

Theorem 6.11 Let f (u) be a function of class C 1 which is defined on an open set that contains [0, 1]d . Then as a function of (t, v), f (t, v) satisfies ∗

|f (t, v) − f (0, v)| ≤ C t 1/k ∇f 

(0 ≤ t < 1),

where k ∗ = 2(k1 + · · · + kr ),

 ∂f    ∇f  = sup max  (u), u∈[0,1]d 1≤j ≤d ∂uj

and C > 0 is a constant which does not depend on t, s, and f . Proof of Theorem 6.11 Let u = (t, v) and u = (0, v). It t > 0 then the Jacobian determinant of the map u → (t, v) is not equal to zero. There exists u∗ ∈ [0, 1]d such that r  ∂f     |f (t, v) − f (0, v)| ≤ |uj − uj | (u∗ ). ∂u j j =1 Hence |f (t, v) − f (0, v)| ≤ ∇f 

r 

|uj − uj |,

j =1

where ∇f  =

 ∂f    max  . d u∈[0,1] ∂uj j =1

d 

If j = a then |uj − uj | = ua . If j = a, then u2a /ka ≤ u2j /kj ,  !1/2   |uj − uj | = uj − u2j − (kj /ka )u2a =

(kj /ka )u2a

uj + u2j − (kj /ka )u2a  ≤ kj /ka ua .

!1/2

Hence there exists C  > 0 such that |f (t, v) − f (0, v)| ≤ C  ∇f ua . On the other hand by eq.(6.72) there exists C  > 0 such that ∗

t = u2k ≥ C  (ua )2k , which completes the theorem.




6.4 Maximum likelihood and a posteriori

207

Theorem 6.12 Assume the fundamental conditions (I) and (II) with index s (s ≥ 6) and that an /n → 0 (n → ∞). Let ψn (w) be an empirical process on {w; K(w) > }, ψn (w) =

n  K(w) − f (Xi , w) , √ nK(w) i=1

and ψn  = sup |ψn (w)|. K(w)>

Then the following hold. (1) For a given  > 0, there exists a constant C > 0, such that, for arbitrary n ≥ 1, C , ns/2 C P (K(w) ˆ > ) ≤ s/2 . n

P (ψn 2 > n) ≤

(2) For a given  > 0, there exists a constant C  > 0, such that, for arbitrary n ≥ 1, E[ψn 2 ]{ψn 2 >n} ≤ E[nK(w)] ˆ {K(w)>} ˆ

C

, ns/2−1 C ≤ s/2−1 . n

Proof of Theorem 6.12 (1) From Theorem 5.8, E[ψn s ] = C < ∞, and it follows that C ≥ E[ψn s ]{ψn 2 >n} ≥ (n)s/2 P (ψn 2 > n). Therefore P (ψn 2 > n) ≤

C . (n)s/2

By the Cauchy–Schwarz inequality,  Rn0 (w) = nK(w) − nK(w) ψn (w) + an σ (w) ≥ 12 (nK(w) − ψn 2 ) + an σ (w).

(6.75)

208

Singular learning theory

A parameter w0 in the set of true parameters satisfies K(w0 ) = 0, hence Rn0 (w0 ) = an σ (w0 ). Therefore, by the definition of w, ˆ Rn0 (w) ˆ ≤ Rn0 (w0 ), and consequently 1 (nK(w) ˆ 2

− ψn 2 ) + an σ (w) ˆ ≤ an σ (w0 ).

Hence, if ψn 2 ≥ n, there exists a constant c1 > 0 such that nK(w) ˆ ≤ ψn 2 + 4an σ  ≤ c1 ψn 2 . Because an /n → 0, P (K(w) ˆ > ) ≤ P (c1 ψn 2 > n ) c2 ≤ . (n)s/2 (2) By using the above results, E[ψn 2 ]{ψn 2 >n} ≤

∞ 

E[ψn 2 ]{j n} ≤

∞ 

E[nK(w)] ˆ {j n 0, an /np → 0 (n → ∞). Let ξn (w) be an empirical process on {w; K(w) < }, 1  {a(Xi , u) − EX [a(X, u)]}, ξn (w) = √ n i=1 n

and ξn  = sup |ξn (u)|. u∈M

Then the following hold. (1) For a given 0 < δ < 1, there exists a constant C > 0, such that, for arbitrary n ≥ 1, C , nsδ/2 C ˆ > nδ ) ≤ sδ/2 , P (nK(g(u)) n P (ξn 2 > nδ ) ≤

ˆ where uˆ is defined by wˆ = K(g(u))). (2) For a given 0 < δ < 1, there exists a constant C  > 0, such that, for arbitrary n ≥ 1, E[ξn 2 ]{ψn 2 >nδ } ≤ δ} ˆ {nK(g(u))>n E[nK(g(u))] ˆ

C

, nδ(s/2−1) C ≤ δ(s/2−1) . n

Proof of Theorem 6.13 This theorem is proved in the same way as the previous theorem. Let σ (u) ≡ σ (g(u)). (1) From Theorem 5.8, E[ξn s ] = C < ∞, and it follows that C ≥ E[ξn s ]{ξn 2 >nδ } ≥ nsδ/2 P (ξn 2 > nδ ). Therefore P (ξn 2 > nδ ) ≤

C . nsδ/2

(6.76)

210

Singular learning theory

By using K(g(u)) = u2k and the Cauchy–Schwarz inequality, Rn0 (g(u)) = nu2k − uk ξn (u) + an σ (u) ≥ 12 (nu2k − ξn 2 ) + an σ (u). A parameter u0 in the set of true parameters satisfies K(g(u0 )) = u2k 0 = 0, hence Rn0 (g(u0 )) = an σ (u0 ). ˆ Rn0 (u) ˆ ≤ Rn0 (w0 ), Therefore, by the definition of u, 1 (nuˆ 2k 2

ˆ ≤ an σ (u0 ). − ξn 2 ) + an σ (u)

Hence, if ξn 2 ≥ nδ , since an is smaller than any power of np (p > 0), there exists a constant c1 > 0 such that nuˆ 2k ≤ ξn 2 + 4an σ  ≤ c1 ξn 2 . Therefore ˆ > nδ ) ≤ P (c1 ξn 2 > nδ ) P (nK(g(u))) c2 ≤ sδ/2 . n (2) By using the above results, E[ξn  ]{ξn 2

2 >nδ }

≤

∞ 

E[ξn 2 ]{j nδ n E[nK(g(u))] ˆ

∞ 

ˆ {j nδ 0, lim

n→∞

an = 0. np

Let M = {Mα } be the manifold found by resolution of singularities and its local coordinate. (1) If an ≡ 0, then

 2  lim nE[Rg ] = 14 E max max max{0, ξ (u)} , n→∞

α

u∈Mα0



 2  lim nE[Rt ] = − 14 E max max max{0, ξ (u)} ,

n→∞

α

u∈Mα0

where maxα shows the maximization for local coordinates and Mα0 = {u ∈ Mα ; K(g(u)) = 0}. (2) If limn→∞ an = a ∗ , then

2   lim nE[Rg ] = 14 E max max{0, ξ (u∗ )} , n→∞ α

2   lim nE[Rt ] = − 14 E max max{0, ξ (u∗ )} , n→∞

α

∗

where u is the parameter in Mα0 that maximizes 1 4

max{0, ξ (u)}2 − a ∗ σ (g(u)).

(3) If limn→∞ an = ∞, then

 2  lim nE[Rg ] = 14 E max max max{0, ξ (u)} ,

n→∞

α

u∈Mα00



 2  lim nE[Rt ] = − 14 E max max max{0, ξ (u)} ,

n→∞

α

u∈Mα00

where Mα00 is the set of parameters which minimizes σ (g(u)) in the set Mα0 . Proof of Main Theorem 6.4 Let  > 0 be a sufficiently small constant. The proof is divided into several cases. ˆ ≤ c1 ψn 2 . The Case (A), ψn 2 > n. By the proof of Theorems 6.12, nK(w) generalization error of the partial expectation is bounded by Theorem 6.12, E[nK(w)] ˆ {ψn 2 >n} ≤

C , n2

(6.77)

212

Singular learning theory

Also the training error of the partial expectation is ˆ {ψn 2 >n} ≤ 12 E[3nK(w) ˆ + ψn 2 ]{ψn 2 >n} ≤ E[nKn (w)]

C  . n2

(6.78)

Therefore the event ψn 2 > n does not affect the generalization and training errors asymptotically. Case (B), ψn 2 ≤ n. As is shown by the proofs of Theorems 6.12 and 6.13, if ˆ ≤ c1 n. Let us use resolution of singularities and Main ψn 2 ≤ n, then K(w) Theorem 6.1. The function to be minimized, which is called a loss function, is √ Rn0 (g(u)) = nu2k − n uk ξn (u) + an σ (g(u)), where u ∈ [0, 1]d . By using parameterization (t, v) ∈ T × S of each local coordinate, the loss function is given by √ Rn0 (g(t, v)) = nt 2 − n t ξn (t, v) + an σ (t, v), where we use the notation K(t, v) = K(g(t, v)) = t 2 , Kn (t, v) = Kn (g(t, v)), Rn0 (t, v) ≡ Rn0 (g(t, v)), σ (t, v) ≡ σ (g(t, v)). Note that, even if the optimal parameter is on the boundary of [0, 1]d , it asymptotically does not affect the value t because, sufficiently near the point (0, v), the surface v = const. can be taken perpendicular to the boundary. The loss function is rewritten as √ Rn0 (t, v) = nt 2 − n t ξn (0, v) + an σ (0, v) + R1 (t, v), where

 √  R1 (t, v) = − n t ξn (t, v) − ξn (0, v) + an (σ (t, v) − σ (0, v)).

ˆ < c1 ξn 2 , Case (B1), ξn 2 > nδ (0 < δ ≤ 1). By Theorem 6.13 and nK(u) c5 ˆ {ξn 2 >nδ } ≤ δ(s/2−1) , E[nK(u)] n and ˆ {ξn 2 >nδ } ≤ (1/2)E[3nK(w) E[nKn (u)] ˆ + ξn 2 ]{ξn 2 >nδ } c6 ≤ δ(s/2−1) . n

(6.79) (6.80)

Therefore the event ξn 2 > nδ (δ > 0) does not affect the generalization and training errors asymptotically.

6.4 Maximum likelihood and a posteriori

213

ˆ = ntˆ2 is not larger than c7 nδ , Case (B2), ξn 2 ≤ nδ (δ > 0). We know nK(u) hence tˆ ≤ c8 n(δ−1)/2 . Thus we can restrict t in the region, Tδ ≡ {0 ≤ t ≤ c8 n(δ−1)/2 }. By using Theorem 6.11, R1  ≡ sup |R1 (t, v)| t∈Tδ

≤ sup{n1/2 t 1+k0 ∇ξn  + an t k0 ∇σ } t∈Tδ

≤ c9 n−δ/2 ∇ξn  + c10 an nk0 (δ−1)/2 ∇σ ,

(6.81)

where k0 = 1/k ∗ . Therefore R1  → 0 in probability. We need to minimize √ Rn0 (t, v) = nt 2 − n t ξn (0, v) + an σ (0, v) + R1 (t, v) in Tδ × V . For a given v, the parameter t that minimizes Rn0 (t, v) is denoted by t(v). Case (B2-1), ξn (0, v) ≤ 0. If ξn (0, v) ≤ 0 then Rn0 (t(v), v) is not larger than the special case t = 0, Rn0 (t(v), v) ≤ an σ (0, v) + R1 . On the other hand, by removing the nonnegative term, Rn0 (t(v), v) ≥ an σ (0, v) − R1 . Therefore |nt(v)2 −

√

n t(v) ξn (0, v)| ≤ 2R1 .

Moreover, since ξn (0, v) ≤ 0, |nRg | ≤ 2R1 ,

(6.82)

|nRt | ≤ 2R1 .

(6.83)

Case (B2-2), ξ (0, v) > 0. We have √ Rn0 (t, v) = ( nt − ξn (0, v)/2)2 − 14 ξn (0, v)2 + an σ (0, v) + R1 (t, v). √ Then Rn0 (t(v), v) is not larger than the special case t = ξn (0, v)/(2 n), Rn0 (t(v), v) ≤ − 14 ξn (0, v)2 + an σ (0, v) + R1 .

(6.84)

On the other hand, by removing the nonnegative term, Rn0 (t(v), v) ≥ − 14 ξn (0, v)2 + an σ (0, v) − R1 .

(6.85)

214

Singular learning theory

Therefore √ ( n t − ξn (0, v)/2)2 ≤ 2R1 , which means that ξn (0, v)2 /4 − 2R1  ≤ nRg ≤ ξn (0, v)2 /4 + 2R1 ,

(6.86)

−ξn (0, v) /4 − 2R1  ≤ nRt ≤ −ξn (0, v) /4 + 2R1 .

(6.87)

2

2

Then by using the convergence in law ξn (u) → ξn (u), and eq.(6.84) and eq.(6.85), the minimizing procedure for v is divided into three cases. By comparing −ξ (0, v)2 /4 with an σ (0, v), we have following results. (1) If an ≡ 0, the following convergences in probability hold: nRg → (1/4) max max max{0, ξ (u)2 }, α

u∈Mα0

nRt → −(1/4) max max max{0, ξ (u)2 }. α

u∈Mα0

(3) If an → ∞, the following convergences in probability hold: nRg → (1/4) max max max{0, ξ (u)2 }, α

u∈Mα00

nRt → −(1/4) max max max{0, ξ (u)2 }. α

u∈Mα00

(2) If limn an = a ∗ , the following covergences in probability hold: ∗

nRg → (1/4) max max max{0, ξ (u)2 }, α

u∈Mα0 ∗

nRt → −(1/4) max max max{0, ξ (u)2 }, α

u∈Mα0

∗

where max shows the maximization of (1/4)ξ (u)2 − a ∗ σ (u) in the set Mα0 . u∈Mα0

Lastly, from eqs.(6.77), (6.78), (6.79), (6.80), (6.82), (6.83), (6.86), (6.87), and E[R1 2 ] < ∞, both nRg and nRt are asymptotically uniformly integrable,
which completes Main Theorem 6.4. Corollary 6.4 Let wˆ be the maximum likelihood or a posteriori estimator. (1) Assume that q(x), p(x|w) and ϕ(w) satisfy the fundamental conditions (I) and (II) with index s (s ≥ 6). Then Fn = −

n  i=1

log p(Xi |w) ˆ + λ log n − (m − 1) log log n + FnMR ,

6.4 Maximum likelihood and a posteriori

215

where FnMR is a random variable which converges to a random variable F MR in law. (2) Assume that q(x), p(x|w) and ϕ(w) satisfy the fundamental conditions (I) and (II) with index s (s ≥ 6). Then the following convergence of expectation holds, ˆ + λ log n − (m − 1) log log n + E[FnMR ], E[Fn ] = nEX [E[log p(X|w)]] where E[RnMR ] → E[F MR ]. Proof of Corollary 6.4 From Main Theorem 6.4, n  i=1

log

q(Xi ) p(Xi |w) ˆ

converges in law. Its expectation also converges. This corollary is immediately
derived from Main Theorem 6.2. Remark 6.16 (1) In the equation E[Rg ] = −E[Rt ] + o

1 n

the generalization error is represented by the training error; however, the lefthand side contains the entropy +1, whereas the right-hand side has entropy −1. Therefore an information criterion cannot be directly derived from this equation. To construct an information criterion, we need a constant C > 0 such that 1 2C E[Rg ] = E[Rt ] + +o . n n In a regular statistical model C = d/2; however, in a singular model, it depends on the true distribution and a statistical model. (2) If the set of parameters is not compact, then |w| = ∞ may be an analytic singularity. For the behavior of the maximum value of the random process and its application to the maximum likelihood training errors, see, for example, [20, 36, 113]. Although several results were obtained on the asymptotic behavior of the training errors, it is still difficult to know their generalization errors. It is conjectured that the asymptotic generalization error of the maximum likelihood is very large. Remark 6.17 (Phase transition) It seems that, when β → ∞, the Bayes and Gibbs generalization errors converge to that of the maximum likelihood estimation. However, such convergence does not hold in general, even if the set of

216

Singular learning theory

parameters is compact. In Gibbs and Bayes estimations, the main part of the a posteriori distribution is contained in the essential coordinates which minimize λ and maximize its order. However, in the maximum likelihood estimation, such a restriction is not introduced. Therefore, the asymptotic generalization and training errors may not be continuous at β → ∞. Such a phenomenon is called a phase transition in statistical physics. From the viewpoint of statistical physics, a singular learning machine has phase transition at β = ∞, in general.

7 Singular learning machines

Singular learning machines are now being used in artificial intelligence, pattern recognition, robotic control, time series prediction, and bioinformatics. In order to build the foundation on which their learning processes are understood, we need to clarify the effect of singularities. In this chapter, we study the phenomenon caused by singularities in several concrete learning machines.

7.1 Learning coefficient For a given set of the true probability density function q(x), a learning machine p(x|w), and an a priori probability density function ϕ(w), the zeta function is defined by
ζ (z) = K(w)z ϕ(w) dw, where K(w) is the Kullback–Leibler distance,
q(x) K(w) = q(x) log dx. p(x|w) Let (−λ) and m be the largest pole of the meromorphic function ζ (z) and its order. Then λ > 0 is called a learning coefficient. If we obtain the learning coefficient, then we can predict the stochastic complexity and the Bayes generalization error for β = 1 theoretically. Theorem 7.1 Assume that the set of true parameters {w ∈ supp ϕ; K(w) = 0} is not an empty set. Then λ and m satisfy the following conditions. 217

218

Singular learning machines

(1) There exists a constant c1 > 0 such that
(log n)m−1 exp(−nK(w)) ϕ(w) dw = c1 . lim n→∞ nλ (2) This equation holds:  log exp(−nK(w)) ϕ(w) dw . λ = − lim n→∞ log n (3) There exists a constant c2 > 0 such that
1 δ(t − K(w)) ϕ(w) dw = c2 . lim λ−1 t→0 t (− log t)m−1 (4) Let V (t) be a volume function,
V (t) =

ϕ(w) dw. K(w) 0 (a = 1), λ = lim

t→0

log{V (at)/V (t)} . log a

Proof of Theorem 7.1 We have already proved (1), (2), and (3) in Chapter 4. Let us prove (4). The state density function is defined by
v(t) = δ(t − K(w))ϕ(w)dw. By (3), there exists c2 > 0 such that v(t) = c2 t λ−1 (− log t)m−1 + o(t λ−1 (− log t)m−1 ). By the definition,


V (t) =

t

v(s)ds. 0

Using partial integration, we obtain V (t) = c2 t λ (− log t)m−1 + o(t λ (− log t)m−1 ), which shows log{V (at)/V (t)} = λ log a + o(1).


Remark 7.1 (1) Figure 7.1 shows the shape of {w; K(w) < t} near normal crossing singularities. Theorem 7.1 (4) shows that the learning coefficient is equal to the volume dimension of the set of almost correct parameters {w; K(w) < t} when t → 0.

7.1 Learning coefficient

219

Fig. 7.1. Learning coefficient and volume dimension

(2) If it is difficult to find a complete resolution of singularities for K(w) = 0, then we can numerically calculate λ by Theorem 7.1. Remark 7.2 To calculate the learning coefficient for a given Kullback–Leibler distance, the following properties may be helpful. (1) If there exist c1 , c2 > 0 such that K(w) and H (w) satisfy c1 H (w) ≤ K(w) ≤ c2 H (w), then K(w) is said to be equivalent to H (w). If K(w) and H (w) are equivalent, they have the same learning coefficient. For example, K(a, b, c) = a 2 + (a + bc)2 , H (a, b, c) = a 2 + b2 c2 are equivalent. In a compact set which contains the origin, K(a, b, c) = a 2 + a 2 b2 , H (a, b, c) = a 2 + a 2 c2 are equivalent. (2) For pairs (K1 (w), ϕ1 (w)) and (K2 (w), ϕ2 (w)), two zeta functions are defined by
ζi (z) = Ki (w)z ϕi (w) dw (i = 1, 2). Let (λ1 , m1 ) and (λ2 , m2 ) be the pairs of learning coefficients and their orders. If, for all w ∈ W , K1 (w) ≤ K2 (w),

ϕ1 (w) ≥ ϕ2 (w)

220

Singular learning machines

then λ 1 > λ2 or λ 1 = λ2 ,

m1 ≤ m2 .

(3) If w = (wa , wb ) and if K(wa , wb ) = Ka (wa ) + Kb (wb ), ϕ(wa , wb ) = ϕa (wa ) ϕb (wb ), then the coefficient λ and the order m satisfy λ = λ a + λb , m = ma + mb − 1, where (λa , ma ) and (λb , mb ) are pairs of learning coefficients and orders of (Ka , ϕa ) and (Kb , ϕb ) respectively. Theorem 7.2 Let W ⊂ Rd be the set of parameters. If there exists an open set U ⊂ W such that {w ∈ U ; K(w) = 0,

ϕ(w) > 0}

is not the empty set, then d . 2 Proof of Theorem 7.2 Let w0 be a parameter which satisfies K(w0 ) = 0 and ϕ(w0 ) > 0. We can assume w0 = 0 without loss of generality. If  > 0 is a sufficiently small constant,
Z(n) = exp(−nK(w)) ϕ(w) dw λ≤


≥

exp(−nK(w))ϕ(w) dw. |w| 0 K(w) =

d 1  Kij wi wj + o(|w|2 ). 2 i,j =1

7.1 Learning coefficient

221

Hence & d n  2 Z(n) ≥ exp − Kij wi wj − n o(|w| ) ϕ(w) dw. 2 i,j =1 |w| 1),

& d w 1  o(|w  |3 )   ϕ( √ ) dw , ≥ exp − Kij wi wj + √ 2 i,j =1 n n |w  | d/2. Proof of Theorem 7.4 Let us prove that the largest pole of
 K(w)z det I (w)dw ζ (z) = W

7.1 Learning coefficient

223

satisfies ‘−λ = −d/2 and m = 1’ or ‘−λ < −d/2’. It is sufficient to show that there exists a function I0 (w) which satisfies  det I (w) ≤ I0 (w), and the largest pole of
ζ0 (z) =

K(w)z I0 (w)dw

is equal to −d/2 and its order is equal to m = 1. By using resolution of singularities, in each local coordinate, there exists a set of natural numbers k1 , k 2 , . . . , k b > 0 K(w) = w12k1 w22k2 · · · wb2kb , where 1 ≤ b ≤ d. The log density ratio function can be written as f (x, w) = a(x, w)w1k1 w2k2 · · · wbkb . Let us define $ ri (x, w) = Then ∂f (x, w) = ∂wi

$

∂a w ∂wi i ∂a ∂wi

+ ki a(x, w)

(1 ≤ i ≤ b) (b < i ≤ d).

ri (x, w)w1k1 · · · wiki −1 · · · wbkb ri (x, w)w1k1

· · · wbkb

(1 ≤ i ≤ b) (b < i ≤ d).

By using a matrix J (w) defined by
Jij (w) = ri (x, w)rj (x, w)p(x|w)dx, we have 1/2

(det I (w))

=

 b

dk −1 wp p

& (det J (w))1/2 .

p=1

If b = 1, there exists c1 > 0 such that K(w) = w12k1 (det I (w))1/2 ≤ c1 w1dk1 −1 .

(7.2)

224

Singular learning machines

Therefore the theorem holds. Let us study the case b ≥ 2. A transform w = g(u) defined by w1 = u1 , w2 = u2 u1 , .. . w b = ub · · · u 1 can be made by blow-ups, where the other coordinates are symmetrically defined. By the definition eq.(7.2), if ui = 0 for some i (1 ≤ i ≤ b − 1), then det J (g(u)) = 0. Because the determinant det J (g(u)) ≥ 0 is an analytic function of u, there exists a constant c2 > 0 such that det J (g(u)) ≤ c2 u21 · · · u2b−1 . Let σp be a constant defined by σp = k1 + k2 + · · · + kp . There exists a constant c3 > 0 such that K(g(u)) =

b 

2σ

up p

p=1 b−1 

(det I (g(u)))1/2 ≤ c3

up

p=1

|g  (u)| =

b 

b 

dσp −b+p−1

up

p=1

ub−p p .

p=1

From the integration of ub , we obtain the pole (−d/2) (m = 1). From the integration of the other parameters u1 , u2 , . . . , ub−1 , smaller poles than (−d/2)
are obtained, which completes the proof. Remark 7.5 (1) If Jeffreys’ prior is employed, there exists a case λ > d/2. For example, p(y|x, a, b) =

1 2

exp(− 12 (y − abx − a 2 b3 )2 ).

Assume that the true distribution is given by p(y|x, 0, 0). In this case, d = 2 but λ = 3/2. Note that, if ab = c, a 2 b3 = d, this model can be understood as a regular model λ = 2/2. However, the compact set of (a, b) does not correspond to the compact set of (c, d). Therefore, the model represented by (a, b) with

7.1 Learning coefficient

225

compact support is not equivalent to that represented by (c, d) with compact support. (2) Let us study a model of (x, y) ∈ RM × RN ,  1  q(x) 2 p(x, y|w) = exp − y − h(x, w) , (2π σ 2 )N/2 2σ 2 h(x, w) =

K 

ak s(bk , x),

k=1

where the parameter w is given by w = {(ak , bk ) ∈ RN × RL }. Assume that s(b, x) satisfies the condition det I (w) = 0 ⇐⇒ {ak s(bk , x)} are linearly independent. Then λ=

d , 2

m=1

holds [103]. Example 7.1 In a concrete learning machine, let us calculate the learning coefficient, λ, and its order, m. Let N be a random variable which is subject to the standard normal distribution. The probability distribution q(x) of X is assumed to have compact support. We study a statistical model in which a random variable Y for a given X is defined by Y = aσ (bX) + cσ (dX) + N . This is a layered neural network with one input unit, two hidden units, and one output unit. Let us consider the case σ (x) = ex − 1. The true distribution of Y is assumed to be Y = 0 + N. The a priori probability distribution ϕ(w) has compact support and ϕ(0, 0) > 0. The Kullback–Leibler distance as a function of w = (a, b, c, d) is given by
1 K(w) = (aσ (bx) + cσ (dx))2 q(x)dx. 2 By using the Taylor expansion, f (x, w) = aσ (bx) + cσ (dx) =

∞  xk k=1

k!

(abk + cd k ).

226

Singular learning machines

Here {x k } is a set of linearly independent functions. Therefore K(w) ≡ 0 is equivalent to pk = abk + cd k = 0

(∀k = 1, 2, 3, . . .).

As we have shown in Example 3.2, pn ∈ p1 , p2 . In Chapter 3, we have already derived the resolution of singularities of this polynomial in Section 3.6.2. In one of the coordinates, the resolution map w = g(u) is given by a = a, b = b1 d, c = a(b1 − 1)b1 c5 d − ab1 , d = d, where u = (a, b1 , c5 , d). Hence p1 = ab1 (b1 − 1)c5 d 2 , p2 = ab1 (b1 − 1)(1 + c5 )d 2 ,   pk = ab1 (b1 − 1) b1k−2 + b1k−3 + · · · + 1 + c5 d . Thus f (x, g(u)) = ab1 (b1 − 1)d 2 a(x, u), where a(x, u) = c5 x + 12 (1 + c5 )x 2 +

∞  xk k=3

k!

  d k−2 b1k−1 + b1k−2 + · · · + 1 + c5 d .

Hence a(x, 0) = 0, so we have a 2 b12 (b1 − 1)2 d 4 K(g(u)) = 2


a(x, u)2 q(x)dx.

The Jacobian determinant of w = g(u) is |g  | = |a(b1 − 1)b1 d 2 |. The largest pole of the zeta function
z  2 2 ζ (z) = a b1 (b1 − 1)2 d 4 |a(b1 − 1)b1 d 2 | ϕ(g(u)) da db1 dc5 dd

7.2 Three-layered neural networks

227

is −λ = − 34 and its order is 1. We found that the learning coefficient in this case is equal to 34 , hence the normalized stochastic complexity is equal to Fn0 =

3 4

log n + random variable

and the mean Bayes generalization error for β = 1 is equal to E[Bg ] =

1 3 +o . 4n n

Remark 7.6 If a statistical model p(y|x, w)q(x) and a true distribution q(y|x)q(x) are respectively given by p(y|x, w) =

1 2

exp(− 12 (y − f (x, w))2 ),

q(y|x) =

1 2

exp(− 12 (y − f0 (x))2 ),

then the Kullback–Leibler distance is given by K(w) =

1 2


(f (x, w) − f0 (x))2 q(x)dx.

If f (x, w) − f0 (x) =

∞ 

fj (w)ej (x),

j =1

where fk (w) is a set of polynomials and {ek (x)} is a set of linearly independent functions on the support of q(x), then by the Hilbert basis theorem, there exists J such that K(w) is equivalent to K1 (w) =

J 

fj (w)2 .

j =1

7.2 Three-layered neural networks There are several kinds of neural networks: three-layered perceptrons, radial basis functions, and reduced rank regressions. They are created by a superposition of parametric functions. A three-layered neural network, which is

228

Singular learning machines

x1

y1

x2

y2

xM

yN

Fig. 7.2. Three-layered neural network

illustrated in Figure 7.2, is defined as a probability density function of x ∈ RM and y ∈ RN q(x) p(x, y|w) = √ N exp(− 12 |y − h(x, w)|2 ), 2π h(x, w) =

H 

ak σ (bk · x + ck ),

k=1

where | | is the norm of N -dimensional Euclidean space. The probability density function q(x) is not estimated. Sometimes σ (t) = tanh(t) is employed. The parameter is given by w = {(ak , bk , ck )}. If the true probability distribution is given by q(x, y) = p(x, y|w0 ), the log density ratio function is equal to f (x, y, w) = log q(x, y) − log p(x, y|w) = 12 {|y − h(x, w)|2 − |y − h(x, w0 )|2 }. The convergence radius of one variable tanh(x) is π/2. Hence if the support of q(x) is compact then f is an Ls (q(x, y))-valued real analytic function. If the true distribution is given by H = H0 , then the learning coefficient satisfies 

MH1 + 2MN1  , λ ≤ 12 H0 (M + N1 ) + min N1 H1 , MH1 , 3 where H1 = H − H0 N1 = N + 1 [102]. In particular, if M = N = 1 and H0 = 0, then √ √ [ H ]2 + [ H ] + H , λ= √ 4 H +2 √ √ where [ H ] is the largest integer that is not larger than H [11]. In the case σ (x) = x, ch = 0, this model is called the reduced rank regression. If the

7.2 Three-layered neural networks

229

dimensions of input and output are equal to M and N respectively, and if the ranks of the statistical model and the true distribution are H and r respectively, then the following hold [10]. (1) If N + r < M + H , M + r < N + H , H + r < M + N and M + H + N + r is an even integer, then m = 1 and λ = 18 {2(H + r)(M + N ) − (M − N )2 − (H + r)2 }. (2) If N + r < M + H , M + r < N + H , H + r < M + N and M + H + N + r is an odd integer, then m = 2 and λ = 18 {2(H + r)(M + N ) − (M − N )2 − (H + r)2 + 1}. (3) If M + H < N + r then m = 1 and λ = 12 {H M − H r + N r}. (4) If N + H < M + r then m = 1 and λ = 12 {H N − H r + Mr}. (5) If M + N < H + r then m = 1 and MN . 2 Remark 7.7 (1) The reduced rank regression is a statistical model defined by  1  1 2 p(y|x, w) = exp − |y − BAx| , (2π σ 2 )N/2 2σ 2 λ=

where x ∈ RM, y ∈ RN, A is an M × H matrix, B is an H × N matrix, and σ > 0 is a constant. The parameter is w = (A, B). It the true distribution is given by (A0 , B0 ), then the Kullback–Leibler distance is equivalent to K(w) = 12 BA − B0 A0 2 , where   is the norm of the matrix. For example, if M = N = H = 2 and r = 0, then



+ + e f +2 + ab 2K(w) = + + cd g h = (ae + bg)2 + (af + bh)2 + (ce + dg)2 + (cf + dh)2 . By using a blow-up and isomorphism, a = a,

b = ab ,

e = e + b g,

c = ac ,

f  = f + b g,

d = ad  , d  = d  − bc ,

230

Singular learning machines

Fig. 7.3. Mixture model

it follows that 2K(w) = a 2 (e2 + f 2 + (ce + d  g)2 + (cf  + d  h)2 ), which is equivalent to a 2 (e2 + f 2 + d 2 g 2 + d 2 h2 ). Furthermore, using blow-up with the center e , f  , d  , we obtain λ = 3/2. (2) In statistics, there is a problem of how we can numerically approximate the Bayes a posteriori distributions. For example, several Markov chain Monte Carlo methods are studied in Chapter 8. To evaluate the accuracy of such methods, we can compare the numerical stochastic complexity or Bayes generalization error with theoretical values. The reduced rank regression is appropriate for such a purpose because its learning coefficients are completely determined as above.

7.3 Mixture models Mixture models have very wide applications in automatic clustering of data and density estimation of unknown distributions. A mixture model in Figure 7.3 is defined by p(x|w) =

H 

ah s(x, bh ),

h=1

where x ∈ RN , s(x, bh ) is itself a probability density function of x for a given bh ∈ RM , and {ah } is a set of nonnegative real numbers which satisfy H 

ah = 1.

h=1

The set of parameters of a mixture model is given by w = {(a, b) ≡ (ah , bh ); h = 1, 2, . . . , H }.

7.3 Mixture models

231

The dimension of the parameter space is MH+H − 1. For example, if s(x, bh ) is a normal distribution, this model is called a normal mixture, or if s(x, bh ) is a binomial distribution, a binomial mixture. There are several mathematical problems in mixture models. The singularity caused by the condition that ak should be nonnegative can be overcome by ah = aˆ h2 then {aˆ h } can be understood as an element of the smooth manifold, H 

aˆ H2 = 1,

h=1

where the a priori distribution is also transformed. The log density ratio function is given by q(x) p(x|w)  p(x|w) − q(x)  , = − log 1 + q(x)

f (x, w) = log

Hence it is an Ls (q)-valued analytic function if there exists  > 0 such that  p(x|w) − q(x)    sup  H0 or k > S0 , then bh(is) 1 h2 ···hH (i > H0 or hp > S0 ) are free parameters, hence the number of free parameters is D2 = (Y − 1)N S H −H0 + (Y − 1)N (S − S0 )H0 , and therefore λ ≤ (D1 − D2 )/2, by Theorem 7.3.

7.5 Hidden Markov model Hidden Markov models or probabilistic finite automatons in Figure 7.5 are being applied to speech recognition, motion recognition, and gene analysis. Let H be the number of hidden states. At t = 1, the first state is chosen by probability 1. Then at t = 2, 3, . . . , the new state is chosen with respect to the

7.6 Singular learning process

235

transition probability a(i, j ) from the ith state to the j th state. Since the sum of the probabilities is equal to 1 for each i, H 

a(i, j ) = 1,

(i = 1, 2, . . . , H ).

j =1

When the hidden state is at the j th state, the output y is taken from the probability density function f (y|b(j )) where b(j ) ∈ RM . In some models, y is taken from a discrete set whereas, in other models, y is taken from Euclidean space. By this stochastic process, the model outputs a sequence x of length T , x = (x1 , x2 , . . . , xT ). The set of parameters is w = {(a(i, j ), b(j ))}, where the number of parameters is H (H − 1) + H M. The probability distribution of x is p(x|w) = f (x1 |b1 )

T  H 

 a(kt−1 , kt )f (xt |b(kt )) ,

t=2 kt =1

where k1 = 1. If many sequences of length T are obtained independently, singular learning theory can be applied. Assume that the true distribution has H0 hidden variables. If the transition probabilities satisfy a(i, j ) = 0 (1 ≤ i ≤ H0 , H0 < j ≤ H ), then a(i, j ) (H0 < i ≤ H ) and b(i) (H0 < i ≤ H ) are free parameters. Therefore the numbers of free parameters is (H − H0 )(H − 1 + M), giving λ ≤ H0 (H + M − 1)/2 by Theorem 7.3.

7.6 Singular learning process In this section, we consider a case when the true distribution is outside the finite-size parametric model. Even in such a case, singularities of a learning machine affect the learning process. A three-layered perceptron is defined by a probability distribution of (x, y) where x ∈ RM and y ∈ RN , p(x, y|w) = h(x, w) =

q(x) exp(− 12 |y − h(x, w)|2 ), (2π)1/2 H  k=1

ak tanh(bk · x + ck ),

236

Singular learning machines

and the parameter of this model is w = {(ak , bk , ck ); k = 1, 2, . . . , H }. For a given H0 < H , the set of all parameters which satisfies H 

ak tanh(bk · x + ck ) =

k=1

H0 

ak∗ tanh(bk∗ · x + ck∗ )

k=1

contains singularities. If H0 is smaller, then the singularities are more complicated. In other words, simple function ⇐⇒ complicated singularities, complicated function ⇐⇒ simple singularities. Assume that the true conditional distribution q(y|x) is outside the set of the parametric probability distributions, {p(x, y|w); w}. The set of training samples is denoted by Dn = {(xi , yi ); i = 1, 2, . . . , n}. The a posteriori distribution is p(w|Dn ) ∝ ϕ(w)

n 

p(yi |xi , w).

i=1

The neighborhood of a parameter w∗ is defined by U (w ∗ ) = {w ∈ W ; |w − w∗ | < }. The probability that a parameter is contained in U (w ∗ ) with respect to the a posteriori distribution is
∗ P (w ) ≡ p(w|Dn )dw. U (w ∗ )

Let us show the following universal phenomenon. Universal phenomenon. When the number of training samples is small, the parameter that makes P (w ∗ ) largest is a complicated singularity. As the number of training samples increases, the singularity becomes simpler. Let us explain the above phenomenon. The empirical Kullback–Leibler distance is given by 1 q(yi |xi ) log . n i=1 p(yi |xi , w) n

Kn (w) =

7.6 Singular learning process

237

Then the a posteriori distribution is rewritten as p(w|Dn ) =

1 exp(−nKn (w)) ϕ(w) Z0

and P (w ∗ ) = p(w∗ |Dn ) Z(w ∗ ), where 1 Z(w ) = Z0 ∗


U (w ∗ )

exp(−n{Kn (w) − Kn (w ∗ )}) ϕ(w) dw.

Note that the probability P (w∗ ) is determined by not only p(w ∗ |Dn ) but both p(w ∗ |Dn ) and Z(w ∗ ), and that Z(w ∗ ) is larger if w ∗ is the more complicated singularity. Therefore the learning process is determined as the jump from complicated singularities to simpler singularities. The set of parameters W ⊂ Rd can be represented as a union of small subsets, W = ∪α Vα , where Vα is defined by Vα = {w ∈ R d ; |w − wα | ≤ }. The empirical Kullback–Leibler distance is nKn (w) ∼ = nK(w) + (nK(w))1/2 ψ(w). Let pα be the probability that w ∈ Vα , that is to say,
1 pα = e−nKn (w) ϕ(w) dw. Z0 Vα If n is sufficiently large, pα ∝ e−Kα n−λα log n where Kα = K(wα∗ ), wα∗ = arg min K(w), w∈Vα
q(x) K(w) = q(x) log dx, p(x|w) and (−λα ) is the largest pole of
(K(w) − K(wα∗ ))z ϕ(w)dw. ζ (z) = V (w0 )

(7.5)

238

Singular learning machines

Bg (n) W

n Fig. 7.6. Learning curve with singularities

In general, if the analytic set {w ∈ Vα ; K(w) − K(wα∗ ) = 0} is more complicated, then λα is smaller. Therefore, pα ∝ e−fα fα = Kα n + λα log n. The neighborhood of a parameter wα which maximizes fα is realized with the highest probability. As the number of training samples increases, the singularity becomes simpler. In a three-layered perceptron, it is well known that, if Vα is the neighborhood of the parameter that represents k hidden units, Kα and λα can be approximated by C1 , k λ α ≈ C 2 k + C3 ,

Kα ≈

where C1 , C2 , C3 > 0 are constants. Therefore C1 n + {C2 k + C3 } log n. k The Bayes generalization error is equal to the increase of the free energy, C C2 k + C3 1 + . Bg (n) = min k k 2n Figure 7.6 shows the learning process of singular learning machines. The number k which minimizes fα is , Cn k= . log n fα =

7.7 Bias and variance

239

Hence the number of hidden units which is chosen with high probability by the a posteriori distribution is in proportion to (n/ log n)1/2 . Remark 7.8 (1) From the statistical physics point of view, the a posteriori distribution can be understood as the equilibrium state whose Hamiltonian is equal to the Kullback–Leibler distance. The equilibrium state is determined not by the minimization of not the energy Kα n but by the minimization of the free energy, Kα n + λ log n. (2) Singularities of a model make the learning curve smaller than any nonsingular learning machine. If the true distribution is not contained in the model, then the singularity seems to be the virtual true distribution appropriately with respect to the number of training samples. The most appropriate singularities are selected by minimization of the free energy for the given number of training samples, resulting in the generalization errors being made smaller. It is possible that brain-like systems utilize the effect of singularities in the real world.

7.7 Bias and variance In the previous section, the true distribution is outside the finite parametric model but constant compared to the number of training samples. In this section, to analyze the effect of singularities more precisely, we study the balance between bias and variance. Here ‘bias’ is the function approximation error, whereas ‘variance’ is the statistical estimation error. If the bias is much larger than the variance, then the statistical model is too simple, hence we should use a more complicated model. If the variance is much larger than the bias, then the model is too complicated, hence it is better to use a simpler model. If the Kullback–Leibler distance from the singularity to the distribution is in proportion to 1/n, then both the bias and the variance are of the same order 1/n, so the bias and variance problem appears. It is still difficult to study such cases based on the general conditions. In this section, we study the problem using a concrete and simple model. Assume that a set of independent random samples Dn = {(x1 , y1 ), (x2 , y2 ), . . . , (xn , yn )} is taken from the true distribution q(x)q(y|x), where (x, y) ∈ RN × R1 . Let us study the following conditions.

240

Singular learning machines

Learning machine: 1 p(y|x, a, b) = √ exp(− 12 (y − H (x, a, b))2 ), 2π True distribution:   H0 (x) 2  1 , q(y|x) = √ exp − 12 y − √ n 2π

(7.6)

(7.7)

where H0 (x) is a general function and H (x, a, b) =

J 

a hj (b) ej (x),

j =1

which has the set of parameters {(a, b) ∈ R 1 × R N }. We adopt the set of orthonormal functions, that is to say,
ei (x)ej (x)q(x)dx = δij . We use the notation


H0  = 2

H0 (x)2 q(x)dx,


H0j =

H0 (x) ej (x)q(x)dx.

Note that the true distribution is outside the parametric model in general and it depends on the number of random samples. Assume that the a priori probability density function ϕ(a, b) is a C 1 -class function of a, and that ψ(b) ≡ ϕ(0, b) is a compact support function of b (ψ(0) > 0). The a posteriori probability distribution with β = 1 and the predictive distribution are respectively given by n  1 p(yi |xi , a, b), ϕ(a, b) C i=1
p(y|x, Dn ) = p(y|x, a, b) p(a, b|Dn ) da db,

p(a, b|Dn ) =

where C > 0 is a constant. Let Ea,b [ ] be the expectation value using the a posteriori distribution.

7.7 Bias and variance Theorem 7.5 Assume that β = 1 and that
ψ(b) db < ∞. J 2 j =1 hj (b) The four errors of the Bayes quartet are given by 1 1 Bg =  + o , n n 1 1 , Bt = ( − 2ν) + o n n 1 1 , Gg = ( + ν) + o n n 1 1 , Gt = ( − ν) + o n n

241

(7.8)

(7.9) (7.10) (7.11) (7.12)

where =

1 2



1 + H0 2 −

J  j =1

ν=

H0j Eg

1 ∂Z  , Z ∂gj

1 ∂Z  Eg gj , Z ∂gj j =1

J 

and

Z(g) =

J   α 2 hj (b)2 exp − 12 j =1

+

J 

 αhj (b)(gj + H0j ) ψ(b)dα db.

(7.13)

j =1

Here g = {gj } is a random variable which is subject to the J -dimensional normal distribution with zero mean and identity covariance matrix, and Eg [ ] shows the expectation value over g. Remark 7.9 (Universality of equations of states) In this theorem, since the true distribution is outside of the parametric model and it is not fixed but moving as the number of training samples, Theorem 6.10 does not hold. In fact  = λ. If H0 (x) ≡ 0, then this theorem coincides with Theorem 6.10 with β = 1. However, this theorem shows that, even if H0 (x) = 0, the equations of states in Main Theorem 6.3 hold. It seems that equations of states are the universal relations among the Bayes quartet.

242

Singular learning machines

Proof of Theorem 7.5 From the definition of the true distribution and the statistical model, the log density ratio is   H0 (x)  H0 (x) 2 f (x, y, a, b) = 12 H (x, a, b) − √ , + σ H (x, a, b) − √ n n where σ = y − H0 (x) is a random variable which is subject to the standard normal distribution. The log likelihood ratio function is n Kn (a, b) =

n 

f (xi , yi , a, b).

i=1

√ By using a rescaling parameter a = α/ n, we have n   α 1  nKn √ , b = (H (xi , α, b) − H0 (xi ))2 2n i=1 n

1  −√ σi (H (xi , α, b) − H0 (xi )). n i=1 √ By using the central limit theorem, nKn (α/ n, b) converges to the following function E(α, b) in law, n

E(α, b) =

J J  1 2 α hj (b)2 − αhj (b)H0j 2 j =1 j =1

H0 2  αhj (b)gj + σˆ − 2 j =1 J

+

where we used these convergences in law, 1  σi ej (x) → gj , √ n j =1 n

1  σi H0 (xi ) → σˆ . √ n j =1 n

Let us define the renormalized a posteriori distribution,  −E(α,b) F (α, b)ψ(b)dα db e  Eα,b [F (α, b)] = . e−E(α,b) ψ(b)dα db Then, for an arbitrary natural number k, Ea,b [H (X, a, b)k ] =

 1  k E [H (X, α, b) ] + o . α,b nk/2 nk/2 1

7.7 Bias and variance

243

By Theorem 1.3, Remark 1.12, and Subsection 1.4.3, 1 1 Eg [ EX [ Eα,b [H (X, α, b) − H0 (X)]2 ]] + o , 2n n Bt = Bg − Gg + Gt + o(1/n), 1 1 Eg [ EX [ Eα,b [(H (X, α, b) − H0 (X))2 ]]] + o , Gg = 2n n Bg =

Gt = Gg −

J  1 αhj (b)gj ]] + o(1/n). Eg [ Eα,b [ n j =1

Then, by the definition of H (x, α, b), Bg =

 H 2 1 1  0 +o , Eα,b [αhj (b)]2 − 2H0j Eα,b [αhj (b)] + 2n j =1 2n n

Gg =

 H 2 1 1  0 +o . Eα,b [α 2 hj (b)2 ] − 2H0j Eα,b [αhj (b)] + 2n j =1 2n n

J

J

The a posteriori distribution satisfies Eα,b [ (αhj (b))k ] =

1  ∂ k Z(g), Z ∂gj

where Z = Z(g) is defined as in eq.(7.13). By using the partial integration for gj ,

1 ∂Z 

∂  1 ∂Z 

1 ∂ 2Z 

 1 ∂Z 2  E g gj = Eg = Eg 2 2 − Eg . Z ∂gj ∂gj Z ∂gj Z ∂gj Z ∂gj  Moreover, by using the partial integration for dα, J J   1 ∂ 2Z 1  ∂Z  = 1 + (g + H ) . j 0j Z ∂gj2 Z ∂gj j =1 j =1

By using these facts, we obtain the theorem.




If the true distribution is given by H0 (x) ≡ 0, then  = 1/2, which corresponds to the fact that the largest pole of the zeta function
ζ (z) = a 2z |b|2z ϕ(a, b) da db is equal to z = −1/2. Let us consider a more specific learning machine, N   1  1 abj ej (x)))2 . p(y|x, a, b) = √ exp − (y − 2 2π j =1

(7.14)

244

Singular learning machines

Learning coefficient

Dimension

3

λ d/2

Dimension

4 q

Fig. 7.7. Learning coefficients

where a ∈ R1 , b ∈ RN , x ∈ RN (N > 1) and we assume ψ(b) is symmetric for the direction of b hence it can be written as ψ(|b|). If N = 1, eq.(7.8) is not satisfied. If N ≥ 2, by wi = abi , the model is formally equal to the regular model, N   1  1 p(y|x, w) = √ exp − (y − wj ej (x)))2 . 2 2π j =1

Let us compare the generalization error of the singular model represented by {a, bi } with that of the regular model represented by {wi }. In the following we assume H0 (x) = w0 · x. Let (w0 ) be the value in Theorem 7.5, by which the Bayes generalization error is given as Bg = /n + o(1/n). For arbitrary w0 , the learning coefficient of the regular statistical model is N/2. Therefore, we compare N/2 with the learning coefficient of the singular model, (w0 ). If the true model is given by eq.(7.7) with H0 (x) = w0 · x, then the singular model eq.(7.14) has the following learning coefficient:

YN (g)  (7.15) 2(w0 ) = 1 + Eg (|w0 |2 + w0 · g) YN−2 (g) where


YN (g) = 0

π/2

dθ sinN θ exp(− 12 |w0 + g|2 sin2 θ ).

This results can be shown by using polar coordinates. In eq.(7.15) the learning coefficient (w0 ) cannot be represented by a simple function; however, the numerical result is shown in Figure 7.7 for N = 2, 3, . . . , 6. Here the horizontal and vertical lines show |w0 | and 2(w0 )/N respectively. If 2(w0 )/N < 1, then the learning coefficient is smaller than that in the regular statistical model.

7.8 Non-analytic learning machines

245

(1) When |w0 | → ∞, (w0 ) converges to N/2 in every case. (2) If N = 2 or N = 3, then (w0 ) is larger than N/2 when |w0 | becomes large. Whereas, if N ≥ 4, then the learning coefficients are always smaller than N/2. In other words, if the dimension of the parameter space is larger than 3, then singularities always make the generalization error smaller than in the regular statistical model. Also we can show that the learning coefficient (w0 ) has an asymptotic expansion for |w0 | → ∞,  1  (N − 1)(N − 3) 2(w0 ) = N − + o . |w0 |2 |w0 |2 Remark 7.10 (Shrinkage estimation) In regular statistical models, the shrinkage estimation and empirical Bayes method have the same property described in this section [85, 29]. Singularities of a model affect the learning process as the source of shrinkage estimation; moreover, the most appropriate singularities are selected by the a posteriori distribution for a given set of random samples.

7.8 Non-analytic learning machines In Chapter 6, we assume that the log likelihood density function is a functionvalued analytic function of the parameter. The reason why such an assumption is adopted is that our main purpose is to discover the universal theorems without regard to pathological examples. Example 7.2 (Non-analytic model) The following is a non-analytic model  x 2   1 1 , p(y|x, a) = √ exp − y − exp − 2 2 a 2π for a = 0 and p(y|x, 0) is defined by the standard normal distribution of y. If the true distribution is q(y|x) = p(y|x, 0) and if q(x) is the standard normal distribution, then the Kullback–Leibler distance is
 2x  1 K(a) = exp − 2 q(x)dx 2 a   1 1 = exp − 4 . 2 a Hence K(a) = 0 if and only if a = 0 and K(a) is a function of class C ∞ by the definition that the n times differential K (n) (a) = 0 (n = 1, 2, . . .). There are no constants c1 , c2 > 0 and no polynomials K1 (a), K2 (a) which satisfy c1 K1 (a) ≤ K(a) ≤ c2 K2 (a)

246

Singular learning machines

in the neighborhood of the origin. Such a statistical model does not satisfy fundamental condition (I) or (II). In regular statistical theory, such pathological examples can be rejected by the condition that the Fisher information matrix should be positive definite. However, in singular learning theory, we cannot adopt such a condition. Let us study the normal mixture. p(x|w) =

H  h=1

 |x − b |2  ah h exp − , N/2 (2π) 2

(7.16)

where x, bh ∈ RN , ah ∈ R1 , and w = {ah , bh }. The set of parameters is set as W = {(ah , bh ); a1 + · · · + aH = 1, ah ≥ 0, |bh | ≤ B}, where B > 0 is a constant. Let q(x) be a true distribution which is represented by q(x) = p(x|w0 ) for some parameter w ∈ W . The log density ratio function is not an Ls (q)-valued analytic function in general. Example 7.3 Let N = 1 and 1 p0 (x|b) = √ exp(− 12 (x − b)2 ). 2π For a mixture model p(x|a) = (1 − a) p0 (x|0) + a p0 (x|2) and a true distribution q(x) = p(x|0), the log density ratio function is equal to f (x, a) = log(q(x)/p(x|a)) = − log(1 + a (e2x−2 − 1)) =

∞  aj j =1

j

(1 − e2x−2 )j ,

where the last equation is a formal expansion. The convergence radius of this function as an L1 (q)-valued function is zero, because [109]  
 ∂ j 1/j   lim sup = ∞.  j f (x, 0)q(x)dx j →∞ ∂a This example shows that the normal mixture does not satisfy fundamental condition (I). However, since the four Main Theorems in Chapter 6 are universal formulas, we can improve them for a given statistical model if the model is not

7.8 Non-analytic learning machines

247

a pathological one. Let us introduce a function   − log t + t − 1 S(t) = (t − 1)2 1 2

(t = 1) (t = 1).

Then S(t) is an analytic function. A function M(x) is introduced, M(x) ≡ sup S w∈W

 p(x|w)  . q(x)

We can show by the definition eq.(7.16) that there are A0 , A1 , B0 , B1 > 0 such that e−A0 |x|−A1 ≤

p(x|w) ≤ eB0 |x|+B1 q(x)

(∀(x, w) ∈ RN × W ).

Hence there exist C0 , C1 > 0 such that M(x) ≤ C0 |x| + C1 . The Kullback–Leibler distance is bounded by
K(w) =

q(x) log


q(x) dx p(x|w)

  p(x|w) p(x|w) + − 1 dx q(x) − log q(x) q(x)
2  p(x|w)   p(x|w) −1 S dx = q(x) q(x) q(x)
 2 p(x|w) − 1 M(x)q(x)dx ≡ H (w). ≤ q(x) =

Here it is easy to show that (p(x|w)/q(x) − 1) is an Ls (Mq)-valued analytic function for an arbitrary s ≥ 2. Therefore we can apply the resolution theorem to H (w), which means that there exists a resolution map w = g(u) such that K(g(u)) ≤ H (g(u)) = u2k . There exists an Ls (Mq)-valued analytic function a(x, u) such that p(x|w) − 1 = a(x, u)uk , q(x)

248

Singular learning machines

where a(x, u) is not equal to zero as an Ls (Mq)-valued function. On the other hand, for arbitrary compact set C ∈ RN ,
 p(x|w)  K(g(u)) = q(x)a(x, u)2 u2k S dx q(x)
 p(x|w)  2k q(x)a(x, u)2 S dx. ≥u q(x) C By choosing the compact set C sufficiently large, there exists a constant c1 > 0 such that K(g(u)) ≥ c1 u2k . There exists a function c1 (u) > 0 of class C ∞ , K(g(u)) = c1 (u)u2k . In the same way as in Chapter 6, there exists c2 (u, x) which is a function of class C ∞ , f (x, g(u)) = c2 (x, u)uk . The empirical process 1  {c2 (Xj , u) − c1 (u)uk } ξn (u) = √ n j =1 n

is a function of class C ∞ for u which converges in law to a Gaussian process [92] as a random variable of a Banach space defined by a sup-norm on a compact set. Therefore, we can obtain the same result for normal mixtures. Remark 7.11 In this book, the main formulas are proved based on fundamental conditions (I) and (II). These conditions allow that the Fisher information matrix is not positive definite; however, this is a sufficient condition for the main formulas. The necessary and sufficient conditions for the main formulas are still unknown.

8 Singular statistics

In this chapter, we study statistical model evaluation and statistical hypothesis tests in singular learning machines. Firstly, we show that there is no universally optimal learning in general and that model evaluation and hypothesis tests are necessary in statistics. Secondly, we analyze two information criteria: stochastic complexity and generalization error in singular learning machines. Thirdly, we show a method to produce a statistical hypothesis test if the null hypothesis is a singularity of the alternative hypothesis. Then the methods by which the Bayes a posteriori distribution is generated are introduced. We discuss the Markov chain Monte Carlo and variational approximation. In the last part of this chapter, we compare regular and singular learning theories. Regular learning theory is based on the quadratic approximation of the log likelihood ratio function and the central limit theorem on the parameter space, whereas singular learning theory is based on the resolution of singularities and the central limit theorem on the functional space. Mathematically speaking, this book generalizes regular learning theory to singular statistical models.

8.1 Universally optimal learning There are a lot of statistical estimation methods. One might expect that there is a universally optimal method, which always gives a smaller generalization error than any other method. However, in general, such a method does not exist. Assumption. Assume that (w) is the probability density function on Rd , and that a parameter w is chosen with respect to (w). After a parameter w 249

250

Singular statistics

is chosen and fixed, random samples dn = {x1 , x2 , . . . , xn } are independently taken from a conditional distribution P (x|w). In the real world, we can measure only the set of random samples, and (w) and P (x|w) are unknown. Let r(x|dn ) be an arbitrary conditional probability density function of x for a given set of samples dn . The generalization error of r(x|dn ) is defined by the Kullback–Leibler distance from the fixed distribution P (x|w) to r(x|dn ),
P (x|w) . G(w, dn ) = dxP (x|w) log r(x|dn ) The mean generalization error G(r) is defined by the expectation over all w and dn ,
G(r) ≡ dw (w) Edn [G(w, dn )]
=

dw (w)

n 


 P (xi |w)dxi G(w, dn ).

i=1

Then G(r) is a functional of a given conditional probability r(x|dn ). Let us study the minimization problem of G(r). We define the predictive distribution R(x|dn ) based on the true a priori probability density function (w) and the true parametric model P (x|w),
n  dw (w)P (x|w) P (xi |w) R(x|dn ) ≡

i=1


dw (w)

n 

.

(8.1)

P (xi |w)

i=1

Theorem 8.1 The functional G(r) is minimized if and only if r(x|dn ) = R(x|dn ). Proof of Theorem 8.1 The functional G(r) can be rewritten as



R(x|dn )  dx R(x|dn ) log G(r) = dw(w)Edn r(x|dn )



dx P (x|w) log P (x|w) + dw(w)
−


dw(w)Edn

 dx P (x|w) log R(x|dn ) .

8.1 Universally optimal learning

251

The first term of the right-hand side is equal to the Kullback–Leibler distance from R(x|dn ) to r(x|dn ), and neither the second term nor third term depends on r(x|dn ). Therefore, by the property of the Kullback–Leibler distance, the functional G(r) is minimized if and only if r(x|dn ) = R(x|dn ). Under the assumption as above, there is no better statistical estimation than
R(x|dn ).  Let us introduce the mean entropy of p(x|w) by using Ew [ ] = (w)dw, S = −Ew




 P (x|w) log P (x|w)dx

and the stochastic complexity by
Fn (dn ) = −log

dw(w)

n 

P (xi |w).

(8.2)

i=1

Then the minimum of G(r) is equal to
min G(r) = dw (w)Edn+1 [Fn+1 (dn+1 ) − Fn (dn ) − S]. r

Remark 8.1 From Theorem 8.1, we obtain the following facts. (1) If one knows the true a priori distribution (w) and the true statistical model P (x|w), there is no better statistical estimation than Bayes estimation using these. (2) Even if one knows the true statistical model P (x|w), the optimal predictive distribution depends on the a priori probability distribution, hence there is no universally optimal statistical estimation method. (3) In practical applications, we design an a priori distribution ϕ(w) and a model p(x|w). If they are written as ϕ(w|θ1 ) or pθ2 (x|w) using parameters θ1 and θ2 , such parameters are called hyperparameters. We have to optimize hyperparameters or the probability distribution of hyperparameters. If several possible models p1 (x|w), p2 (x|w), . . . , pk (x|w) are compared and optimized, such a procedure is called statistical model selection. (4) For hyperparameter optimization or statistical model selection, we need a statistical evaluation method of the pair (ϕ, p) for a set of random samples. The major concepts used in statistical model evaluation are stochastic complexity and generalization error.

252

Singular statistics

8.2 Generalized Bayes information criterion For a given set of training samples {Xi }, the stochastic complexity of ϕ(w) and p(x|w) with p = 1,
 n Fn (ϕ, p) = −log p(Xi |w)ϕ(w)dw, i=1

can be understood as a likelihood of the pair (ϕ, p). Therefore, if Fn (ϕ, p) is smaller, then the pair (ϕ, p) seems to be more appropriate for the given set of training samples. If the pair (ϕ, p) is too simple to approximate the true distribution q(x), then Fn (ϕ, p) ≈ −

n 

log p(Xi |w ∗ )

i=1

≈ nK(q||p ∗ ) − nSn , where w∗ is the parameter that minimizes K(q||p ∗ ), p∗ (x) = p(x|w∗ ), and Sn is the empirical entropy, 1 log q(Xi ). n i=1 n

Sn = −

Since Sn does not depend on the pair (ϕ, p), the main term of Fn (ϕ, p) is determined by the functional approximation error K(q||p ∗ ) in this case. On the other hand, if K(q||p ∗ ) a =⇒ AH is chosen, where a is a parameter of the rule. Therefore the accuracy of the hypothesis test is determined by the function S(dn ) and a.

8.4 Singular hypothesis test

259

For a given hypothesis test with (S, a), its level Level(S, a) is defined by the probability that AH is chosen under the assumption that the true distribution is ϕ0 (w), Level(S, a) = P (S(dn ) > a|ϕ0 ). The level represents the probability that the null hypothesis is rejected although it is true. In practical applications, a is determined so that the level is sufficiently small, for example, sometimes a is determined such that Level(S, a) = 0.05 or L(S, a) = 0.01. Conversely, a given level p, the event {dn ; S(dn ) > a} such that p = P (S(dn ) > a|ϕ0 ) is said to be a rejection region. The power of a hypothesis test is defined by the probability that AH is chosen when the true distribution is ϕ1 (w), Power(S, a) = P (S(dn ) > a|ϕ1 ). The power represents the probability that the alternative hypothesis is chosen when it is true. Note that, if there is a monotone increasing function f such that S1 (dn ) = f (S2 (dn )) (∀dn ), then (S1 , f (a)) and (S2 , a) give equivalent tests. A test S1 is said to be more powerful than a test S2 if, for arbitrary a1 , a2 , Level(S1 , a1 ) = Level(S2 , a2 ) =⇒ Power(S1 , a1 ) ≥ Power(S2 , a2 ) holds. For a given set of hypotheses ϕ0 (w) and ϕ1 (w), there is an essentially unique test that is more powerful than any other test, which is called the most powerful test. Theorem 8.2 (Neyman–Pearson) The most powerful test is explicitly given by
n  dw ϕ1 (w) p(xi |w) i=1

L(dn ) ≡
dw ϕ0 (w)

n 

.

(8.10)

p(xi |w)

i=1

Proof of Theorem 8.2 Assume that the level of the test L( ) is equal to that of an arbitrary test S, that is to say, there exist a, b such that P (L(dn ) > b|ϕ0 ) = P (S(dn ) > a|ϕ0 ). It is sufficient to prove that P ∗ ≡ P (L(dn ) > b|ϕ1 ) − P (S(dn ) > a|ϕ1 )

(8.11)

260

Singular statistics

is nonnegative. Let A, B be two events defined by A = {dn ; S(dn ) > a}, B = {dn ; L(dn ) > b}. Then P∗ =







−

ϕ1 (w)dw


 n

B





=

A

p(xi |w)dxi

i=1


−

ϕ1 (w)dw B∩Ac

n  A∩B c

p(xi |w)dxi ,

i=1

where Ac is the complementary set of A. Note that B ∩ Ac ⊂ B and A ∩ B c ⊂ B c . By using the definition of L(d n ) and eq.(8.11), P∗ ≥










ϕ0 (w)dw B∩Ac





=

ϕ0 (w)dw

−

−

n  A∩B c


 n

B

A

p(xi |w)dxi

i=1

p(xi |w)dxi

i=1

= 0.




Therefore L(dn ) is the most powerful test.

Remark 8.4 If ϕ0 (w) = δ(w − w0 ) and ϕ1 (w) = δ(w − w1 ), then Neyman– Pearson theorem shows that the most powerful test results in the likelihood ratio test, n 

L(dn ) =

i=1 n 

p(xi |w1 ) . p(xi |w0 )

i=1

When the null hypothesis is defined by w0 and the alternative hypothesis is w = w0 , then the following test is sometimes employed, n 

ˆ n) = L(d

i=1 n 

p(xi |w) ˆ

p(xi |w0 ),

i=1

,

8.4 Singular hypothesis test

261

where wˆ is the maximum likelihood or a posteriori estimator. However, this is not the most powerful test. In singular learning machines, this test has often very weak power in general.

8.4.2 Example of singular hypothesis test To find the most powerful test, we need the probability distribution of
n  dw ϕ1 (w) p(xi |w) i=1

L(dn ) =
dw ϕ0 (w)

n 

, p(xi |w)

i=1

where dn = (x1 , x2 , . . . , xn ) are taken from w that is subject to ϕ0 (w). If the null hypothesis test is given by one parameter, ϕ0 (w) = δ(w − w0 ), then
L(dn ) = exp(−nKn (w)) ϕ1 (w) dw, where Kn (w) is the log likelihood ratio function, 1 p(xi |w0 ) log . n i=1 p(xi |w) n

Kn (w) =

As we have already proved in Chapter 6, under the condition that the null hypothesis is true, L(dn ) has an asymptotic expansion, L(dn ) = exp(−λ log n + (m − 1) log log n − F R ). To construct a hypothesis test, we need the probability distribution of the random variable F R . Example 8.1 (Hypothesis test of changing point) Let {x1 , x2 , . . . , xn } be a set of different, fixed points in R1 and dn = {y1 , y2 , . . . , yn } be a set of random variables which are subject to a conditional probability distribution. Let us construct a hypothesis test for a model,  1  1 p(y|x, a, b) = √ exp − 2 (y − aT (bx))2 , 2σ 2π σ 2 where σ 2 > 0 is a constant, and T (x) = tanh(x). The null and alternative hypotheses are respectively fixed as NH : ϕ0 (a, b) = δ(a)δ(b), AH : ϕ1 (a, b),

262

Singular statistics

where ϕ1 (a, b) is a fixed probability density function. The most powerful test is given by
L(dn ) =

n 2 1  exp − 2 (yi − aT (bxi ) ϕ1 (a, b)da db 2σ i=1 n  1   exp − 2 yi2 2σ i=1


=



n n  1    exp − 2 a 2 T (bxi )2 − 2 yi aT (bxi ) ϕ1 (a, b)da db. 2σ i=1 i=1

Under the condition that the null hypothesis is true and ϕ1 (0, 0) > 0, the a posteriori distribution converges to δ(a)δ(b) because the origin (a, b) = (0, 0) is the singularity of p(y|x, a, b) = p(y|x, 0, 0). By the Taylor expansion in the neighborhood of the origin, we have a tanh(bx) = abx + c1 ab3 x 3 + c2 ab5 x 5 + · · · ,

(8.12)

where c1 , c2 are constants. Since ab3 , ab5 , . . . , are contained in the ideal ab, they do not affect the asymptotic distribution. Therefore, if the null hypothesis is true, the most powerful test is asymptotically equal to
√ 2 2 (8.13) L∗ (dn ) ∼ = e−nAa b + nBab ϕ1 (a, b)da db, where

 (1/n) ni=1 xi2 , A= 2σ 2 √  (1/ n) ni=1 xi yi . B= σ2

(8.14) (8.15)

Here A is a constant and B is a random variable which is subject to the normal distribution with mean zero and variance 2A. Therefore, if the null hypothesis is true, the probability distribution of L(dn ) is asymptotically equal to that of
√ 2 2 (8.16) L∗ (g) ≡ e−nAa b +g 2nAab ϕ1 (a, b)da db, where g is a random variable which is subject to the standard normal distribution. Let us study the case when the alternative hypothesis is given by  1/4 (|a| ≤ 1, |b| ≤ 1, a = 0, b = 0) (8.17) ϕ1 (a, b) = 0 otherwise.

8.4 Singular hypothesis test

263

Then the random variable L∗ (g) is equal to
1
√ δ(t − ab) ∗ dt da db exp(−nAt 2 + g 2nAt) L (g) = 4 −1 [−1,1]2
1
√n
1  t  √ 1 2 da db √ δ √ − ab e−At +g 2At . = √ dt 4 n n − n −1 −1 By using the Mellin transform as in Chapter 4, we can prove
1
1   t |t| da db δ √ − ab = −2 log √ . n n −1 −1 Therefore,


∗

L (g) =

√

n

√ − n √
n

√ |t|  dt  2 √ − log √ e−At +g 2At 2 n n

√ |t|  dt  2 √ − log √ e−At +g 2At 2 n n 0
0  √ |t|  dt 2 + √ √ − log √ e−At +g 2At n − n 2 n √ √
√n |t|  −At 2  eg 2At + e−g 2At  dt  = √ − log √ e 2 n n 0
√n   √ |t| dt 2 = √ − log √ e−At cosh(g 2At). n n 0

=

By using this equation, we can construct a test for a given level. Let P0 ( ) be the standard normal distribution. For a given level  > 0, the function f () is determined so that  = P0 (|g| ≥ f ()) is satisfied. For example, if  = 0.05, then f () ≈ 1.96, or if  = 0.01, then f () ≈ 2.56. Then the most powerful test is determined by L(dn ) ≤ a() =⇒ NH L(dn ) > a() =⇒ AH, where a() is a function determined by
√n √ |t|  dt  2 a() ≡ √ − log √ e−At cosh(f () 2A t). n n 0

264

Singular statistics

Remark 8.5 In the above example, if the null hypothesis is true, then the approximation L(dn ) ∼ = L∗ (dn ) holds. Hence asymptotically L(dn ) > a() ⇐⇒ L∗ (dn ) > a() √ ⇐⇒ B > f () 2A holds. In other √words, the asymptotic test upon the null hypothesis can be done by B > b() 2A for a given level , which has the same level as the most powerful test. This test does not depend on the a priori probability distribution. However, it is not so powerful as the most powerful test. In fact, if the alternative hypothesis is true, then the approximation L(dn ) ≈ L∗ (dn ) does not hold. When we determine the reject region, we can use the approximation L(dn ) ≈ L∗ (dn ) because the reject region is determined by the null hypothesis. However, when we test the hypotheses, we should use L(dn ); we cannot use L∗ (dn ).

8.5 Realization of a posteriori distribution In singular statistical estimation, Bayes estimation is recommended because the a posteriori distribution has more information about singularities than one point estimation such as the maximum likelihood or a posteriori estimation. However, in singular statistical models, the a posteriori distribution has very singular shape which cannot be easily approximated. A good algorithm is necessary to construct the a posteriori distribution.

8.5.1 Markov chain Monte Carlo For a model p(x|w) (x ∈ RN , w ∈ Rd ) and an a priori distribution ϕ(w), a function H (w) is defined by H (w) = −

n 

log p(Xi |w) −

i=1

1 log ϕ(w). β

Then the a posteriori distribution is rewritten as p(w) = =

n  1 ϕ(w) p(Xi |w)β Zn i=1

1 exp(−βH (w)). Zn

8.5 Realization of a posteriori distribution

265

In the Metropolis algorithm, parameters {wt ∈ Rd ; t = 1, 2, 3, . . .} are sampled by the following procedure. Metropolis Algorithm (1) The initial point w1 is set. Let t = 1. (2) For a given wt , the new trial parameter w is sampled by w  = wt + N , where N is some fixed and symmetrical random variable, for example, a d-dimensional normal distribution. (3) If H (w  ) ≤ H (w), then wt+1 = w  . Otherwise, wt+1 = w  with probability exp(−β(H (w  ) − H (w))), or wt+1 = wt with probability 1 − exp(−β(H (w  ) − H (w))). (4) t := t + 1 and go to (2). Then the detailed balance condition p(wb |wa )p(wa ) = p(wa |wb )p(wb )

(∀wa , wb ∈ Rd )

is satisfied for every step, and the a posteriori distribution is a fixed point of this probabilistic iteration, which means that the probability distribution T 1  δ(w − wt ) T t=1

converges to the a posteriori distribution p(w) when n → ∞. The mean of a function f (w) by the a posteriori distribution is approximated by Ew [f (w)] ≈

T 1  f (wi ). T t=1

The evidence or stochastic complexity can be numerically calculated. The evidence can be given by
ˆ Z(β) = e−β H (w) ϕ(w)dw, where Hˆ (w) = −

n  i=1

log p(Xi |w).

266

Singular statistics

Then, by using Z(0) = 1, the evidence is equal to Z(1) =

J −1 j =0

=

J −1

Z(βk+1 )  Z(βk )

 ˆ Ew(βk ) e−(βk+1 −βk )H (w) ,

(8.18)

j =0

where E (β) [ ] shows the a posteriori distribution with the inverse temperature β > 0. The stochastic complexity is given by Fn = − log Z(1). To calculate E (β) [ ], the Metropolis algorithm with 1 H (w) = Hˆ − log ϕ(w) β is employed. Remark 8.6 (1) Let F (β) = − log Z(β). Then
1 dF (β)dβ F (1) = 0 dt
1 = E (β) [Hˆ (w)]dβ. 0

Equation (8.18) is essentially equivalent to this calculation. (2) In calculation of the evidence or the stochastic complexity, expectations with different β1 , . . . , βJ −1 are necessary: E (β1 ) [ ], E (β2 ) [ ], . . . , E (βJ −1 ) [ ]. In other words, a simultaneous probability distribution of W = (w1 , w2 , . . . , wJ −1 ), P (W ) =

J 

p(βj ) (wj ),

j =1

is needed, where p(j ) (w) =

1 exp(−βj Hˆ (w)). Z(βj )

For the purpose of sampling from the distribution P (W ), there is a method called the tempered or exchange Monte Carlo method. This method consists of two kinds of Markov chain Monte Carlo procedures. One is the ordinary process of Monte Carlo in each sequence in the distribution of βj . The other is

8.5 Realization of a posteriori distribution

267

the exchanging process of wj in βj -distribution and wj +1 in βj +1 -distribution. Two parameters wj and wj +1 are exchanged with probability exp{−(βj +1 − βj )(Hˆ (wj ) − Hˆ (wj +1 ))}. The exchanging process with this probability satisfies the detailed valance condition and makes P (W ) invariant, hence P (W ) is approximated as the limiting distribution by these two processes. In general, the equilibrium state for small β can be realized rather easily. The exchanging process is expected to make the Monte Carlo process for large β faster than the conventional method using the process with smaller β. By using singular learning theory, the exchange probability between β1 , β2 (β2 > β1 ) is asymptotically equal to 1 β2 − β1 (λ + 1/2) P (β1 , β2 ) = 1 − √ , (λ) π β1 where λ is the learning coefficient [63]. In the design of the sequence of the temperatures {βj }, the exchange probabilities need to satisfy P (β1 , β2 ) = P (β2 , β3 ) = · · · . For such a purpose, the geometrical progression is optimal for {βj }. Remark 8.7 The Markov chain Monte Carlo method is important in Bayesian estimation, and a lot of improved algorithms are being studied. The a posteriori distribution of a singular statistical model provides a good target distribution for the purpose of comparing several Monte Carlo methods, because the probability distributions used in practical applications are singular distributions or almost singular distributions. Singular learning theory is useful to construct a base on which Markov chain Monte Carlo methods are optimized, because the evidence is theoretically clarified. We can compare the numerical results by several Monte Carlo methods with theoretical results.

8.5.2 Variational Bayes approximation To find the maximum likelihood estimator of a mixture model, the expectation and maximization algorithm (EM) algorithm is sometimes employed. However, in singular statistical models, the maximum likelihood estimator (MLE) often does not exist or diverges. Even if it exists, the generalization error of MLE is very large, and hypothesis testing using MLE is very weak. Therefore, in singular learning machines, MLE is not appropriate for statistical estimation and hypothesis testing. Recently the EM algorithm was improved from the point of Bayes estimation, which is called the variational Bayes approximation

268

Singular statistics

or mean field approximation. Let us study a normal mixture model of x ∈ RM , p(x|w) =

K 

√

k=1

ak 2π

M

 1  exp − |x − bk |2 , 2

(8.19)

where K is the number of mixtures, and w = (a, b) = {(ak , bk ); k = 1, 2, . . . , K} is the set parameters which satisfies 0 ≤ ak ≤ 1, a1 + a2 + · · · + aK = 1, and bk ∈ RM . In variational Bayes approximation, we employ the conjugate prior ϕ(w) = ϕ1 (a)ϕ2 (b),  (Kφ0 )  φ −1 δ a − 1 ak 0 , k (φ0 )K k=1 k=1 K

ϕ1 (a) =

ϕ2 (b) =

K

K  β KM/2 

 β  0 exp − |bk − b0 |2 , 2 k=1

0

2π

where φ0 > 0, β0 > 0, and b0 ∈ RM are hyperparameters. Let us introduce a random variable Y = (Y 1 , Y 2 , . . . , Y K ) which takes values in the set C = {(1, 0, 0, . . . , 0), (0, 1, 0, . . . , 0), . . . , (0, 0, 0, . . . , 1)}. In other words, only one of {Yk } is equal to 1, and the others are equal to zero. Such a random variable is said to be a competitive random variable. By introducing a probability distribution of (x, y), p(x, y|w) =

K   k=1

√

ak 2π

M

 |x − b |2 y k k exp − , 2

where y = (y 1 , y 2 , . . . , y K ) ∈ C, it follows that p(x|w) is equal to the marginal distribution,  p(x, y|w), p(x|w) = y∈C



where y shows the sum of y over the set C. Therefore Y can be understood as a hidden variable in p(x, y|w). The sets of random samples are respectively denoted by dn = {xi ∈ RM ; i = 1, 2, . . . , n}, hn = {yi ∈ C; i = 1, 2, . . . , n},

8.5 Realization of a posteriori distribution

269

where dn ∈ RMn is the set of data and hn ∈ C n is the set of corresponding hidden variables. The simultaneous probability density function of (dn , hn , w) is given by P (dn , hn , w) = ϕ(w)

n 

p(xi , yi |w).

(8.20)

i=1

The conditional probability distribution of (hn , w) for a given set of data is equal to P (hn , w|dn ) = where Zn is a constant, Zn =




1 P (dn , hn , w), Zn

dw P (dn , hn , w)

hn ∈C n


=

dw ϕ(w)

n 

p(xi |w)

i=1

is the evidence of the pair p(x|w) and ϕ(w). Let us introduce the Kullback–Leibler distance K which is defined for the probability distributions on the set of hidden variables and the set of the parameters, C n × Rd . The Kullback–Leibler distance from an arbitrary probability density q(hn )r(w) to the target probability density P (hn , w|dn ) is equal to 
q(hn )r(w) dw q(hn )r(w) log K(q, r) = . P (hn , w|dn ) h ∈C n n

The set of all probability density functions on C n × Rd in which the hidden variable and the parameter are independent is denoted by S = {q(hn )r(w)}. The probability density function P (hn , w|dn ) is not contained in S in general. However, in the variational Bayes approximation, P (hn , w|dn ) is approximated by finding the optimal (q, r) ∈ S that minimizes K(q, r). The optimal (q, r) is said to be the variational Bayes approximation or mean field approximation. Minimization of K(q, r) is equivalent to minimization of 
q(hn )r(w) dw q(hn )r(w) log F(q, r) = . P (hn , w, dn ) h ∈C n n

If P (hn , w|dn ) is contained in S, equivalently if K(q, r) can be made to be zero, then the minimal value of F(q, r) is equal to Fn = − log Zn , which is

270

Singular statistics

the stochastic complexity of the original pair p(x|w) and ϕ(w). In general, the stochastic complexity of the variational Bayes is defined by Fˆn ≡ min F(q, r). (q,r)∈S

It follows that Fˆn gives the upper bound of the Bayes stochastic complexity, Fˆn ≥ Fn . Remark 8.8 (1) (Gibbs variational principle) In general, for a given function H (x), a functional of a probability distribution p(x) defined by

F(p) = q(x) log q(x)dx + β q(x)H (x)dx is minimized if and only if 1 exp(−βH (x)). Z This is called the Gibbs variational principle. It is well known in statistical mechanics that the equilibrium state p(x) is uniquely characterized by the minimization of the free energy. (2) The difference Fˆn − Fn is equal to the Kullback–Leibler distance from the variational Bayes to the true Bayes. Hence the difference can be understood as an index of how precise the variational Bayes is. (3) As is shown in Chapter 1, the Bayes generalization error Bg with β = 1 is equal to the increase of the mean stochastic complexity, p(x) =

E[Bg ] = E[Fn+1 ] − E[Fn ] for arbitrary natural number n. However, the variational Bayes generalization error Bˆ g does not satisfy the same relation. More explicitly, E[Bˆ g ] = E[Fˆn+1 ] − E[Fˆn ]. If the true parameter is originated from the a priori distribution, then Bayes estimation gives the smallest generalization error, hence E[Bg ] ≤ E[Bˆ g ] with β = 1. Otherwise, both cases E[Bˆ g ] ≥ E[Bg ] and E[Bˆ g ] ≤ E[Bg ] happen in general. (4) In general, the optimal q that minimizes K(q||p) is a more localized distribution than p. In other words, the variational Bayes or the mean field approximation gives the localized approximated distribution compared to the target. In singular learning machines, the a posteriori distribution is not localized. Hence the variational Bayes has different stochastic complexity and generalization error from the true Bayes estimation.

8.5 Realization of a posteriori distribution

271

To derive the variational Bayes learning algorithm, we need the Dirichlet distribution. Remark 8.9 (Dirichlet distribution) The Dirichlet distribution of a = (a1 , a2 , . . . , aK ) ∈ [0, 1]K is defined by    φ −1 1  ak ak k , δ 1− Z k=1 k=1 K

ϕ(a1 , a2 , . . . , aK ) =

K

where {φk > 0} is a set of hyperparameters and Z is the normalizing constant, #K Z=



k=1 (φk ) !. K k=1 φk

Then two averages satisfy
aj p(a1 , a2 , . . . , aK )

K 

φj dak = K k=1

k=1


(log aj )p(a1 , a2 , . . . , aK )

K 

φk

,

dak = ψ(φj ) − ψ

K 

k=1

 φk ,

k=1

where ψ(x) = (log (x)) is a digamma function. Let us derive an algorithm by which the variational Bayes approximation can be numerically found. Let the Lagrangian be L(q, r, α1 , α2 ) = F(q, r) + α1 + α2








q(hn ) − 1

hn

 r(w)dw − 1 .

The optimal q(hn ) and r(w) should satisfy the variational equations, L(q + δq, r) = L(q, r + δr) = 0 and ∂L/∂α1 = ∂L/∂α2 = 0. Then the optimal q(dn ) and r(w) should satisfy the relations   1 exp Er [log P (dn , hn , w)] , C1   1 r(w) = exp Eq [log P (dn , hn , w)] , C2

q(hn ) =

(8.21) (8.22)

272

Singular statistics

where Er [ ] and Eq [ ] are expectations over r(w) and q(hn ) respectively, and C1 , C2 > 0 are normalizing constants. By using eq.(8.20), for P = P (dn , hn , w),

log P =

K 

n 

(log ak )

k=1

−

+

1 2

i=1 K  n  k=1

K 

yik + β0



i=1

bk ·

n 

k=1

=



yik + φ0 − 1

yik xi + β0 b0 + A

(8.23)

i=1

n  K   i=1 k=1

1 1 log ak − |bk − xi |2 + log √ + B, 2 2π

(8.24)

where A is a constant for parameter w and B is a constant for yik . Let y ki = Eq [yik ] for a given q(dn ). Then by eq.(8.22) and eq.(8.23) K  T   k (ak )Sk −1 exp − |bk − Uk |2 , r(a, b) ∝ 2 k=1

where Sk , Tk and Uk are determined by y ki , Sk =

n 

y ki + φ0 ,

i=1

Tk =

n 

y ki + β0 ,

i=1

1  k y i xi + β0 b0 . Tk i=1 n

Uk =

Then by eq.(8.21) and eq.(8.24), q(hn ) can be obtained using r(a, b):

q(hn ) ∝

n  K  i=1 k=1

exp(yik Lki ),

8.5 Realization of a posteriori distribution

where

273

1M + |xi − Uk |2 , 2 Tk ! exp Lki k . y i = K j! j =1 exp Li

Lki = ψ(Sk ) −

Hence we obtain the recursive formula, (Sk , Tk , Uk ) ⇐ y ki , y ki ⇐ (Sk , Tk , Uk ). From some appropriate initial point, if this algorithm converges to the global minimum of F(q, r), then nM Fˆn = − log(2π) − 2 + log +

K 

n 

log

i=1

ψ(Sk )(Sk − φ0 ) −

K   M k=1

k

e Li



k=1

(n + Kφ0 )(φ0 )K (Kφ0 )(S1 ) · · · (SK )

k=1

+

K 

2

log Tk +

MK (1 + log β0 ) 2

 β0  M + |Uk − b0 |2 . 2 Tk

There is a theoretical bound of this value. Theorem 8.3 Assume that the true distribution is a normal mixture in eq.(8.19) with K0 components. Then the variational stochastic complexity Fˆn satisfies the inequality Sn + λ1 log n + nKn (w) ˆ + c1 < Fˆn < Sn + λ2 log n + c2 , where Sn is the empirical entropy of the true distribution, Kn (w) ˆ is the log likelihood ratio of the optimal parameter of wˆ with respect to the variational Bayes, c1 , c2 are constants, and λ1 , λ2 are respectively given by  (K − 1)φ0 + M/2 (φ0 ≤ (M + 1)/2) λ1 = (MK + K − 1)/2 (φ0 > (M + 1)/2)  (K − K0 )φ0 + (MK0 + K0 − 1)/2 (φ0 ≤ (M + 1)/2) λ2 = (MK + K − 1)/2 (φ0 > (M + 1)/2).

274

Singular statistics

Remark 8.10 (1) The proof of this theorem is given in [96]. This theorem shows that the variational Bayes has phase transition with respect to the hyper parameter φ0 = (M + 1)/2. (2) If the Dirichlet distribition is employed as the a priori distribution, then the learning coefficient λ of Bayes estimation is given by eq.(7.4) with K = H , K0 = H0 , M = N . The difference between Bayes stochastic complexity Fn ∼ = ˆ Sn + λ log n and the variational one Fn is not so large. (3) There are theoretical results of variational Bayes for general mixture models [97], hidden Markov models [42], and reduced rank regressions [65]. The generalization error by the variational Bayes estimation is still an open problem. In [65], the behavior of the variational generalization error is different from that of stochastic complexity.

8.6 From regular to singular In this book, we have shown that singular statistical theory is established by functional methods in modern mathematics. The reasons why functional methods are necessary are as follows. (1) The problem of singularities is resolved in the functional space. (2) Asymptotic normality of the limiting process made by training samples is shown only in the functional space. (3) On the functional space, statistical learning theory is completely described by its algebraic structure. Let us compare regular statistical theory with singular learning theory. The well-known conventional statistical theory of regular models consists of the following parts. (1) (2) (3) (4) (5)

The set of the true parameters is one point. The Fisher information matrix at the true parameter is positive definite. The log likelihood ratio function can be approximated by a quadratic form. The Bayes a posteriori distribution converges to the normal distribution. The distribution of the maximum likelihood estimator also converges to the normal distribution. (6) Both the learning coefficient and the singular fluctuation are d/2, where d is the dimension of the parameter space. On the other hand, singular learning theory consists of the following properties.

8.6 From regular to singular

275

Table 8.2. Correspondence between regular and singular Regular Algebra Geometry Analysis Probability theory Parameter set Model and parameter True parameter Fisher inform. matrix Cramer–Rao inequality Asymptotic normality ML estimator Bayes a posteriori Standard form Basic transform Learning coefficient Singular fluctuation Stochastic complexity Information criterion Phase transition Examples

Linear algebra Differential geometry Real-valued function Central limit theorem Manifold Identifiable One point Positive definite Yes Yes Asymptotic efficient Normal distribution Quadratic form Isomorphic d/2 d/2 (d/2) log n AIC No Normal, binomial Linear regression Linear prediction

Singular Ring and ideal Algebraic geometry Function-valued func. Empirical process Analytic set Nonidentifiable Real analytic set Semi-positive def. No meaning No Not efficient Singular dist. Normal crossing Birational λ ν λ log n WAIC Yes Mixtures Neural networks Hidden Markov

(1) The set of true parameters consists of a real analytic set with singularities. (2) The Kullback–Leibler distance is made to be normal crossing by resolution of singularities. (3) The log likelihood ratio function can be approximated by the standard form as in Main Theorem 6.1. (4) The Bayes a posteriori distribution converges to the singular distribution. (5) The distribution of the maximum likelihood estimator converges to that of the maximum value of the Gaussian process. (6) Neither the learning coefficient nor the singular fluctuation is equal to d/2, in general. Table 8.2 shows the correspondence between regular and singular learning theory. Regular statistical theory was built via the probability distribution on parameter space. Samples → Parameter space → Prediction.

276

Singular statistics

Singular learning theory is established via the probability distribution on functional space. Samples → Functional space → Prediction. In regular statistical theory, “statistic” is defined as a function from samples to parameter space. However, in singular learning theory it should be defined as a function from samples to functional space. On the functional space, asymptotic normality holds in both regular and singular cases. We have proved that, even in singular learning theory, there are universal formulas, which are mathematically beautiful and statistically useful. We expect that these are the case for the future study of statistical learning theory. Remark 8.11 In mathematical physics, quantum field theory and statistical mechanics are characterized by the probability distribution of exp(−βH (x)) where H (x) is a Hamiltonian function. It is well known in [12] that physical problems are determined by the algebraic structure of H (x). Statistical learning theory can be understood as mathematical physics where the Hamiltonian is a random process defined by the log likelihood ratio function. This book clarifies the point that the algebraic geometrical structure of the log likelihood ratio function determines the learning process.

Bibliography

[1] H. Akaike. A new look at the statistical model identification. IEEE Transactions on Automatic Control, Vol. 19, pp. 716–723, 1974. [2] H. Akaike. Likelihood and Bayes procedure. In Bayesian Statistics, ed. J. M. Bernald. University Press, Valencia, Spain, 1980, 143–166. [3] S. Amari. Differential Geometrical Methods in Statistics. Springer Lecture Notes in Statistics. Berlin: Springer-Verlag, 1985. [4] S. Amari. A universal theorem on learning curves. Neural Networks, Vol. 6, No. 2, pp. 161–166, 1993. [5] S. Amari and H. Nagaoka. Methods of Information Geometry. Oxford: AMS and Oxford University Press, 2000. [6] S. Amari and N. Murata. Statistical theory of learning curves under entropic loss. Neural Computation, Vol. 5, pp. 140–153, 1993. [7] S. Amari. Natural gradient works efficiently in learning. Neural Computation, Vol. 10, pp. 251–276, 1998. [8] S. Amari and H. Nakahara. Difficulty of singularity in population coding. Neural Computation, Vol. 17, No. 4, pp. 839–858, 2005. [9] S. Amari, H. Park, and T. Ozeki. Singularities affect dynamics of learning in neuromanifolds. Neural Computation, Vol. 18, No. 5, pp. 1007–1065, 2006. [10] M. Aoyagi and S. Watanabe. Stochastic complexities of reduced rank regression in Bayesian estimation. Neural Networks, Vol. 18, No. 7, pp. 924–933, 2005. [11] M. Aoyagi and S. Watanabe. Resolution of singularities and generalization error with Bayesian estimation for layered neural network. IEICE Transactions. Vol. J88-D-II, No. 10, pp. 2112–2124, 2005. [12] H. Araki. Mathematical Theory of Quantum Fields. International Series of Monographs on Physics. Oxford: Oxford University Press, 1999. [13] H. Araki. Relative entropy of states of von Neumann algebras. Publications of the Research Institute for Mathematical Sciences, Vol. 11, No. 3, pp. 809–833, 1975. [14] M. F. Atiyah. Resolution of singularities and division of distributions. Communications of Pure and Applied Mathematics, Vol. 13, pp. 145–150, 1970. [15] A. R. Barron. Approximation and estimation bounds for artificial neural networks. Machine Learning, Vol. 14, No. 1, pp. 115–133, 1994.

277

278

Bibliography

[16] I. N. Bernstein. The analytic continuation of generalized functions with respect to a parameter. Functional Analysis and Applications, Vol. 6, pp. 26–40, 1972. [17] J. E. Bj¨ork. Rings of Differential Operators. Amsterdam: North-Holland, 1970. [18] J. Bocknak, M. Coste, and M. -F. Roy. Real Algebraic Geometry. Berlin: Springer Verlag, 1998. [19] G. Bodn´ar and J. Schicho. A computer program for the resolution of singularities. In Resolution of Singularities, Progress in Mathematics, Vol. 181, ed. H. Hauser. Basel: Birkh¨auser, 1997, pp. 231–238. [20] H. Chernoff. On the distribution of the likelihood ratio. Annals of Mathematical Statistics, Vol. 25, pp. 573–578, 1954. [21] D. A. Cox, J. B. Little, and D. O’sea. Ideals, Varieties, and Algorithms, 3rd edn. New York: Springer, 2007. [22] H. Cramer. Mathematical Methods of Statistics. Princeton, NJ: Princeton University Press, 1949. [23] D. Dacunha-Castelle and E. Gassiat. Testing in locally conic models, and application to mixture models. Probability and Statistics, Vol. 1, pp. 285–317, 1997. [24] D. A. Darling and P. Erd¨os. A limit theorem for the maximum of normalized sums of independent random variables. Duke Mathematics Journal, Vol. 23, pp. 143–155, 1956. [25] B. Davies. Integral Transforms and their Applications. New York: Springer, 1978. [26] P. Diaconis and B. Sturmfels. Algebraic algorithms for sampling from conditional distributions. The Annals of Statistics, Vol. 26, No. 1, pp. 363–397, 1998. [27] W. Donoghue. Distributions and Fourier Transforms. New York: Academic Press, 1969. [28] M. Drton, B. Sturmfels, and S. Sullivant. Algebraic factor analysis: tetrads, pentads and beyond. Probability Theory and Related Fields, Vol. 138, No. 3–4, pp. 1432–2064, 2007. [29] B. Efron and C. Moriis. Stein’s estimation rule and its competitors – an Empirical Bayes approach. Journal of American Statistical Association, Vol. 68, pp. 117– 130, 1973. [30] K. Fujiwara and S. Watanabe. Hypothesis testing in singular learning machines and its application to time sequence analysis. IEICE Transactions, Vol. J91-D, No. 4, pp. 889–896, 2008. [31] K. Fukumizu. Likelihood ratio of unidentifiable models and multilayer neural networks. The Annals of Statistics, Vol. 31, No. 3, pp. 833–851, 2003. [32] I. M. Gelfand and G. E. Shilov. Generalized Functions. San Diego, CA: Academic Press, 1964. [33] I. I. Gihman and A. V. Shorohod. The Theory of Stochastic Processes, Vols. 1, 2, 3. Berlin: Springer-Verlag, 1974. [34] I. J. Good. The Estimation of Probabilities, Cambridge, MA: MIT Press, 1965. [35] K. Hagiwara. On the problem in model selection of neural network regression in overrealizable scenario. Neural Computation, Vol. 14, pp. 1979–2002, 2002. [36] J. A. Hartigan. A failure of likelihood asymptotics for normal mixtures. In Proceedings of the Berkeley Conference in Honor of J. Neyman and J. Kiefer, Vol. 2, ed. L. LeCam and R. A. Olshen. Belmoant, CA: Wadsworth, 1985, pp. 807–810.

Bibliography

279

[37] H. Hauser. The Hironaka theorem on resolution of singularities (Or A proof we always wanted to understand). Bulletin of the American Mathematical Society. Vol. 40, No. 3, pp. 323–403, 2003. [38] D. Haussler and M. Opper. Mutual information, metric entropy and cumulative relative entropy risk. The Annals of Statistics. Vol. 25, No. 6, pp. 2451–2492, 1997. [39] T. Hayasaka, M. Kitahara, and S. Usui. On the asymptotic distribution of the least-squares estimators in unidentifiable models. Neural Computation, Vol. 16, No. 1, pp. 99–114, 2004. [40] H. Hironaka. Resolution of singularities of an algebraic variety over a field of characteristic zero. Annals of Mathematics, Vol. 79, pp. 109–326, 1964. [41] L. H¨ormander. An Introduction to Complex Analysis in Several Variables. Princeton, NJ: Van Nostrand, 1966. [42] T. Hosino, K. Watanabe, and S. Watanabe. Stochastic complexities of variational Bayesian hidden Markov models. Proceedings of 2005 IEEE International Joint Conference on Neural Networks, Vol. 2, pp. 1114–1119, 2005. [43] H. Hotelling. Tubes and spheres in n-spaces, and a class of statistical problems. American Journal of Mathematics, Vol. 61, pp. 440–460, 1939. [44] M. Inoue, H. Park, and M. Okada. On-line learning theory of soft committee machines with correlated hidden units – Steepest gradient descent and natural gradient descent. Journal of Physical Society of Japan, Vol. 72, No. 4, pp. 805– 810, 2003. [45] S. Janson. Gaussian Hilbert Space. Cambridge University Press, 1997. [46] M. Kashiwara. B-functions and holonomic systems. Inventiones Mathematicae, Vol. 38, pp. 33–53, 1976. [47] M. Knowles and D. Siegmund. On Hotelling’s approach to testing for a nonlinear parameter in regression. International Statistical Review, Vol. 57, pp. 205–220. 1989. [48] J. Koll´ar. The structure of algebraic thresholds – an introduction to Mori’s program. Bulletin of the American Mathematical Society, Vol. 17, pp. 211–273, 1987. [49] J. Koll´ar. Lectures on Resolution of Singularities. Princeton, NJ: Princeton University Press, 2007. [50] J. Koll´or, S. Mori, C. H. Clemens, and A. Corti. Birational Geometry of Algebraic Varieties. Cambridge Tract in Mathematics. Cambridge University Press, 1998. [51] F. Komaki. On asymptotic properties of predictive distributions. Biometrika, Vol. 83, No. 2, pp. 299–313, 1996. [52] S. Kuriki and A. Takemura. Tail probabilities of the maxima of multilinear forms and their applications. The Annals of Statistics, Vol. 29, No. 2, pp. 328–371, 2001. [53] M. R. Leadbetter, G. Lindgren, and H. Rootz´en. Extremes and Related Properties of Random Sequences and Processes. Berlin: Springer-Verlag, 1983. [54] E. Levin, N. Tishby, and S. A. Solla. A statistical approaches to learning and generalization in layered neural networks. Proceedings of IEEE, Vol. 78, No. 10, pp. 1568–1674, 1990. [55] X. Liu and Y. Shao. Asymptotics for likelihood ratio tests under loss of identifiability. The Annals of Statistics, Vol. 31, No. 3, pp. 807–832, 2003.

280

Bibliography

[56] D. J. Mackay. Bayesian interpolation. Neural Computation, Vol. 4, No. 2, pp. 415– 447, 1992. [57] G. McLachlan and D. Peel. Finite Mixture Models. New York: John Wiley, 2000. [58] H. N. Mhaskar. Neural networks for optimal approximation of smooth and analytic functions. Neural Computation, Vol. 8, pp. 164–177. 1996. [59] E. Miller and B. Sturmfels. Combinatorial Commutative Algebra. Graduate Texts in Mathematics, vol. 227, New York: Springer-Verlag, 2005. [60] D. Mumford. The Red Book of Varieties and Schemes, 2nd edn. Berlin: SpringerVerlag, 1999. [61] N. Murata, S. Yoshizawa, and S. Amari. Network information criterion – determining the number of hidden units for an artificial neural network model. IEEE Transactions on Neural Networks, Vol. 5, No. 6, pp. 797–807, 1994. [62] M. Mustata. Singularities of pairs via jet schemes. Journal of the American Mathematical Society, Vol. 15, pp. 599–615, 2002. [63] K. Nagata and S. Watanabe. Asymptotic behavior of exchange ratio in exchange Monte Carlo method. Neural Networks, Vol. 21, No. 7, pp. 980–988, 2008. [64] S. Nakajima and S. Watanabe. Generalization performance of subspace Bayes approach in linear neural networks. IEICE Transactions, Information and Systems, Vol. E89-D, No. 3, pp. 1128–1138, 2006. [65] S. Nakajima and S. Watanabe. Variational Bayes solution of linear neural networks and its generalization performance. Neural Computation, vol. 19, no. 4, pp. 1112–1153, 2007. [66] K. Nishiue and S. Watanabe. Effects of priors in model selection problem of learning machines with singularities. Electronics and Communications in Japan (Part II: Electronics), Vol. 88, No. 2. pp. 47–58, 2005. [67] T. Oaku. Algorithms for b-functions, restrictions, and algebraic local cohomology groups of D-modules. Advances in Applied Mathematics, Vol. 19, pp. 61–105, 1997. [68] T. Oaku. Algorithms for the b-function and D-modules associated with a polynomial. Journal of Pure Applied Algebra, Vol. 117–118, pp. 495–518, 1997. [69] K. Oka. Sur les fonctions analytiques de plusieurs variables. Tokyo: Iwanami shoten, 1962. [70] M. Opper and D. Haussler. Bounds for predictive errors in the statistical mechanics of supervised learning. Physical Review Letters, Vol. 75, No. 20, pp. 3772– 3775, 1995. [71] L. Pachter and B. Sturmfels. Algebraic Statistics for Computational Biology, Cambridge University Press, 2005. [72] P. C. B. Pillips. Partially identified econometric models. Econometric Theory, Vol. 5, pp. 181–240, 1989. [73] K. R. Parthasarathy. Probability Measures on Metric Spaces. New York: Academic Press, 1967. [74] G. Pistone, E. Riccomagno, and H. Wynn. Algebraic Statistics: Computational Commutative Algebra in Statistics. Boca Raton, FA: Chapman and Hall/CRC, 2001. [75] J. Rissanen. Stochastic complexity and modeling. Annals of Statistics, Vol. 14, pp. 1080–1100, 1986. [76] D. Ruelle. Thermodynamic Formalism. Reading, MA: Addison Wesley, 1978.

Bibliography

281

[77] D. Rusakov and D. Geiger. Asymptotic model selection for naive Bayesian network. Journal of Machine Learning Research. Vol. 6, pp. 1–35, 2005. [78] M. Saito. On real log canonical thresholds, arXiv:0707. 2308v1, 2007. [79] M. Saito, B. Sturmfels, and N. Takayama. Gr¨obner deformations of hypergeometric differential equations. Algorithms and Computation in Mathematics, Vol. 6. Berlin: Springer, 2000. [80] M. Sato and T. Shintani. On zeta functions associated with prehomogeneous vector space. Annals of Mathematics, Vol. 100, pp. 131–170, 1974. [81] G. Schwarz. Estimating the dimension of a model. Annals of Statistics, Vol. 6, No. 2, pp. 461–464. 1978. [82] I. R. Shafarevich. Basic Algebraic Geometry. Berlin: Springer-Verlag, 1974. [83] R. Shibata. An optimal model selection of regression variables. Biometrika, Vol. 68, pp. 45–54, 1981. [84] K. E. Smith, L. Kahanp¨aa, P. Kek¨al¨ainen, and W. Traves. An Invitation to Algebraic Geometry. New York: Springer, 2000. [85] C. Stein. Inadmisibility of the usual estimator for the mean of a multivariate normal distribution. In Proceedings of the 3rd Berkeley Symposium on Mathematical Statistics and Probabilities. Berkeley, CA: University of California Press, 1956, pp. 197–206. [86] B. Sturmfels. Gr¨obner Bases and Convex Polytopes. University Lecture Series. American Mathematical Society, 1995. [87] M. Sugiyama and K.-R. M¨uller. The subspace information criterion for infinite dimensional hypothesis spaces. Journal of Machine Learning Research, Vol. 3, pp. 323–359, 2003. [88] M. Sugiyama and H. Ogawa. Optimal design of regularization term and regularization parameter by subspace information criterion. Neural Networks, Vol. 15, No. 3, pp. 349–361, 2002. [89] N. Takayama. An algorithm for constructing cohomological series solutions of holonomic systems, Journal of Japan society for symbolic and algebraic computation, Vol. 10, No. 4, pp. 2–11, 2003. [90] A. Takemura and T. Kuriki. On the equivalence of the tube and Euler characteristic methods for the distribution of the maximum of the gaussian fields over piecewise smooth domains. Annals of Applied Probability, Vol. 12, No. 2, pp. 768–796, 2002. [91] K. Tsuda, S. Akaho, M. Kawanabe, and K. -R. M¨uller. Asymptotic properties of the Fisher kernel. Neural Computation, Vol. 16, No. 1, pp. 115–137, 2004. [92] A. W. van der Vaart and J. A. Wellner. Weak Convergence and Empirical Processes. New York: Springer,1996. [93] V. N. Vapnik. Statistical Learning Theory. New York: John Wiley, 1998. [94] S. Veres. Asymptotic distributions of likelihood ratios for overparameterized ARMA processes. Journal of Time Series Analysis, Vol. 8, No. 3, pp. 345–357, 1987. [95] R. Walter. Principles of Mathematical Analysis. International Series in Pure and Applied Mathematics. New York: McGraw-Hill, 1976. [96] K. Watanabe and S. Watanabe. Stochastic Complexities of Gaussian Mixtures in Variational Bayesian Approximation. Journal of Machine Learning Research, Vol. 7, No. 4, pp. 625–644, 2006.

282

Bibliography

[97] K. Watanabe and S. Watanabe. Stochastic complexities of general mixture models in variational Bayesian learning. Neural Networks, Vol. 20, No. 2, pp. 210–217, 2007. [98] S. Watanabe. Generalized Bayesian framework for neural networks with singular Fisher information matrices. In Proceedings of the International Symposium on Nonlinear Theory and Its Applications, 1995, pp. 207–210. [99] S. Watanabe. Algebraic analysis for singular statistical estimation. Algorithmic Learning Theory, Lecture Notes on Computer Sciences, Vol. 1720. Springer, 1999, pp. 39–50. [100] S. Watanabe. Algebraic analysis for nonidentifiable learning machines. Neural Computation, Vol. 13, No. 4, pp. 899–933, 2001. [101] S. Watanabe. Algebraic geometrical methods for hierarchical learning machines. Neural Networks, Vol. 14, No. 8, pp. 1409–1060, 2001. [102] S. Watanabe. Learning efficiency of redundant neural networks in Bayesian estimation. IEEE Transactions on Neural Networks, Vol. 12, No. 6, 1475–1486, 2001. [103] S. Watanabe. Algebraic information geometry for learning machines with singularities. Advances in Neural Information Processing Systems, Vol. 13, pp. 329– 336, 2001. [104] S. Watanabe. Algebraic geometry of singular learning machines and symmetry of generalization and training errors. Neurocomputing, Vol. 67, pp. 198-213, 2005. [105] S. Watanabe. Algebraic Geometry and Learning Theory. Tokyo: Morikita Publishing, 2006. [106] S. Watanabe. Generalization and training errors in Bayes and Gibbs estimations in singular learning machines. IEICE Technical Report, December, 2007. [107] S. Watanabe. Equations of states in singular statistical estimation. arXiv:0712. 0653, 2007. [108] S. Watanabe. A forumula of equations of states in singular learning machines. In Proceedings of IEEE World Congress in Computational Intelligence, 2008. [109] S. Watanabe. On a relation between a limit theorem in learning theory and singular fluctuation. IEICE Technical Report, No. NC2008–111, pp. 45–50, 2009. [110] S. Watanabe and S. Amari. Learning coefficients of layered models when the true distribution mismatches the singularities. Neural Computation, Vol. 15, No. 5, pp. 1013–1033, 2003. [111] S. Watanabe, K. Yamazaki, and M. Aoyagi. Kullback information of normal mixture is not an analytic function. IEICE Technical Report, NC2004-50, 2004, pp. 41–46. [112] H. Wei, J. Zhang, F. Cousseau, T. Ozeki, and S. Amari. Dynamics of learning near singularities in layered networks. Neural Computation, Vol. 20, No. 3, pp. 813–843, 2008. [113] H. Weyl. On the volume of tubes. American Journal of Mathematics, Vol. 61, pp. 461–472, 1939. [114] K. Yamanishi. A decision-theoretic extension of stochastic complexity and its applications to learning. IEEE Transactions on Information Theory. Vol. 44, No. 4, pp. 1424–1439, 1998.

Bibliography

283

[115] K. Yamazaki and S. Watanabe. A probabilistic algorithm to calculate the learning curves of hierarchical learning machines with singularities. IEICE Transactions, Vol. J85-D-II, No. 3, pp. 363–372, 2002. [116] K. Yamazaki and S. Watanabe. Stochastic complexity of Bayesian networks. Proceedings of International Conference on Uncertainty in Artificial Intelligence, 2003. [117] K. Yamazaki and S. Watanabe. Singularities in mixture models and upper bounds of stochastic complexity. Neural Networks, Vol. 16, No. 7, pp. 1029–1038, 2003. [118] K. Yamazaki and S. Watanabe. Singularities in Complete bipartite graph-type Boltzmann machines and upper bounds of stochastic complexities. IEEE Transactions on Neural Networks, Vol. 16, No. 2, pp. 312–324, 2005. [119] K. Yamazaki and S. Watanabe. Algebraic geometry and stochastic complexity of hidden Markov models. Neurocomputing, Vol. 69, pp. 62–84, 2005. [120] K. Yamazaki, M. i Kawanabe, S. Watanabe, M. Sugiyama, and K. -R. M¨uller. Asymptotic Bayesian generalization error when training and test distributions are different. In Proceedings of International Conference on Machine Learning, Corvalis, Oregon, 2007, pp. 1079–1086.

Index

a posteriori distribution, 19 a priori distribution, 19 affine space, 80 AIC, 39 almost surely convergence, 46 alternative hypothesis, 258 analytic continuation, 49 analytic function, 48 analytic isomorphism, 54 analytic manifold, 72 associated convergence radii, 142 asymptotic normality, 31 asymptotically uniformly integrable, 138 b-function, 128 Bayes estimation, 19 Bayes generalization error, 177 Bayes quartet, 22 Bayes training error, 177 Bayesian network, 233 Bernstein-Sato polynomial, 128 BIC, 34 birational invariant, 65 blow-up, 91 blow-up with center, 91 Borel field, 43 bounded variation, 116 C 0 , C ∞ Class function, 105 Cauchy’s integral formula, 142, 143 central limit theorem, 144 central part of partition function, 124 complete metric space, 42 Completeness of Schwartz distribution, 108 consistency of estimation, 208

convergence in law, 133 convergence in mean, 46 convergence in probability, 46 coordinate ring, 85 covariance matrix, 144 Cramer–Rao inequality, 8 critical point, 53 defining ideal, 81 delta function, 107 desingularization, 94 detailed balance condition, 265 dimension of real algebraic set, 86 Dirichlet distribution, 271 EM algorithm, 267 Empirical process, 149 empirical variance, 180 essential family of local coordinates, 172 Evidence, 168 evidence, 19 exceptional set, 93 expectation, 45 factor theorem, 67 family of open sets, 72 finite critical values, 70 fluctuation function, 154 free energy, 20, 168 function valued analytic function, 141 functional metric space, 148 fundamental condition II, 169 generalization error, 4 generalized a posteriori distribution, 177 Gibbs estimation, 22

284

Index

Gibbs generalization error, 177 Gibbs training error, 177 Gibbs variational principle, 270 gradient vector, 53 H¨older’s inequality, 141 Haussdorff space, 72 Heaviside function, 109 hidden Markov model, 234 Hilbert’s basis theorem, 79 Hilbert’s Nullstellensatz, 83 Hironaka’s theorem, 58, 97, 98 homogeneous ideal, 88 hyperparamater, 268 hypothesis test, 258 ideal, 78 identifiability, 10 implicit function theorem, 57 integral domain, 85 inverse function theorem, 56 inverse Mellin transform, 116 inverse temperature, 19 irreducible, 84, 85 irreducible set, 83 Jacobian determinant, 56 Jacobian matrix, 56 Jeffreys’ prior, 221 Kolmogorv’s extension theorem, 145 Kullback–Leibler distance, 3 large deviation, 46 Laurent series expansion, 131 learning coefficient, 172, 217 level of hypothesis test, 259 likelihood function, 5 local coordinate, 72 local maximum, 53 local maximum point, 53 locally integrable, 107 log density ratio function, 5 log likelihood ratio function, 5 manifold, 72 marginal likelihood, 168 maximal ideal, 84 maximum a posteriori estimator, 25 maximum likelihood estimator, 25 MDL, 34

mean, 45 mean field approximation, 268 measurable function, 44 measurable space, 43 Mellin transform, 116 metric, 42 metric space, 42 Metropolis method, 265 most powerful test, 259 multi-index, 48, 120 Noetherian ring, 79 normal crossing, 66 normal mixture, 231 null hypothesis, 258 orientable, 88 partial expectation, 45 partition function, 123, 168 partition of integral, 76 partition of parameter space, 170 partition of unity, 73 phase transition, 215 Polish space, 42 polynomial, 48 polynomial ring, 77 positive definite metric, 10 power of hypothesis test, 259 power series, 48 pre-empirical process, 144 predictive distribution, 19 prime ideal, 83 probability density function, 44 probability distribution, 44 probability measure, 43 probability space, 43 Prohorov’s theorem, 150 proper map, 58 quasiprojective variety, 91 radical ideal, 82 radical of ideal, 82 random variable, 44 real algebraic set, 51 real algebraic variety, 97 real analytic set, 52 real log canonical threshold, 65 real projective space, 87 real projective variety, 89

285

286

Index

reduced rank regression, 228 regular model, 10 regular statistical model, 182 relative entropy, 3, 4 resolution of singularities, 58 Riemann zeta function, 132 saddle point, 54 Sard’s theorem, 58 Schwartz distribution, 107 Schwartz distribution with a regular integral, 107 separable metric space, 42 set-valued random variable, 44 several complex variables, 142 shrinkage estimation, 245 sigma algebra, 43 simultaneous resolution, 69 singular fluctuation, 155, 196, 203 singular integral, 105 singular model, 10 singularity, 55 standard form of log likelihood ratio function, 165

state density function, 114 stochastic complexity, 20, 168 strict transform, 93 tempered Monte Carlo, 266 three-layered neural network, 228 tight process, 149 Topological space, 72 toric modification, 104 total transform, 93 training error, 4 trinomial mixture, 232 uniformly continuous, 134 variational Bayes, 268 WAIC, 254 weak convergence, 133 Weierstrass approximation theorem, 149 Zariski topology, 90 zeta function, 128 zeta function of a statistical model, 31, 172