No title

Pattern Recognition 33 (2000) 177}184 A geometric approach to consistent classi"cationq Yoram Baram* Department of Comp...

Author: Elsevier

120 downloads 359 Views 10MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Pattern Recognition 33 (2000) 177}184

A geometric approach to consistent classi"cationq Yoram Baram* Department of Computer Science, Technion, Israel Institute of Technology, Haifa 32000, Israel Received 19 February 1998; received in revised form 14 September 1998; accepted 15 January 1999

Abstract A classi"er is called consistent with respect to a given set of class}labeled points if it correctly classi"es the set. We consider classi"ers de"ned by unions of local separators (e.g., polytopes) and propose algorithms for consistent classi"er reduction. The proposed approach yields a consistent reduction of the nearest-neighbor classi"er, relating the expected classi"er size to a local clustering property of the data and resolving unanswered questions raised by Hart (IEEE Trans. Inform. Theory IT-14(3) (1968)) with respect to the complexity of the condensed nearest neighbor method. ( 1999 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Classi"cation; Local separation; Consistent reduction; Nearest neighbor; Condensed nearest neighbor; Reduced nearest neighbor

1. Introduction Solutions to the classi"cation problem have been characterized in terms of parameterized separation surfaces. In statistical classi"cation (e.g., [1]), such surfaces may represent the boundaries of the support sets for probability density functions or the intersection surfaces of such functions. Other solutions have been based on the direct construction of separation surfaces. A simple example is that of a hyperplane [2]. The construction of separation surfaces of complexity higher than that of a single hyperplane or a single sphere has been limited to weighted sums of such simple elements (e.g., [3,4]). Probabilistic characterizations of the classi"cation power of such elements were presented by Cover [5] and by Vapnik and Chervonenkis [6]. Yet the order of the separation surface, or the classi"er size, has been largely selected in a trial and error fashion. Furthermore, characterizing the classi"ability of a data set with respect to its geometric properties, such as clustering, has largely remained an open problem [7,8]. q This work was supported in part by the NASA Ames Research Center and in part by the Fund for the Promotion of Research at the Technion. * Tel.: #972-4-8294356; fax: #972-4-8221128 E-mail address: [email protected] (Y. Baram)

Parametric classi"ers do not necessarily classify the sample points used in their design correctly. A single hyperplane cannot correctly classify a given set of class}labeled points, unless the latter are linearly separable. Such misclassi"cation is often intentional, as, in certain cases, the underlying assumption is that the data is `noisya and a relatively simple separation surface provides a `smoothinga e!ect. However, in many classi"cation problem the data is the only knowledge available, and there is no reason to assume that it represents a simple model of a particular structure. Regarded as the `trutha, the data should then be classi"ed correctly by the classi"er. We call classi"ers which correctly classify the data consistent. The nearest-neighbor classi"er, possessing attractive accuracy properties [9] is perhaps the most popular non}parametric classi"cation method. It is, by de"nition, consistent. In an attempt to reduce the memory requirement of the nearest-neighbor method, Hart [10] proposed an algorithm which "nds a consistent subset of the class}labeled sample points, that is, a subset which, when used as a stored reference set for the nearest-neighbor rule, correctly classi"es all of the remaining sample points. Questions on the complexity of the algorithm and the expected size of the resulting consistent subset were raised, but left unanswered. In this paper we present a geometric approach to consistent classi"cation. We observe that the nearest

0031-3203/99/$20.00 ( 1999 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 1 - 3 2 0 3 ( 9 9 ) 0 0 0 5 0 - 3

178

Y. Baram / Pattern Recognition 33 (2000) 177}184

neighbor criterion imposes a division on Rn which is similar to that of the Voronoi diagram [11]. However, while the Voronoi cells are the minimum}distance domains of each of the points with respect to its neighbors, our basic cells are the minimum}distance local separators of each of the points with respect to its neighbors of the competing class. Such local separators, like Voronoi cells, are multi}linear domains (or polytopes), but, in contrast to Voronoi cells, their number may be reducible without leaving `holesa in the input space. We call the average number of points of a given class that fall in the minimum distance local separator of a point of the same class the local clustering degree of the data. It is a measure of the classi"ability of the data, and it will make it possible to specify the expected sizes of classi"ers. The union of local separators of labeled points of the same class de"nes a cover for these points, which we call a separator. In the case of minimum}distance local separators, the domain covered by a separator corresponding to a class consists of points of Rn that would be assigned to the same class by the nearest-neighbor method. Moreover, the separator is exclusive: it does not cover any of the points of Rn which are closer to any of the points of the competing class. A new point will be assigned to a class if it falls under the corresponding separator. This is a crude way of performing nearest-neighbor classi"cation. It allows us, however, to "nd reduced consistent subsets, hence, reduced classi"ers. We propose a consistent reduction of the nearest-neighbor classi"er and, employing the local clustering degree of the data, derive bounds on its design complexity and on the expected classi"er size. The latter is also shown to bound the expected size of Hart's classi"er. Observing that the nearest-neighbor method de"nes a multi}linear separation surface between two classes, we consider, for comparison, direct consistent multi}linear separation. The performances of the proposed algorithms in classifying real data is compared to that of the nearest-neighbor method.

We call the set of those points of Rn that are closer to a point x3X than to any point of X the minimum} 1 2 distance local separator of x with respect to X . It is 2 generated by the following procedure: Connect each of the points of X to x and place a 2 hyperplane perpendicular to the line segment connecting the two points at the mid}point. Each of these hyperplanes divides the space into two half}spaces. The intersection of the half}spaces containing x de"nes the minimum}distance local separator of x3X with respect 1 to X . 2 The minimum}distance local separators of the points of X with respect to the set X can be found in a similar 2 1 manner. The de"nition of the minimum}distance local separator of a point resembles that of its Voronoi cell, but the two should not be confused. Given a "nite number of points, x(1) ,2, x(N)3Rn, the Voronoi cell of x(i) consists of those points of Rn that are at least as close to x(i) as to any other point x(j), jOi (see, e.g., [12]). The class assignment of the given points is immaterial. While each of the faces of a Voronoi cell is shared by another cell, the local separators of points of either the same or di!erent classes do not generally share faces. While the Voronoi cell of a point does not contain any of the other given points, the local separator of a point may contain other points of the same class. This di!erence is crucial to our purposes, since, in contrast to Voronoi cells, local separators may be eliminated without leaving `holesa in the data space. Fig. 1 shows the minimum}distance local separator of a point x3X , where X is represented by the set of 1 1 black}"lled circles, from the set X , represented by the 2 white}"lled circles. It should be noted that while only the positive part of R2 is considered in this illustration, local separators need not be bounded. It is quite obvious that the ability to classify a data set depends somehow on its structure. A correspondence between the classi"ability of a data set and the clustering

2. Local clustering Consider a "nite set of points X"Mx(i), i"1,2, NN in some subset of Rn, the real space of dimension n. Suppose that each point of X is assigned to one of two classes, and let the corresponding subsets of X, having N and N points, respectively, be denoted X and X . 1 2 1 2 We shall say that the two sets are labeled ¸ and ¸ , 1 2 respectively. It is desired to divide Rn into labeled regions, so that new, unlabeled points can be assigned to one of the two classes. We de"ne a local separator of a point x of X with 1 respect to X as a convex set, s(xD2), which contains x and 2 no point of X . A separator family is de"ned as a rule that 2 produces local separators for class}labeled points.

Fig. 1. Separating x3X from X by a minimum}distance local 1 2 separator.


of points in it has been noted by many authors. The underlying notion is that data points belonging to the same class are grouped, or clustered, in some geometrically consistent manner. However, not only is a formal characterization of this correspondence lacking, but, oddly enough, it appears that the notion of a data cluster has never been formally de"ned. Anderberg [7] states that `the term &cluster' is often left unde"ned and taken as a primitive notion, in much the same way as a &point' is treated in geometrya. Everitt [8] argues that a formal de"nition of a cluster is `not only di$cult but may even be misplaceda. He notes observations to this e!ect by Bonner [13], who suggested that the ultimate criterion for evaluating the meaning of such terms is the value judgment of the user, and by Cormack [14] and Gordon [15], who attempted to de"ne a cluster using such properties as `internal cohesion and external isolationa. Jain and Dubes [16] suggest that cluster analysis `should be supplemented by techniques for visualizing dataa. The general notion seems to be that while we cannot de"ne a cluster, we know it when we see one. However, biological vision is physically con"ned to two or, at most, three dimensions and, for our classi"cation purposes, we need a cluster characterization that will apply to higher dimensional spaces. We approach the classi"ability characterization problem by de"ning a clustering property of the data. Consider the minimum}distance local separator, s(xD2), of a point x3X with respect to X . Let s be the fraction of 1 2 points of X which are within s(xD2). Suppose that the 1 points of X and X are independently selected at ran1 2 dom from the class domains ¸ and ¸ in Rn. We de"ne 1 2 the local clustering degree of X with respect to X as the 1 2 expected value of s and denote it c . It is the probability 1@2 that, in independent sampling, a random point of ¸ will 1 be closer to a randomly chosen point of X than to any 1 point of X . The clustering degree of X with respect to 2 2 X is de"ned similarly and denoted c . The clustering 1 2@1 degree of the entire data set X is c"min Mc , c N. 1@2 2@1 Suppose, for instance, that X forms a single convex 1 cluster. Then its clustering degree will close to 1. If, on the other hand, the two sets X and X are highly mixed, 1 2 their clustering degrees will be close to 0. The clustering degree makes it possible to quantify the classi"ability of a given data set. The higher the clustering degree of the data, the more classi"able it is. If the clustering degree is high, then, as we shall see, few local separator will su$ce for solving the classi"cation problem. If it is low, then the classi"cation problem is hard and many local separators will be required for solving it.

3. Consistent reduction of the nearest-neighbor classi5er The nearest-neighbor classi"cation criterion assigns to ¸ an unlabeled point that is closer to a labeled point 1

179

of ¸ than to any labeled point of ¸ , and vice versa. The 1 2 rationale behind it is that a point which is close to a labeled point is `similara to it and should be labeled the same way. The Condensed Nearest-Neighbor (CNN) algorithm proposed by Hart [10] "nds a consistent subset of the data, that is, a subset which, when used as a reference set for the nearest-neighbor classi"er, correctly classi"es the remaining data points. The algorithm is described below: Hart's CNN algorithm: Bins called STORE and GARBAGE are set. The data is "rst placed in GARBAGE while STORE is empty. The "rst sample is placed in STORE. The ith sample is classi"ed by the nearestneighbor rule, using as a reference set the current contents of STORE. If classi"ed correctly, it is placed in GARBAGE; otherwise it is placed in STORE. The procedure continues to loop through GARBAGE until termination. Hart has left "nding the complexity of the algorithm and the expected size of the resulting consistent subset (that is, the resulting classi"er size) as unsolved problems. The following algorithm for "nding a consistent subset, which, for distinction, we call the Reduced NearestNeighbor (RNN) algorithm, is simpler than the CNN algorithm, and its expected computational complexity and the expected size of the resulting consistent subset are relatively easy to derive. Algorithm RNN. For convenience, let the data be given in separate records of X and X . Let the sets of points 1 2 of X and X already selected for the consistent subset 1 2 be denoted X and X , respectively. For each point 1,S 2,S of X put in X , "nd the minimum}distance local 1 1,S separator with respect to X . Include a new point of 2 X in X only if it does not fall in any of the local 1 1,S separators of the points already in X . The construction 1,S of X is similar, with the sets X and X interchanging 2,S 1 2 roles. A more detailed implementation of the RNN algorithm is given below. 1. A point x of X is placed in X . 1 1 1,S 2. For the ith point, x , of X , perform the following: i 1 (a) For the jth point, y , of X , "nd the distance d(i, j) j 1,S from x to y . i j (b) Find the minimum distance d (iD2) from x to the .*/ i set X . 2 (c) If d(i, j))d (iD2), eliminate x from X ; otherwise, .*/ i 1 increase j by 1 and go to (a). 3. If x is not eliminated, add it to X . Increase i by i 1,S 1 and go to (2), unless X is empty. 1 The complexity of the algorithm and the expected size of the resulting consistent set are speci"ed by the following result:

180


Theorem 1. The complexity of Algorithm RNN is O(N2). The expected size of the resulting consistent set is no greater than (1!c)N#1. Proof. For each of the points of X , only the distances to 1 some of the other points of X and to all the points of 1 X are required ("nding the minimal distances comes at 2 the same cost), which proves the "rst part of the theorem. The second part follows immediately from the geometric interpretation of the algorithm: Consider the "rst point, say, x, included in X . Its local separator contains, on 1,S average, a fraction c of the points of X . Since these 1@2 1 will be discarded down the line, a fraction 1!c will, 1@2 on average, remain (the remaining set may, of course, be further reduced). Since x itself also remains, the assertion follows. h The analysis of the RNN algorithm, which, in contrast to the CNN algorithm, avoids repetitious calculations, facilitates an analysis of the CNN algorithm, which is summarized by the following result: Theorem 2. The complexity of the CNN algorithm is O(N3). The expected size of the consistent subset generated by the CNN algorithm, is no greater than (1!c)N#1. Proof. It is not di$cult to see that there cannot be more than N runs through GARBAGE, with the number of points in GARBAGE reduced by, at least, one on each iteration. On each iteration, the distance between each of the points in GARBAGE (at most N) and each of the points in STORE (at most N) is calculated ("nding the shortest distance comes at the same cost). It follows that the complexity of CNN is O(N3). The expected size of the resulting consistent subset is, however, smaller than that of the RNN algorithm. This is a consequence of the fact that the RNN algorithm adds a point of the "rst class to the consistent subset if it is closer to the set of all the points of the second class than to the points of the "rst class already included in the subset, while the CNN algorithm adds a point of the "rst class to the consistent subset if it is closer to the smaller set of points of the second class already included in the subset than to the points of the "rst class already included in the subset. Since, in independent sampling, a larger set of random points is likely to be closer to a random point than a smaller set, the RNN algorithm is likely to include more points in the consistent subset than the CNN algorithm. It follows that the expected size of the consistent subset generated by the CNN algorithm, is no greater than (1!c)N#1, as asserted. h The higher complexity of the CNN algorithm is explained by the fact that, since data points that are correctly classi"ed are returned to GARBAGE, there are

repetitions of distance calculations for the same pairs of points and, subsequently, comparisons between the same distances. The RNN algorithm avoids such repetitions.

4. Consistent multi+linear classi5cation We de"ne a separator S(1D2) of X with respect to X as 1 2 a set that includes X and excludes X . 1 2 Given a separator family, the union of local separators s(x(i)D2) of the points x(i), i"12, N , of X with respect 1 1 to X , 2 S(1D2)"Z

s(x(i)D2) x(i)3X1

(1)

is a separator of X with respect to X . It consists of 1 2 N local separators. 1 Let X be a subset of X . The set 1,c 1 S (1D2)"Z (i)3 s(x(i)D2) c x X1,c

(2)

will be called a consistent separator of X with respect to 1 X if it contains all the points of X . The set X will 2 1 1, c then be called a consistent subset with respect to the given separator family. Given a separator, S(1D2), a new point will be assigned to X if it is included in S(1D2). It should be noted that 1 S(1D2) and S(2D1) may, but need not, be either mutually exclusive or complementary, even when they are based symmetrically on the same rules, interchanging the roles of X and X . Consequently, the separators S(1D2) and 1 2 S(2D1) need not produce the same results. Employing minimum}distance local separators, the nearest-neighbor method will assign a point to ¸ if it is 1 covered by S(1D2) (note that in this case S(1D2) and S(2D1) are mutually exclusive and complementary). This, however, is not an e$cient approach to nearest-neighbor classi"cation. The memory requirement for each of the minimum distance separators of the points of X with 1 respect to X is N . Since there are N points in X , the 2 2 1 1 memory requirement of S(1D2) is O(N N )"O(N2). In 1 2 contrast, the memory requirement for the direct nearestneighbor method ("nding the minimum distance from a new point to the labeled set) is O(N #N )"O(N). 1 2 Yet, the approach has a conceptual value for understanding reduced classi"ers of the form S (1D2). c Suppose that we have the Voronoi cells of the combined set X"X XX (that is, all points are treated 1 2 equally, regardless of class assignment). Then, clearly, the union of the Voronoi cells of the points of X de"nes the 1 domain of points which will be assigned to ¸ according 1 to the nearest-neighbor criterion. Any point outside this union will be assigned to ¸ . Since the Voronoi cells are 2 disjoint, none can be eliminated without leaving `holesa in the input space (including, of course, the point whose


cell is eliminated). Since the number of data points may be large, the use of Voronoi cells in classi"cation may be impractical. In addition, the complexity of constructing a Voronoi diagram in high-dimensional space may be considerably higher than that of constructing a minimum}distance separator. (It is known ([17]) that the complexity of constructing a Voronoi diagram is O(N log N) in R2 and N2 log N in R3. Results for 2 2 higher dimensional spaces do not appear to be available). A reduced minimum distance (RMD) classi"er may be produced by selecting a consistent subset of the labeled points corresponding to one of the two classes, whose minimum}distance local separators cover the entire set. This may be done by selecting an arbitrary labeled point for the consistent subset, eliminating all the labeled points of the same class that are within its local separator, selecting an arbitrary point from the remaining points of the same class, etc. Such a selection is illustrated by Fig. 4. It shows that the minimum}distance separators of three out of the "ve points of X su$ce for construct1 ing a separator for X and the local separators of three of 1 the four points of X cover X . In each case, although the 2 2 sample set is reduced, the remaining points, constituting a consistent subset, still produce a separating surface between the two sets of class}labeled points. The surface is the boundary of the union of their local separators. It can be seen that the boundaries produced for the two cases are di!erent. Each may be used for classi"cation, producing possibly di!erent results for new points. If all the minimum}distance separators were used, the two boundaries would be identical (see Fig. 2). The nearest-neighbor criterion, and the CNN, RNN and RMD classi"ers impose multi}linear separations between the classes. We note that multi}linear separation as a sole criterion has been used in previous classi"cation concepts, such as the perceptron [2] and neural networks consisting of linear threshold elements [18]. Next, we present a direct, seemingly natural construction of a consistent multi}linear classi"er. This classi"er may not pos-

Fig. 2. (a) The minimum}distance local separators of x , x and 1 2 x and the associated separator of X with respect to X . (b) The 3 1 2 minimum}distance local separators of y , y and y and the 1 2 3 associated separator of X with respect to X . 2 1

181

sess good predictive abilities, and its memory requirement may be higher than that of the nearest-neighbor method. Yet, it will be of value as a reference for performance evaluation (showing that consistency does not guarantee quality). A multi}linear local separator (MLLS) of a point x of X with respect to X , denoted s(xD2) is generated by the 1 2 following algorithm: Algorithm MLLS. 1. Let XI "X . 2 2 2. Find the point of XI nearest to x. 2 3. Place a hyperplane perpendicular to the line segment connecting the two points at the midpoint. 4. Let the new XI be the set of points of the old XI which 2 2 are on the side of the hyperplane which includes x. 5. While XI is non}empty, go to (2). 2 The local multi}linear separator is the intersection of the half}planes on the sides of the hyperplanes found in step (2) which include x. The local multi}linear separator of a point of X with respect to the set X can be found in 2 1 a similar manner. The union of the multi}linear local separators of the points x(i), i"1 ,2, N , of X with respect to X , 1 1 2 is a separator of X with respect to X . It consists 1 2 of N local separators. A reduced separator, consisting, 1 generally, of less local separators, is generated by the following Multi}Linear Classi"er (MLC) design algorithm, which is written in terms of S(1D2), but is applicable to S(2D1), with an obvious change of variables. Algorithm MLC. 1. Let XI "X . 1 1 2. Select an arbitrary point of XI . Find its multi}linear 1 separator with respect to X , employing MLLS. 2 3. Let the new XI be the set of points of the old XI which 1 1 are outside the separator found in (1). 4. While XI is non}empty, go to (2). 1 The separator is the union of the local separators found in step (2). In the rest of this paper we shall use the terms multi}linear local separator and multi}linear separator to represent the objects generated by algorithms MLLS and MLC, respectively. Fig. 3 shows the multi}linear local separators of points of X and X , and the resulting separators (indicated by 1 2 the thick lines). These can be seen to be di!erent from the ones shown in Fig. 2 for reduced minimum}distance classi"cation. We de"ne the size of the multi}linear classi"er as the total number of local separator faces, which represents the total amount of memory needed for implementing the classi"er. Bounds on the complexity of the algorithm and on the expected size of the classi"er are speci"ed next.

182


5. Examples

Fig. 3. (a) The multi}linear local separators of x , x and x and 1 2 3 the associated separator of X with respect to X . (b) The 1 2 multi}linear separators of y and y and the associated separ1 2 ator of X with respect to X . 2 1

Theorem 3. The complexity of Algorithm MLC is O(N2). The expected size of the resulting classixer is smaller than 0.25[(1!c)N#2]2.

Proof. Consider "rst the procedure of "nding the separator of an arbitrary point x of X . Finding the point y of 1 X nearest to x for constructing the "rst face of the 2 separator is, clearly, O(N ). Each of the points of X is 2 2 now checked for being on the side opposite to x of the hyperplane separating x from y, in which case it is eliminated from the remainder of this procedure. This is another O(N ) process. On average, fewer than N (1!c ) 2 2 2@1 of the points of X remain after this step (since the 2 expected total number of points of X outside the separ2 ator of y is N (1!c ), there are, on average, fewer than 2 2@1 that number of points of X on the side of the hyperplane 2 which includes x). This procedure has to be repeated for the subset of X , which falls outside the local separator of 1 x. The size of this subset is, clearly, smaller than N , and 1 its expected value is N (1!c ). Since N N )0.25N2, 1 1@2 1 2 and since, on the one hand, the face corresponding to y belongs to the separator of x, and, on the other, x belongs to consistent subset de"ning the separator of X , 1 the assertion follows. h The following modi"cation of the MLC algorithm will generally reduce the size of the classi"er. Algorithm MLCR. Same as Algorithm MLC, but at step (2), instead of randomly selecting a point from XI , select 1 the one whose separator has the largest volume of all the separators of the points of XI . Since the separator vol1 umes are generally hard to calculate, select the point whose separator contains the maximal number of points of XI . 1 The classi"er constructed by the MLCR algorithm can be expected to have a smaller size than the one employing random selection, since each separator selected for S(1D2) will potentially contain more points of X , and less 1 separators will be needed for covering the set X . 1

Example 1. Given a sequence of k daily trading (`closea) values of a stock, it is desired to predict whether the next day will show an increase or a decrease with respect to the last day in the sequence. An initial value of k"3 is selected, but it is increased if there are ambiguities (di!erent predictions for the same inputs) in the training data. Records for 10 di!erent stocks, each containing, on average, 1260 daily values were used. About 60% of the data were used for training and the rest for testing. The results achieved by the nearest-neighbor (NN), the condensed nearest-neighbor (CNN), the reduced nearest-neighbor (RNN) and the multi}linear (ML) classi"ers are given in Fig. 4, for each of the stocks, whose code names are speci"ed in the leftmost column. The memory reduction rates achieved by the CNN and the RNN algorithms with respect to the nearest-neighbor method were 37.3 and 35.3%, respectively. The ML algorithm transformed the data into a set of, on average, 0.2N separators, each possessing "ve faces, hence, no data reduction. The table shows that, on average, the nearest-neighbor method has produced the best results. The performances of the CNN and the RNN classi"ers (the latter producing only slightly better results) are somewhat lower and that of the ML classi"er is lower yet. This shows that di!erent consistent classi"ers do not necessarily produce results of similar quality. While the average performances are of certain interest, the individual results of each of the algorithms for each of the stocks are of practical signi"cance, since in actual stock trading, one has the choice of both the prediction method and the stock to be traded. Example 2. The Pima Indians Diabetes Database [19] has 768 instances of eight real}valued measurements, corresponding to 268 ill patients and 500 healthy ones. These two classes were found to be highly mixed and di$cult to characterize, examining any subset or linear combination of the measurements, including the principal directions of the covariance. Five di!erent training sets were de"ned, as shown in Fig. 5, with the rest of the data serving as test sets. The success rates achieved by the di!erent classi"ers are given in Fig. 5 for each of the cases, along with the average values. The results are similar in nature to those of the previous example.

6. Conclusion Solutions to the consistent classi"cation problem have been speci"ed in terms of local separators of data points of one class with respect to the other. Reduced consistent classi"ers for the nearest-neighbor and the multi}linear separation criteria have been presented, and their design


183

Fig. 4. Success rates in the prediction of rise and fall in stock values.

Fig. 5. Success rates in the classi"cation of the diabetes data base.

complexities and expected sizes have been speci"ed in terms of the local clustering degree of the data.

7. Summary The nearest-neighbour classi"cation criterion imposes a division on Rn which is similar to that of the Voronor diagram. However, while the Voronoy cells are the minimum}distance domains of each of the points with respect to its neighbors, the basic cells of the nearestneighbor classi"cation method are the minimum} distance local seperators of each of the pionts with respect to its neighbors of the competing class. Such local seperators, like Voronoy cells, are multi}linear domains

(or polytypes), but, in contrast to Voronoy cells, their number may be reducible without leaving `holesa in the input space. The average number of points of a given class that fall in the minimum}distance local separator of a point of the same class is called the local clustering degree of the data. It is a measure of the classi"ability of the data, and it makes it possible to specify the expected sizes of classi"ers. The union of local seperators of labeled points of the same class de"nes a cover for these points, which is called a separator. In the case of minimum}distance local separators, the domain covered by a separator corresponding to a class consists of points of Rn that would be assigned to the same class by the nearest-neighbor method. Moreover, the seperator is exclusive: it does not cover any of the points of Rn which are closer to any of the points of the competing class. A new point will be assigned to a class if it falls under the corresponding separator. This is a crude way of performing nearest-neighbor classi"cation. It allows us, however, to "nd reduced consistent subsets, hence, reduced classi"ers. A consistent reduction of the nearest-neighbor classi"er is proposed and, employing the local clustering degree of the data, bounds on its design complexity and on the expected classi"er size are derived. The latter is also shown to bound the expected size of Hart's condensed nearest-neighbor classi"er (1996). The existence of reduced consistant versions of the nearest-neighbor, which are likely to produce higher error rates, supports an objection raised by Webb (1996) against a previously proposed utility of Occam's razor in classi"cation. An observation that the nearest-neighbor method de"nes

184


a multi}linear separation surface between two classes suggests direct consistent multi}linear separation. The performances of the proposed algorithms in predicting stock behaviour is compared to that of the nearestneighbour method, providing yet further experimental evidence against the utility of Occam's razor. Acknowledgements The author acknowledges Dr. Amir Atiya of Cairo University for providing the stock data used in the examples and for valuable discussions of the corresponding results. References [1] K. Fukunaga, Introduction to Statistical Pattern Recognition, Academic Press, San Diego, 1990. [2] F. Rosenblatt, The perceptron: a probabilistic model for information storage and organization in the brain, Psychol. Rev. 65 (1958) 386}408. [3] G. Cybenko, Approximation by superposition of sigmoidal functions, Math. Control Signals Systems 2 (1989) 303}314. [4] Y. Baram, Classi"cation by balanced binary representation, Neurocomputing 10 (1996) 347}357. [5] T.M. Cover, Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition, IEEE Trans. Electron. Comput. EC-4 (1965) 326}334.

[6] V.N. Vapnik, A. Ya. Chervonenkis, On the uniform convergence of relative frequencies of events to their probabilities, Theory Probab. Appl. No. 2 (1971) 264}280 ("rst published in Russian, May 1969). [7] M.R. Anderberg, Cluster Analysis for Applications, Academic Press, New York, 1973. [8] B.S. Everitt, Cluster Analysis, third ed., Edward Arnold, London, 1993. [9] T.M. Cover, P.E. Hart, Nearest-neighbor pattern classi"cation, IEEE Trans. Inform. Theory, IT-13 (1) (1967) 21}27. [10] P.E. Hart, The condensed nearest-neighbor rule, IEEE Trans. Inform. Theory IT-14 (3) (1968) 515}516. [11] D.T. Lee, F.P. Preparata, Computational geometry } a survey, IEEE Trans. Computers, C-33 (12) (1984) 1072}1101. [12] J.H. Conway, N.J.A. Sloane, Sphere Packings, Lattices and Groups, Springer, New York, 1988. [13] R.E. Bonner, On some clustering techniques, IBM J. Res. Dev. 8 (1964) 22}32. [14] R.M. Cormack, A review of classi"cation, J. Roy. Statist. Soc. (134) (1971) 321}367. [15] A.D. Gordon, Classi"cation, Chapman & Hall, London, 1980. [16] A.K. Jain, R.C. Dubes, Algorithms for Clustering Data, Prentice Hall, Englewood Cli!, NJ, 1988. [17] I. Ya. Akimova, Application of Voronoi diagrams in combinatorial problems (A Survey), Tekh. Kibern. 22 (2) (1984) 102}109; Eng. Cybernet. 22 (4) (1984) 6}12. [18] E.B. Baum, On the capabilities of multilayer perceptrons, J. Complexity 4 (1988) 193}215. [19] University of California at Irvine: Machine Learning Data Bases (www.ics.uci.edu/AI/ML/Machine-Learning.html).

About the Author*YORAM BARAM recieved the B.Sc. degree in aeronautical engineering from the Technion-Israel Institute of Technology, Haifa, the M.Sc. degree in aeronautics and astronautics, and the Ph.D. degree in electrical engineering and computer science, both from the Massachusetts Institute of Technology, Cambridge, in 1972, 1974, and 1976, respectively. In 1974}1975 he was with the Charles Stark Draper Laboratory, Cambridge, MA. In 1977}1978 he was with the Analytic Sciences Corporation, Reading, MA. In 1978}1983 he was a faculty member at the Department of Electronic Systems, School of Engineering, Tel-Aviv University, and a consultant to the Israel Aircraft Industry. Since 1983 he has been with the Technion-Israel Institute of Technology, where he is an Associate Professor of Computer Science. In 1986}1988 he was a Senior Research Associate of the National Research Council at the NASA-Ames Research Center, Mo!ett Field, CA, where he has also spent the following summers. His current research interests are in pattern recognition and neural networks.


Adaptive linear dimensionality reduction for classi"cation Rohit Lotlikar, Ravi Kothari* Artixcial Neural Systems Laboratory, Department of Electrical and Computer Engineering and Computer Science, University of Cincinnati, P.O. Box 210030, Cincinnati, OH 45221-0030, USA Received 7 August 1998; accepted 3 February 1999

Abstract Dimensionality reduction is the representation of high-dimensional patterns in a low-dimensional subspace based on a transformation which optimizes a speci"ed criterion in the subspace. For pattern classi"cation, the ideal criteria is the minimum achievable classi"cation error (the Bayes error). Under strict assumptions of the pattern distribution, the Bayes error can be analytically expressed. We use this as a starting point to develop an adaptive algorithm that computes a linear transformation based on the minimization of a cost function that approximates the Bayes error in the subspace. Using kernel estimators we then relax the assumptions and extend the algorithm to more general pattern distributions. Our simulations with three synthetic and one real-data set indicate that the proposed algorithm substantially outperforms Fisher's Linear Discriminant. ( 1999 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Dimensionality reduction; Classi"cation; Adaptive algorithms

1. Introduction Input variables of high-dimensional patterns are often correlated such that the intrinsic dimensionality of these patterns is signi"cantly lower than the input dimensionality. Reducing the dimension of the input patterns removes redundant information from the patterns, and allows for more reliable classi"cation in the subspace with limited patterns. For some classi"ers, especially those which are sensitive to the diluting e!ect of extraneous information (e.g. nearest-neighbor classi"ers), there is often an improvement in classi"cation accuracy. Consequently, classi"cation of high-dimensional patterns is often preceded by mapping these patterns to a lower dimension subspace. The transformation from high-dimensional space to a lower dimensional subspace is constructed to optimize a speci"ed criteria in the subspace.

* Corresponding author. Tel.: #513-556-4766; fax: #513556-7326. E-mail address: [email protected] (R. Kothari)

Broadly, the majority of the techniques proposed for pattern classi"cation can be categorized as supervised or unsupervised depending on whether or not the class label is used in arriving at the subspace. Other categorizations such as linear and non-linear, or parametric and nonparametric are also possible. Probably, the most well-known example of (linear) unsupervised dimensionality reduction is based on principal component analysis (PCA) [1]. PCA provides the optimum representation (in a mean-squared error sense) for a given number of features, though it may lead to identifying combinations which are entirely unsuitable from the perspective of classi"cation. Although unsupervised techniques do not use a class label, they may impose other constraints to "nd the lower dimension subspace. For example, (non-linear) Sammon's mapping [2], self-organizing feature maps (SOFM) [3], and curvilinear component analysis (CCA) [4] attempt to preserve the local structure by imposing a greater emphasis on the preservation of shorter distances over longer distances. Supervised dimensionality reduction techniques on the other hand take advantage of the class label and typically can project the data such that it is more amenable to


186

R. Lotlikar, R. Kothari / Pattern Recognition 33 (2000) 185}194

classi"cation. Discriminant analysis, for example, uses the concept of a within-class scatter matrix S and a bew tween-class scatter matrix S to maximize a separation b criterion, such as J"tr(S~1S ). (1) w b The advantage of linear discriminant analysis is that it is non-recursive, though as given above it is not applicable for multi-modal distributions. It is worth noting that the criterion in Eq. (1) is not directly related to classi"cation accuracy. The above-mentioned techniques along with some neural network methods for dimensionality reduction are discussed and compared by Mao and Jain [5]. For pattern classi"cation, the ideal criteria is the minimum achievable classi"cation error or the Bayes error. The increase in Bayes error is a measure of the loss of relevant information when dimensionality is reduced. When reducing dimensionality, the Bayes error rate in the reduced-dimension space is therefore an ideal measure to assess the classi"ability of the projected data. However, Bayes error cannot be easily expressed analytically for arbitrary pattern distributions and hence there have been few studies that use Bayes error as the criteria. Buturovic` [6] uses a k-NN method to obtain an estimate of the Bayes error, computed on the training data set projected to a low-dimensional subspace using conventional methods of dimensionality reduction [6]. The simplex method is then used as the optimization algorithm. An optimal value of k is not easy to "nd and the estimate of the Bayes error is directly dependent on k. The authors suggest estimating it based on standard methods of error estimation such as leave-one-out and re-substitution methods. However, the approach is computationally expensive. On the other hand, the reverse procedure of "rst selecting a classi"er and then using the decision boundaries produced by the classi"er to "nd suitable subspaces has also been considered [7]. The fundamental drawback of this technique is that the quirks of the selected classi"er precludes optimal extraction of features. The approach we adopt in this paper is to derive an adaptive algorithm for dimensionality reduction based on some assumptions about the pattern distribution. Under these assumptions, the Bayes error is analytically obtained allowing for an algorithm which provides nearoptimal performance when the assumed pattern distribution matches the true pattern distribution. We present this algorithm in Section 2. In Section 3, we extend this approach using kernel estimators for the case when the pattern distribution does not satisfy the assumed pattern distribution. We present results using three synthetic and one real-world data sets in Section 4. We "nd that the basic algorithm performs nearly as well as the algorithm extended to deal with arbitrary pattern distributions if a whitening transform is used to preprocess the data. We present our conclusions in Section 5.

2. Reducing dimensionality We begin the development of the algorithm by considering multi-category exemplars in an n-dimensional real-space Rn. Two assumptions are made regarding the distribution of samples: (A1) Each class has a normal distribution characterized by a Gaussian density function and a covariance matrix +"p2I (where I is the identity matrix). The covariance matrix is the same for all classes but each class may have a di!erent mean. (A2) The classes have equal a priori probabilities. These assumptions are unlikely to hold for real-world problems, however they are not overly restrictive. We use it to develop the basic algorithm and establish it on a "rm theoretical foundation. It simpli"es our analysis and also allows us to build a classi"er with an error equal to Bayes error in the case when the pattern distribution satis"es the assumed distribution. Later in the paper we relax these assumptions. Our objective is to "nd among all m-dimensional linear subspaces of Rn, where m has been "xed a priori and (m)n), a subspace S-Rn in which Bayes error is minimized. We also wish to "nd a corresponding transformation ¹ : RnPS(m)n). Since Bayes error depends only upon the subspace, ¹ can be any linear transformation with range space S. We assume without loss of generality, that ¹ is orthonormal and is parameterized by an orthonormal matrix = . The columns of = will form an nCm orthonormal basis for S. Under such an orthonormal transformation, a Gaussian distribution with covariance matrix +"p2I remains Gaussian and its covariance (nCn) matrix in the subspace is +K "p2I .1 (mCm) When there are only two classes, the Bayesian boundary is a hyperplane and is the orthogonal bisector of the line segment joining the two class centers. The probability that samples from class 1 are incorrectly a$liated to class 2 can be expressed in terms of the marginal distribution p (t) of class 1 along the line joining the two m centers

P

e"

=

p (t) dt, (2) m dK @2 where dK is the distance between the class centers in the output subspace. This distance depends on the distance between the centers in the input space and the transformation = (we will make the relationship explicit shortly). The marginal distribution is a Gaussian with

1 As done here, for each input space variable, the corresponding output space variable is denoted by a `hata above the variable.


variance p2 so that

P

1 = e" exp (!t2/2p2) dt. J2pp dK @2

(3)

When the two classes do not have identical covariance matrices, the Bayesian boundary is not a hyperplane and the overlap between the pdfs of the two classes cannot be expressed in terms of a marginal distribution. When we have c classes MC : k"12cN, the Bayesian k boundaries form a Voronoi tessellation of the output space. In this case the probability of misclassi"cation for class C is the probability mass of class C that lies k k outside the Voronoi polygon it is associated with. This quantity cannot be expressed in a simple analytic form for c'2. Instead, as an approximation, we consider each pair of classes and minimize the sum total of the pairwise errors e . ij Let k(i) be the mean of class i in the input space. Let k( (i) denote its mean in the output space. The two are related through k( (i)"=Tk(i). Let dK "DDk( (i)!k( (j)DD deij note the Euclidean distance between the means of class i and class j in the subspace. The sum total of pairwise errors forms our objective function and is given by

P

= c c 1 exp (!t2/2p2) dt. J"2 + + i/1 j/i`1 J2pp dK ij@2

(4)

The objective function of Eq. (4) is highly non-linear, therefore, a closed form solution for = that minimizes it cannot be obtained. So we resort to a gradient descent scheme for minimizing J. Di!erentiating Eq. (4) with respect to = we obtain LJ 1 c c LdK "! + + exp (!dK 2 /8p2) ij . ij L= L= J2pp i/1 j/i`1

(5)

Denote v(ij),k(i)!k(j), so that dK "DD=Tv(ij)DD, ij whereby

(6)

LdK 1 ij " v(ij)v(ij)T=. L= 2dK ij Combining Eqs. (5) and (7) we obtain

(7)

LJ 1 "! L= 2J2pp

C

D

c c 1 ] + + exp (!dK 2 /8p2)v(ij)v(ij)T = ij dK ij i/1 j/i`1 and the new = is given by LJ = "= !g , new old L=

(8)

(9)

187

where, g is the step size. Before repeating this step to reduce J further, we note that there is no constraint to ensure that the updates will result in an orthonormal transformation matrix =. However, it is important that each iteration be started o! with an orthonormal =, because our assumption that the class-speci"c covariance matrices remain "xed (+K "p2I) is dependent upon it. This may be achieved by the following procedure. After each update, we "nd an orthonormal basis for the subspace spanned by the columns of = and construct a new orthonormal = by using those basis vectors for its columns. Since selection of an orthonormal basis does not change the subspace, the procedure leaves the cost J una!ected. We use this = to start of the next iteration. The adaptive subspace theorem [3] ensures that the subspace spanned by the columns of = will always converge, provided that g is su$ciently small. We could also add penalty terms to the cost function that penalizes J when = deviates from orthonormality, i.e. m m m JI "J#j + + =(i)T=(j)#j + (DD=(i)DD!1)2. (10) 1 2 i/1 jEi i/1 Here =(i) is the ith column of =. j controls the empha1 sis on orthogonality and j controls the emphasis on 2 normality of the columns of =. These penalty terms are at best soft constrains and do not guarantee that orthonormality is achieved. Therefore, the orthonormalization procedure is always required. In our simulations we have not used such a term because of the added complications in appropriately determining j and j . 1 2 The "nal update rule for = therefore is

C

D

c c 1 = "= #g + + exp (!dK 2 /8p2)v(ij)v(ij)T = , new old ij old dK i/1 j/1 ij jEi (11) where the constant factor 1/2J2np is absorbed into g. Each update of = using Eq. (11) rotates the subspace so that dK , the length of the projection of v(ij) on the ij subspace is increased unless v o=. It may also be noted ij that it is possible to obtain at most (c!1) linear features with this technique, because v(ij)v(ij)T has a rank of at most (c!1). The procedure for computing the optimal transformation = : RnPRm, (m)n) may be summed up in the nCm following steps. The available information includes the class centers k(i) : i"12c, and the value of p2. 1. Generate an n]m matrix = with random entries. 2. Repeat the following sequence until J no longer decreases (a) Select an orthonormal basis for the subspace spanned by the columns of =. This may be done by using the Gram}Schmidt orthonormalization procedure.

188


(b) Project the class centers in the input space (k(i)) onto the output space. The class centers in the output space are given by k( (i)"=Tk(i). (c) Compute the cost J using Eq. (4). (d) Back in the input space "nd the vector di!erence vij,k(i)!k(j), of the two means. (e) Use Eq. (11) update =. As the algorithm proceeds, the cost J typically decreases rapidly at "rst and then slowly as J reaches it steady-state value. The decrease in cost is monotonic with occasional small upward jumps that are a consequence of "nite step size. The algorithm is terminated when the cost J reaches a steady-state value. Getting stuck in local minima is a standard weakness of gradient descent algorithms, however, our experiments indicate that the algorithm is fairly robust to local minima. Nevertheless, it is necessary to run the algorithm a few times with di!erent randomly chosen initial conditions (i.e. starting =) and select the "nal = from the run that gave the lowest J. Also, one can apply various heuristics that have been traditionally applied to gradient descent algorithms to improve their convergence properties. Eq. (11) thus provides an adaptive rule that ensures that when the pattern distribution matches the assumed distribution, a near-optimal subspace representation is achieved. When the assumed and true pattern distribution di!er, the representation is less than optimal and we present a simple extension in the next Section.

3. Extension to other distributions When the data are such that assumptions (A1) and (A2) are not satis"ed, we model the density function using a kernel estimator as a superposition of Gaussians. A Gaussian with a covariance matrix p2I, where p is the kernel width, is placed centered at each training point of that class. The modi"ed cost function is then,

P

q q 1 = J" + + (1!d ) exp (!t2/2p2) dt, (12) ij i/1 j/i`1 J2pp dK ij@2 where we have q training points x(i), i"12q from c(c(q) classes, dK is the distance between two samples ij with indices i and j, and d is 1 when patterns i and j are ij of the same class. The modi"ed update equation is then given by LJ 1 "! L= 2J2pp

C

q c 1 + + (1!d ) ij dK ij i/1 j/i`1

D

]exp (!dK 2 /8p2)v(ij)v(ij)T =, ij

(13)

where v(ij) is the corresponding vector di!erence of the two samples in the input space.

The issue here is the choice of the width p of the kernel estimator. A kernel width that is optimal in the sense of having the minimum mean-square error in its estimate of the density, may not be optimal from the view of classi"cation performance. This is because most of the misclassi"cation typically occurs at the tails of the distributions where sample densities are low. Also, the surface of the cost function will be riddled with local minima, particularly for small p. We therefore suggest that = be "rst computed using only the class centers, and this procedure be subsequently used to `"ne-tunea = using a small learning rate. p may be chosen by trial and error, the mean within-class intersample distance in regions where misclassi"cation is observed can be used as a guide for the initial choice. When the data satisfy assumptions (A1) and (A2), use of this extension will be most likely in increased classi"cation error. There are two contributing factors, one is the error in the estimate of the density function, and the other that the pairwise error becomes increasingly less representative of Bayes error as the number of kernels grows.

4. Simulations The algorithm was applied to four data sets, three of which are synthetic and one of which is a real data set. For the real data set, a k-NN (nearest neighbor) classi"er was used to classify the data projected onto a lower dimension space. The value of k was selected by using leave-one-out cross-validation. To put the performance of the proposed dimensionality reduction technique into perspective, we include the results obtained with Fisher's linear discriminant (FLD) [8]. We also refer to the algorithm of Section 2 as the adaptive algorithm and the extension of the algorithm using kernel estimators as the extended adaptive algorithm. In the simulations, the columns of = were randomly initialized corresponding to a random orientation in the m-dimensional space. Typically, the minimum value of J is reached in a few hundred iterations. The initial learning rate was chosen g"0.05. While we did not, g maybe adapted to obtain faster convergence. We performed "ve repeat trials with di!erent initial conditions (for =) for each case. The `besta =, among all the "ve runs is retained as the end result. Here `besta implies the = that resulted in the lowest J. For Simulation 2 (which used the image segmentation data set [9]) we also report the results obtained in each of the "ve runs to provide an indication of the variability introduced through di!erent initial conditions. 4.1. Simulation 1 This simulation is intended to serve as an illustrative example. The data consist of six classes in


three-dimensional space. Each class has a Gaussian distribution with a covariance matrix that is a 0.04 I, where I is the identity matrix. There are 50 samples per class. The six class centers are at c1"[!3 0 0]T, c2" [3 0 0]T, c3"[0 !2 0]T, c4"[0 2 0]T, c5"[0 0 !1]T, c6"[0 0 1]T. The mean of the data is the origin. Quite clearly, FLD would produce the three projection directions of [1 0 0]T, [0 1 0]T, and [0 0 1]T in decreasing order of importance. Now if we wished to reduce the data to two dimensions, class 5 and class 6 would be projected to the same region in the two-dimensional space. This is what happens as shown in Fig. 1. In contrast, the projection obtained with the adaptive technique is also shown in Fig. 1 and is superior to that obtained using FLD in that the six classes are well separated.

189

tory [9]. The data contain instances drawn randomly from a database of seven outdoor images. Nineteen continuous-valued features are extracted for each 3]3 region, and the class label is obtained by manual segmentation. Attribute 3 is a constant by de"nition and was deleted from the data set leaving 18 features. The classes are brickface, sky, foliage, cement, window, path, and grass. There are a total of 210 training patterns and 2100 testing patterns with 30 patterns for each of the seven classes in the training data and 300 patterns per class in the test data. The only preprocessing used was to whiten the data set. The whitening transformation, when applied to a data set consisting of c classes MC : i"12cN, coni verts the within-class covariance matrix

4.2. Simulation 2

c S " + + (x(j)!k(i)) (x(j)!k(i))T w i/1 x(j)|Ci

For this simulation, we used the image segmentation data set available from the UCI machine learning reposi-

into an identity matrix. This transformation is necessary because of our assumption that the individual-class

(14)

Fig. 1. The lower dimensional representation for the data set of Simulation 1 as obtained with FLD (left) and the adaptive algorithm (right).

Table 1 Classi"cation accuracy for the image segmentation data set over the "ve runs with the adaptive algorithm. The optimal value of k as determined using cross-validation is identi"ed as k 015

J at convergence Training accuracy Testing accuracy k 015

Trial d 1

Trial d 2

Trial d 3

Trial d 4

Trial d 5

0.0019292 88.9 88.5 5

0.0019297 89.5 88.9 3

0.0019371 88.6 88.8 5

0.0019382 88.5 89.0 5

0.0019222 88.5 88.8 5

190


Fig. 2. The lower dimensional representation for the image segmentation data set as obtained with the adaptive algorithm in the "rst four trials reported in Table 1. Table 2 Classi"cation accuracy for the image segmentation data set with FLD and the adaptive algorithm. m denotes the dimensionality of the subspace m"2

Training accuracy Testing accuracy k 015

m"3

FLD

Adaptive

FLD

Adaptive

77.6 73.7 7

88.5 88.8 5

85.7 86.8 3

91.4 90.4 1

covariance matrices are identity. Unless all the classes had identical-class covariance matrices, the individualclass covariance matrices will not be identity, however the average class conditional covariance matrix will be identity. The algorithm however performs fairly well nevertheless. Table 1 lists the training accuracy, testing accuracy, the "nal value of J, and the optimal value of k (as determined through cross-validation) obtained in each of the "ve runs when the dimensionality was reduced to 2. One may observe that since the data distribution does not satisfy


191

Fig. 3. The lower dimensional representation for the image segmentation data set as obtained with FLD (left) and with the `besta = (Triald 5 of Table 1) obtained with the adaptive algorithm.

Fig. 4. The lower dimensional representation for the modi"ed image segmentation data set. There are a total of seven classes. The lower dimensional representation as obtained with FLD (left) and the adaptive technique (right).

the assumptions made by the adaptive algorithm, the `besta = (in terms of the one that led to the lowest J) does not minimize classi"cation error. However, the variability is small. For this simulation the `besta = corresponds to Trial d 5 in Table 1.

Fig. 2 shows the representations obtained with the "rst four trials. Fig. 3 shows the representation obtained with the "fth trial (corresponding to the `besta =) along with the the representation obtained using FLD.

192


Fig. 5. The original data set (top left); the reduced dimension representation obtained with FLD (top right), the adaptive algorithm proposed (bottom left), and the extended adaptive algorithm (bottom right).

Table 2 compares the classi"cation accuracy obtained using FLD and the adaptive algorithm for subspace dimensionality of 2 and 3. The results for m"2 correspond to Trial d 5 in Table 1. 4.3. Simulation 3 In Simulation 2, the data did not satisfy the assumption that each class had a Gaussian pdf with an identity covariance matrix. In this simulation we take the means

of the seven classes of the data set of Simulation 2 and arti"cially generate samples around each mean so that each class has a Gaussian distribution with a covariance matrix given by 0.04 I. Four-hundred patterns are generated per class. After computing the transformation matrix =, the patterns and their means are projected to the lower dimension subspace. Classi"cation consists of assigning to each pattern the class label of the true mean that is closest to it. Fig. 4 shows the projected patterns for the case m"2 dimensions. It may be observed that there


is considerable overlap in "ve of the seven classes in the reduced dimension representation obtained using FLD. In contrast, the adaptive algorithm allows for clear separation of all the seven classes. In our plots, for the sake of clarity, we have displayed only a subset consisting of 50 points randomly chosen from each class. 4.4. Simulation 4 This simulation considers the modi"cation of the algorithm for arbitrary distributions. Speci"cally, a distribution which does not satisfy the assumptions of equal a priori probabilities and where class covariance matrices are not a scalar multiple of the identity matrix. For visualization purposes, we start with a data set which has three classes in two dimensions (see Fig. 5). Two of the classes have identity covariance matrices and equal a priori probabilities. The third class has a non-identity covariance matrix and a priori probability thrice that of either of the other two classes. On the assumption of equal a priori probabilities and a class covariance matrix that is a multiple of the identity matrix, the = vector (for projection to one-dimension) is the !453 line. The centers of the three classes are uniformly spaced. Histogram of the projection obtained by reducing the dimensionality from 2 to 1 is shown in Fig. 5. With FLD, the projection obtained shows signi"cant overlap while that obtained with the adaptive algorithm has minimal overlap. When the suggested modi"cation for arbitrary distribution (i.e. the extended adaptive algorithm) is used, with p"0.5, the = vector is rotated clockwise by a few degrees, increasing the spacing between the centers and class 1 and class 3 and reducing the spacing between the centers of class 1 and class 2. This resulted in a further decrease of the total overlap.

5. Conclusions In this paper we introduced an approach to "nd a subspace of a priori chosen dimensionality in which Bayes error is the minimum when the patterns have a speci"ed distribution. We also extended the algorithm so as to be applicable to more general pattern distributions. The adaptive algorithm performed quite well even when the pattern distribution did not agree with the assumed distribution. Additional improvement in performance was always obtained with the extended adaptive algorithm though the amount of improvement may vary with the problem. Computationally, both the adaptive algorithm and the extended adaptive algorithm are more expensive than

193

FLD. Both these algorithms are based on a gradientdescent procedure making it di$cult to predict the time to convergence. However, in our simulations, the minimum J was found in less than 500 iterations. In practical terms, the adaptive algorithm took about a minute to converge for the image segmentation data set used in this paper, while the extended adaptive algorithm took about 5 min to converge for the data set of Simulation 4 on a SPARCstation 5. A further insight may be obtained by observing that the weight update equation in the adaptive algorithm is based on c(c!1)/2 summations, where c is the total number of classes. This dependence on the number of classes (as opposed to the number of training points) allows the algorithm to be computationally e$cient. In the extended adaptive algorithm however, the number of kernels we chose were equal to the number of training points resulting in considerably slower performance. For large problems, it may be worthwhile to reduce the number of kernels used in the kernel estimator. The Bayes error as an optimization criteria, as used in this paper, is superior to criteria based on "rst- and second-order statistics of the data which are not directly related to classi"cation accuracy.

References [1] I.T. Jolli!e, Principal Component Analysis, Springer, New York, 1986. [2] J.W. Sammon, A non-linear mapping algorithm for data structure analysis, IEEE Trans. Comput. 19 (1969) 401}409. [3] T. Kohonen, Self Organizing Maps, Springer, Berlin, 1997. [4] P. Demartines, J. Herault, Curvilinear component analysis: a self-organizing neural network for nonlinear mapping of data sets, IEEE Trans. Neural Networks 8 (1997) 1197}1206. [5] J. Mao, A.K. Jain, Arti"cial neural networks for feature extraction and multivariate data projection, IEEE Trans. Neural Networks 7 (1995) 296}317. [6] L.J. Buturovic`, Towards Bayes-optimal linear dimension reduction, IEEE Trans. Pattern Anal. Mach. Intell. 16 (1994) 420}424. [7] C. Lee, D. Landgrebe, Feature extraction based on decision boundaries, IEEE Trans. Pattern Anal. Mach. Intell. 15 (1993) 388}400. [8] K. Fukunaga, Introduction to Statistical Pattern Recognition, Academic Press, New York, 1990. [9] C.J. Merz, P.M. Murphy, UCI Repository of Machine Learning Databases, [http://www.ics.uci.edu/˜mlearn/ MLRepository.html]. University of California, Department of Information and Computer Science, Irvine, CA, 1996.

194


About the Author*ROHIT LOTLIKAR received the B.Tech (1993) and M.Tech (1995) degrees in Electrical Engineering from the Indian Institute of Technology, Bombay, India. In 1995 he was a senior research assistant at the Center for Studies in Resources Engineering, Bombay, India. Since 1995 he has been a Ph.D. student at the University of Cincinnati. His current research interests include pattern recognition, neural networks, computer vision, and image analysis. His current research focuses on dimensionality reduction techniques for pattern classi"cation. About the Author*RAVI KOTHARI received his B.E. degree (with Distinction) in 1986 from Birla Institute of Technology (India), his M.S. from Louisiana State University in 1988, and his Ph.D. from West Virginia University in 1991, all in Electrical Engineering. He is currently an Associate Professor in the Department of Electrical and Computer Engineering and Computer Science at the University of Cincinnati and Director of the Arti"cial Neural Systems Laboratory there. His research interests include arti"cial neural networks, pattern recognition, and image analysis. He received the Eta Kappa Nu Outstanding Professor of the Year Award in 1995, and the William E. Restemeyer Teaching Excellence Award in 1994 from the Department of Electrical and Computer Engineering and Computer Science at the University of Cincinnati. Dr. Kothari serves on the Editorial Board of the Pattern Analysis and Applications Journal (Springer-Verlag), and is a member of the IEEE, the International Neural Network Society, and the Eta Kappa Nu and Phi Kappa Phi honor societies.


Skew detection and reconstruction based on maximization of variance of transition-counts Yi-Kai Chen!, Jhing-Fa Wang!,",* !Institute of Information Engineering, National Cheng Kung University Tainan, Taiwan, Republic of China "Department of Electrical Engineering, National Cheng Kung University Tainan, Taiwan, Republic of China Received 31 March 1997; received in revised form 30 April 1998; accepted 12 January 1999

Abstract The input document images with skew can be a serious problem in the optical character recognition system. A robust method is proposed in this paper for skew detection and reconstruction in document images which can contain less text areas, high noises, tables, "gures, #ow-chart, and photos. The basic idea of our approach is the maximization of variance of transition counts for the skew detection and text-orientation determination. Once the skew angle is determined, the scanning-line model is applied to reconstruct the skew images. 103 documents with great varieties have been tested and successfully processed. The average detection time of A4 size image is 4.86 s and the reconstruction time is 5.52 s. The proposed approach is also compared with the existing algorithms published in the literature and our method gets some signi"cant improvements in skew detection and reconstruction. ( 1999 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Skew detection; Skew reconstruction; Scanning line; Transition-counts; Transition-counts variance; SNR; Scanning-line model

1. Introduction Document image processing using computers has become more and more useful in di!erent kinds of applications. When the document image is scanned, it may be skew because of some reasons. The skew image will cause serious problems in document analysis, such as incorrectness of character segmentation or recognition. To solve these problems, skew detection and reconstruction should be performed before document analysis. Hough transform is a common technique to detect the skew angle of a document image [2}5]. It is well known that Hough transform is time consuming. Besides, methods based on Hough transform are sensitive to non-text areas and noises. If the input document contains too much

* Corresponding author. Tel.: #886-6-2746867; fax: #8866-2746867 E-mail address: [email protected] (J-F. Wang)

non-text areas or noises, the results will be unexpected while using these methods. Some of the other methods are based on the features of local text areas [6}8]. Baird [1] proposed an algorithm to determine the skew angle using an energy function on sets of projection counts of character locations. These methods are also sensitive to non-text areas and may fail if the local region of text areas cannot be found. Another method uses connected component analysis and line "tting to detect skew angle [9,10]. These methods are also time consuming and may contain large errors. A few of the other methods are based on mathematical theory [11,12]. They are too complex and ine$cient. In our paper, a new approach based on maximization of variance of transition counts is used to detect the skew angle and reconstruct the skew image e$ciently. Using our method, the document image can contain less text areas, high noises, tables, "gures, #ow-charts, and photos. In addition, the page orientation (column-wise or row-wise) can also be detected.

0031-3203/99/$20.00 ( 1999 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 1 - 3 2 0 3 ( 9 9 ) 0 0 0 4 5 - X

196

Y-K. Chen, J-F. Wang / Pattern Recognition 33 (2000) 195}208

The approach in this paper can be mainly divided into two parts: 1. Maximization of variance of transition counts for skew detection and page orientation determination. 2. Skew reconstruction using scanning-line model. The proposed algorithm is described in detail as follows.

Fig. 2. The example of computing transition-counts.

2. Maximization of variance of transition-counts for skew detection and page orientation determination

Then the transition-counts variance (TCV) at each angle from !453 to #453 can be computed by using the following equations. (Subscript h and v mean horizontal and vertical orientations, respectively.)

2.1. Scanning-line model Any digital straight line of slope h"tan~1(A/B) can be expressed [13] approximately by the chain code of the form PP2QPP2Q"(PmQ)n where

+W~1(¹C [h][ j]!¹M [h])2 v v , ¹C< [h]" j/0 v =

AB

A , B

P"NIN¹

G

Q"

P#1, if A!B]P'0,

P!1, if A!B]P(0,

A

m"NIN¹

+H~1(¹C [h][i]!¹M [h])2 h h ¹C< [h]" i/0 , h H

B

B !1, DA!B]PD

where +H~1¹C [h][i] h ¹M [h]" i/0 , h H +W~1¹C [h][ j] v ¹M [h]" j/0 , v =

n"DA!B]PD and `NINT a stands for `Nearest Integera. An example is shown in Fig. 1. It is a straight line of 303 skew. If we let A"256 and B"148, the digital line of 303 skew can be expressed in the chain code (231)40 by the modeling method described above. 2.2. Maximization of variance of transition-counts for skew detection The transition counts (TC) on each scanning line is de"ned as the number of transition phenomenon (pixel from black to white or white to black) on the scanning line and an example is shown in Fig. 2.

Fig. 1. The example of a digitized line with 303 skew.

where ¹C< [h] is the horizontal transition-counts h variance at h3; ¹C< [h] is the vertical transitionv counts variance at h3; ¹C [h][i] is the horzontal h transition-counts of the ith row at h3; ¹C [h][ j] is v the vertical transition-counts of the jth column at h3; ¹M [h] is the horizontal transition-counts mean at h3; h ¹M [h] is the vertical transition-counts mean at h3; H v is the image height; = is the image width and h"!453ǅ. We "rst choose the strip of 256 pixels in horizontal direction at the middle of the document as well as a strip of the same width in the vertical direction. Then we compute the total counts of transition in the strip. If the total counts of transition exceed the threshold from experimental results, then the strips are regarded as containing enough text and are used to compute transitioncounts variance for skew detection. Otherwise, the strips are shifted in horizontal and vertical direction until we con"rm that the strips contain enough text. After the horizontal and vertical ¹C< at each angle are computed, the maximum ¹C< in horizontal and vertical direction are labeled as ¹C< } and ¹C< } , .!9 v .!9 h respectively. If ¹C< } is larger than ¹C< } , we .!9 h .!9 v regard that the text is row-wise and the skew angle is the angle with ¹C< } . Otherwise, the text is column-wise .!9 h and the skew angle is the angle with ¹C< } . In this .!9 v


197

Fig. 4. The transition-counts variance of each angle in horizontal direction.

Fig. 5. The transition-counts variance of each angle in vertical direction. Table 1 The experimental results of skew detection to documents with di!erent SNR Image with SNR(1

Image with SNR'"1

Success 2

Success 14

Failure 5

Fig. 3. The test document image with #193 skew.

Fig. 6. (a) A skew document with SNR"5.7; (b) the image reconstructed by our approach.

Failure 0

198





way, the skew angle and page orientation can be detected at the same time. The basic idea of using transition-counts variance to detect the skew angle is the fact that peaks of the transition-counts histogram at the skew angle appear

199

periodically and the transition-count histograms at the other angles look #at. In other words, the transitioncounts variance will be the maximum at the skew angle and so we can apply this concept to detect the skew angle and the page orientation.

Fig. 9. (a) Caption on p. 201.

200


Fig. 9. (b) Caption opposite.

For example, the transition-counts variance of the image in Fig. 3 is computed and plotted in Fig. 4 (horizontal) and Fig. 5 (vertical). It can be seen that ¹C< } is 16.2 at the angle #193 .!9 h while ¹C< } is 5.9 at the angle !193. So the skew .!9 v angle of the image in Fig. 3 is #193 and the page orientation is row-wise. Here we do experiments on 21 noisy documents by knowing the SNR. The signal-to-noise ratio (SNR) in our paper is de"ned as follows:

¹ransition count of original image SNR" . ¹ransition count of noises

Fig. 6 is a document with SNR"5.7 and Fig. 7 is another with SNR"1.2. Table 1 lists our experimental results. With the results in Table 1, we justify the noise robustness of our algorithm. Since most noisy signals are random, the in#uence of noise to transition-count variance at each angle is almost the same. So, the maximum transition-count variance is invariant at the skew angle. According to our experiment, if the SNR of the document is greater than 1, our algorithm can still work properly. If the SNR is less than 1, some documents are successfully detected and some fail. Fig. 8 is a document with SNR"0.8 and our algorithm still works well.


201

Fig. 9. (a) A row-wise document with !133 skew; (b) the document reconstructed by our approach; (c) the document reconstructed by PhotoStyler.

3. Skew reconstruction using scanning-line model After the skew angle is detected by the method described above, the scanning-line model at the skew angle is used to calculate the vertical and horizontal o!sets for skew reconstruction. Using the chain code of the scanning-line at skew angle with the parameters P, Q, m and n de"ned in Section 2, the vertical o!set >}shift of the jth column can be computed j as follows:

CD CD CD

r r >}shift "b](m#1)# , if (m, j P P >}shift "b](m#1)#m, if j

r *m, P

where b is the quotient of j!1/m]P#Q; r the ( j!1) mod(m]P#Q) and DxD stands for the greatest integer not greater than x. In the same manner, the horizontal o!set X}shift of i the ith row can be computed as follows:

CD CD

X}shift "b](m#1)# j

r r , if (m, P P

X}shift "b](m#1)#m, j

if

CD

r *m, P

where b is the quotient of i!1/m]P#Q and r the (i!1)mod(m]P#Q).

202


Fig. 10. (a) Caption on p. 204.


Fig. 10. (b) Caption overleaf.

203

204


Fig. 10. (a) A column-wise document with #73 skew; (b) the document reconstructed by our approach; (c) the document reconstructed by PhotoStyler.


205

Table 2 Some classi"cations of our test samples

Row-wise mode Column-wise mode Document with noises Document with large skew angle Document with less text area Tabular form

Chinese

English

Chinese/English mixed

Japanese

Total

31 25 17 8 8 9

32 0 14 7 10 0

5 0 1 1 0 2

6 4 0 0 2 0

74 29 32 16 20 11

Table 3 Comparison of performance Paper no.

Platform

Range of detected angle

Skew detection Skew recontime struction time

Able to detect less-text areas, high-noise, tabular, and hand-written documents

Text orientation determination

Algorithm complexity analysis

[2] [3]

Sun 4-280 DELL486D/50 PC MC68020 486DX2-66 Sun SPARC-2 HP 9000/720 Sun SPARC-10 Pentium 133

N.A. #15}!153

67 s 3.8 s

N.A. N.A.

Fair Fair

Yes Yes

O(N3) O(N3)

#20}!203 #45}!453 #30}!303 #15}!153 #5}!53 #11}!113

14.5 s 3.5 s 1.4 s 0.32 s 10 s Almost real time 4.86 s

11.9 s N.A. N.A. N.A. N.A. Almost real time 5.52 s

Fair Fair Fair Fair Fair Fair

No No No No No No

O(N3) O(N3) O(N2) N.A. N.A. N.A.

Good

Yes

O(N2)

[4] [5] [7] [8] [11] [14] Our approach

Pentium 133

Horizontal #45}!453 and vertical #45}!453

N.A."not available.

After skew reconstruction, the reconstructed image becomes larger than the original image. The new width and height of the reconstructed image are calculated as follows: new}width"old}width#X}shift } old height new}height"old}height#>}shift } old width where the parameters X}shift } and >}shift } old height old width are the horizontal and vertical o!sets at the positions of the old}height and old}width, respectively. Then, the pixel at the ith row and the jth column on the skew document with positive angle is mapped to the (i#>}shift )th row and the ( j!X}shift # j i X}shift } )th column in reconstructed image. In conold height )th trast, it is mapped to the (i!>}shift #>}shift } j old width row and the ( j!X}shift #X}shift } )th column for i old height the document with negative angle.

Here we compare our approach with that in PhotoStyler, which is one of the famous image-processing software. Examples in Fig. 9b and Fig. 10b are reconstructed by our approach while Fig. 9c and Fig. 10c are processed by PhotoStyler. The average processing time of our approach for an A4 size documents is 5.52 s and the average processing time of PhotoStyler is 67 s. It can be seen that we save much processing time without reducing the quality of reconstructed image.

4. Testing and results The algorithm described above was implemented on a PC (Pentium 133). The program was tested by 103 document images scanned from magazines, newspapers, manual documents, and so on. Some classi"cations of these documents are shown in Table 2. All of the document

206


Fig. 11. (a) Caption opposite.


207

Fig. 11. (a) A hand-written document with #73 skew; (b) the document reconstructed by our approach.

images were scanned in 300 dpi binary format. The range of skew angle is between $453 in both horizontal and vertical orientation. Some of the test samples are shown in Figs. 6}11. The average processing time of skew detection is 4.86 and reconstruction is 5.52 s. Our approach has the advantages of dealing with large skew angle and is successful to handle the documents with less text areas, high noises, tabular form and hand-written characters. Table 3 is the performance comparison between our method and others. In Table 3, papers 2}4 are based on

Hough transform. The non-text areas and noises in the document may result in errors or failure. Paper 5 can also process large skew angle with very fast skew detection time. However, their method requires left-aligned documents. Our approach does not have this kind of limitation. The algorithm of papers 7 and 8 are mainly based on analysis in text area. The non-text area and noises in documents may cause errors or failure. Paper 11 is based on mathematical theory. It is too time consuming. No.14 is the commercial software. Its processing time is almost

208


real time but 53 documents failed in our 103 test samples. Only 5 documents failed by using our approach. In Table 3, we list the complexity of each algorithm and we compare our algorithm with some others. In summary, the overall performance of our approach is superior.

[3]

[4]

5. Conclusion The algorithms of skew and page-orientation detection are based on the maximization of variance of transitioncounts. The skew reconstruction approach comes from scanning-line modeling. The algorithms in this paper are quite reliable, robust and fast. It is insensitive to noises and non-text areas in the skew document image. Experiments and tests are done with documents of wide variety and a pretty good result has been obtained. This result has shown that our approach has achieved a good performance and e$ciency in skew detection, page orientation determination and skew reconstruction. In the future, we will try to apply these concepts to the color-document images. References [1] H.S. Baird, The skew angle of printed documents, Proceedings of SPIE Symposium on Hybrid Imaging Systems, Rochester, N.Y., 1987, pp. 21}24. [2] S.C. Hinds, J.L. Fisher, D.P. D'Amato, A document skew detection method using run-length encoding and the

[5] [6] [7] [8] [9] [10]

[11]

[12]

[13] [14]

Hough transform, Proceedings of the 10th International Conference On Pattern Recognition, 1990, pp. 464}468. D.S. Le, G.R. Thoma, H. Wechsler, Automated page orientaion and skew angle detection for binary document images, Pattern Recognition, 1994, pp. 1325}1344. Y. Nakano, Y. Shima, H. Fujisawa, J. Higashino, M. Fujiwara, An algorithm for the skew normalization of document image, Proceedings 10th International Conference On Pattern Recognition, vol. 2, 1986, pp. 8}13 . H.F. Jiang, C.C. Han, K.C. Fan, A Fast approach to detect and correct the skew document, OCRDA, 1996, pp. 67}68. H. Yan, Skew correction of document images using interline cross-correlation, CVGIP, 1993, pp. 538}543. K. Toshiba-cho, S. Ku, Document skew detection based on local region complexity, Proc. IEEE, 1993, pp. 125}132. R. Smith, A simple and e$cient skew detection algorithm via text row accumulation, ICDAR, 1995, pp. 1145}1148. F. Hones, J. Licher, Layout extration of mixed mode document, Machine Vision Appl., 1994, pp. 237}246. C.L. Yu, Y.Y. Tang, C.Y. Suen, Document skew detection based on the fractal and least squares method, ICDAR, 1995, pp. 1149}1152. S. Chen, R.M. Haralick, I.T. Phillips, Automatic text skew estimation in document image, ICDAR, 1995, pp. 1153}1156. H.K. Aghajan, B.H. Khalaj, T.Kailath, Estimation of skew angle in text-image analysis by SLIDE: subspace-based line dection. S.-X. Li, M.H. Loew, Analysis and modeling of digitized straight-line segments, Proc. IEEE, 1988, pp. 294}296. Sequoia Data Corporation, ScanFix Image Optimizer, V2.30 for Windows.

About the Author*JHING-FA WANG received the Ph.D. degree in electrical engineering and computer science from Stevens Institute of Technology, Hoboken, in 1983. He is an IEEE fellow and was elected as the general chairman of the Chinese Image Processing and Pattern Recognition Society in 1993. He was the director of Institute of Information Engineering in National Cheng Kung University from 1990 to 1996. At present he is a professor in the department of Electrical Engineering and Institute of Information Engineering at National Cheng Kung University. He is also the present Chairman of Taiwan Information Software Association and the Chairman of Computer Center of National Cheng Kung University. His current research interests include graph theory, CAD/VLSI, neural nets for Image processing, neural nets for computer speech processing, and optical character recognition.

About the Author*YI-KAI CHEN received the B.S. and M.S. degrees in electrical engineering from National Cheng Kung University in 1994 and in 1996, respectively. His interests include image processing and optical character recognition.


Appearance-based object recognition using optimal feature transformsq Joachim Hornegger*, Heinrich Niemann, Robert Risack Lehrstuhl f u( r Mustererkennung (Informatik 5), Universita( t Erlangen}Nu( rnberg, Martensstr. 3, D-91058 Erlangen, Germany Received 9 December 1997; accepted 7 January 1999

Abstract In this paper we discuss and compare di!erent approaches to appearance-based object recognition and pose estimation. Images are considered as high-dimensional feature vectors which are transformed in various manners: we use di!erent types of non-linear image-to-image transforms composed with linear mappings to reduce the feature dimensions and to beat the curse of dimensionality. The transforms are selected such that special objective functions are optimized and available image data provide some invariance properties. The paper mainly concentrates on the comparison of preprocessing operations combined with di!erent linear projections in the context of appearance-based object recognition. The experimental evaluation provides recognition rates and pose estimation accuracy. ( 1999 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Appearance-based object recognition; Pose estimation; Feature transform; Manifold models; Statistical modeling

1. Introduction Even up to these days, the e$cient and robust recognition and localization of arbitrary 3-D objects in graylevel images is a generally open problem. There exists no uni"ed technique which allows the reliable recognition of arbitrary-shaped objects in cluttered scenes. The available algorithms are mostly restricted to special types of objects. Standard identi"cation and pose estimation techniques use segmentation operations in order to detect geometrical features like corners or lines [1]. The classi"cation itself is based on geometrical relationships between observations and suitable models like geometric representations which use, for instance, wire frame or

* Corresponding author. Tel.: #49-9131-852-7894; fax: #49-9131-303811. E-mail address: [email protected] (J. Hornegger) q This work was funded partially by the Deutsche Forschungsgemeinschaft (DFG) under grant numbers Ho 1791/2-1 and GRK 244/1-96. The authors are solely responsible for the contents.

CAD models [1]. The main problems of these approaches are due to the automatic generation of models using training samples, and the robust detection of geometric features. Recently, appearance-based methods have become more and more popular, and are used to deal with object recognition tasks [2,3]. These techniques consider the appearance of objects in sensor signals instead of the reconstruction of geometrical properties. This overcomes quite a lot of problems related to standard approaches as, for example, the geometric modeling of fairly complex objects and the required feature segmentation. Preliminary comparative studies prove the power and the competitiveness of appearance-based approaches to solve recognition problems [2] and suggest further research and experiments. Now well-known and classical pattern recognition algorithms can be used for computer vision purposes: feature selection methods [4,5], feature transforms [4,6], or even more recent results from statistical learning theory [7]. This paper will consider and compare di!erent transforms of high-dimensional feature vectors for object recognition and pose estimation purposes.


210

J. Hornegger et al. / Pattern Recognition 33 (2000) 209}224

2. Contribution and organization of the paper Appearance-based approaches treat images as feature vectors. If we consider M]N images, the dimension of associated feature vectors is m " : NM. Obviously, these high-dimensional feature vectors will not allow the implementation of e$cient recognition algorithms [8] and the curse of dimensionality will prohibit classi"cation [5]. For that reason, transforms are necessary to reduce the dimensions of features. Commonly used transforms are the principal component analysis [9}11] or in more recent publications the Fisher transform [12]. Variations of feature vectors dependent on di!erent illumination or pose parameters are modeled by interpolating between di!erent feature vectors and considering the resulting manifolds as object models [9]. These models are called eigenfaces or Fisherfaces } dependent on the chosen transform. This work extends the existing appearance-based methods with respect to di!erent linear transforms of feature vectors. The considered linear transformations are based on optimization criteria which are basically known from standard pattern recognition literature [8]. In addition, we also consider various types of non-linear preprocessing operations which eliminate, for instance, noise or dependencies of illumination. The main contribution of this paper is therefore twofold and includes f the comparison of di!erent preprocessing operations, and f the application of various feature transforms for the reduction of dimensions. The experimental evaluation provides an extensive characterization of distinct feature transforms. We summarize several methods for improving the recognition rates and pose estimation accuracy of existing algorithms for 3-D object recognition. The "nal judgement of methods depends on recognition rates and pose estimation errors. The paper is organized as follows: the next section gives a brief overview of related work and discusses parallels and di!erences to the main contributions of this paper. Before we introduce mathematical and technical details, we clarify and specify the general formal framework (Section 4). The restriction of already published approaches to the principal component analysis for the reduction of features' dimensions motivates to consider and to compare experimentally di!erent types of linear projections from high-dimensional image into lowerdimensional feature spaces. Feature transforms and the e$cient solution of optimization problems related to these projections form the main part of Sections 5 and 6. Instead of using gray-level images as features, some nonlinear image transforms, which can be applied within a preprocessing stage, are summarized in Section 5.2. Computational aspects of the involved algorithms are

included in Section 7. The experimental evaluation of introduced concepts is summarized in Section 9: the recognition and pose estimation experiments are evaluated with various combinations of image transforms. The paper ends with a summary, draws some conclusions, and gives several hints to further unsolved research problems concerning appearance-based recognition. Mathematical details, which are less essential for the basic understanding of the proposed techniques, are provided in the appendix.

3. Related work Appearance-based approaches discussed in the literature are mostly restricted to the principal component analysis to map gray-level images to low-dimensional feature vectors and neglect preprocessing operations [2]. Here, the considered feature transforms are incorporated into an optimization-based framework, which allows geometrical interpretations within the feature space. The mathematical tools which are required for a practical implementation are provided by Murase and Lindenbaum [13]. Fields of application are medical image processing, face recognition [14], or 3-D object recognition and pose estimation. The major problems related to appearance-based methods are due to unknown background objects and occlusion. Classi"cation in cluttered scenes is discussed and su$ciently solved in Ref. [15]. The application of appearance-based methods in the presence of occlusion is considered in Ref. [16], whereas the in#uence of varying illumination to eigenfaces is experimentally evaluated in Ref. [17]. The authors of Ref. [17] show that 5-D vectors in the eigenspace are su$cient for modeling di!erent lighting conditions. For that reason, this work does not discuss various methods which work with cluttered background and occlusion, but concentrates on the comparison of di!erent optimization criteria with respect to the computation of linear projections.

4. General framework A digital image f is mathematically represented by a matrix [ f ] , where the range of f is i,j 1xixN,1xjxM i,j determined by the quantization of intensity values. The parameters N and M denote the number of rows and columns. Let us assume, we have K di!erent classes ) , ) ,2, ) of objects. Examples of di!erent objects 1 2 K are shown in Fig. 1. These objects are assigned to the pattern classes ) , ) ,2, ) . The classi"cation pro1 2 5 cedure is thus a discrete mapping which assigns an image, showing one of these objects, to the pattern class the present object corresponds to. If we compute the pose parameters, the position and orientation of the object


211

Fig. 1. Object classes considered in the experiments (Columbia images).

Fig. 2. Structure of feature computations: images and preprocessed sensor data are transformed into feature vectors.

with respect to the world coordinate system are calculated. Usually, there exists no closed-form analytical description of these mappings. Most systems decompose this function into a sequence of } mostly independent } procedures [18]. It is suggesting to consider images [ f ] i,j 1xixN,1xjxM as feature vectors f3Rm, where m"NM. Because of the geometric nature of objects, however, this is not selfevident. Due to the dimension of (N]M)-images, classi"ers using these high-dimensional features directly will not provide e$cient algorithms for several reasons: in high-dimensional vector spaces the de"nition of similar vectors is somehow di$cult, since nearly all vectors are considered to be neighbors [18]. Furthermore, the comparison of vectors is the most often used operation within the classi"cation module and should be as e$cient as possible. The use of high-dimensional feature vectors contradicts this requirement [8]. To reduce the data, it is important to select or project features from a gray-level image. Especially for object recognition, traditional methods use the segmentation of (hopefully discriminating) features in the image, like edges or corners of an

object. These features allow the explicit use of geometrical relationships between 3-D models and 2-D observations. The geometry of object transforms and the projection of 3-D models into the 2-D image plane are well-understood and mathematically formalized [19]. But the usage of segmentation results shows some substantial disadvantages, which partially con"ne their practical use: f the quality of segmentation highly depends on the chosen algorithm and on illumination conditions as well as selected viewpoints, f robust, optimal, and reliable detection of features is far from its implementation, and "nally f the huge amount of data reduction induces a decrease of information, which might also decrease the discriminative power of resulting classi"ers. Appearance-based approaches to object recognition, however, prohibit the use of geometry, but the algorithms do not depend on a reliable and accurate detection of points or lines. The computation of features directly from graylevel images can be done by di!erent types of mappings:

212


1. The "rst stage might transform the gray-level image into another image which shows special properties. The discrete Fourier transform and spectrum [20], for example, results in features which are invariant with respect to translations of the objects in the image plane and thus reduce the search space with respect to pose parameters. Other "ltering operations, like highpass "lters, abate dependencies on illumination. These transforms again lead to large feature vectors for input signals and do not reduce dimensions. 2. For e$cient algorithms, however, it is essential that features are projected or selected to obtain small but still discriminating feature vectors. Because selecting the subset of best features is an NP complete problem, only an approximation of the optimal set can be computed within practical applications [18,21]. The transforms which reduce the dimension of features can have two di!erent motivations: one might be the application of some heuristic principles which project the feature vectors and show satisfying recognition results. Here we will consider a restricted class of transforms which have the property to be optimal with respect to objective functions. The applied objective functions f are based on basic assumptions concerning the distribution of feature vectors, f are comparatively easy to calculate, and f induce e$cient algorithms for the analytical computation of optimal feature transforms. Fig. 2 summarizes the general idea the subsequent analysis is based on.

5. Gray-level features, non-linear preprocessing, and feature selection The classi"cation and pose estimation task is generally formalized as a sequence of mappings which assign an image f3Rm showing one object to a preprocessed image h3Rm, then to a feature vector c3Rn(n;m), and "nally to the class ) of the observed pattern class. Furtheri more, related pose parameters, which are de"ned by three rotations and three components de"ning the translation vector, have to be computed. The classi"cation and localization of objects crucially depends on postulates which are the basic requirements of most pattern recognition algorithms. These postulates } as far as they are relevant for our application } are summarized in the following subsection, and they form the base of all subsequent linear feature transforms. 5.1. Postulates Usually, feature vectors suitable for 3-D recognition are expected to show a high discriminating power and to

allow reliable classi"cation as well as pose estimation. For that reasons, features have to satisfy basic postulates for decision making. f Similarity: objects belonging to the same pattern class show similar feature vectors independent of the associated classes. f Distinction: objects of distinct pattern classes have different feature vectors, which provide a high discriminating power. f Smoothness: small variations in pose or illumination induce small variations in associated features. Using these basic assumptions for the construction of good features, we derive di!erent types of linear feature transforms from high-dimensional into lower-dimensional feature spaces. The basic idea here is to select the transform such that the resulting features are optimal with respect to above postulates. In detail we will consider transforms which: f maximize the distance of all features among each other independent of pattern classes, f maximize the distance of features belonging to di!erent pattern classes (interclass distance), f minimize the distance of features belonging to the same pattern class (intraclass distance), and f optimize combinations of the above measures. However, it should be clear to the reader that linear transforms will not improve the recognition rate of classi"ers, even if we choose m"n. 5.2. Non-linear image transforms Before we transform the image matrix into lowerdimensional vectors, we transform sensor data into images, which show distinguished properties [22]. Examples for preprocessing operations are high-pass "lters, low-pass "lters, the application of the 2-D Fourier transform or the use of segmented images. The application of segmented images gives also a fundamental hint how recognition results are in#uenced by segmentation. A comparison of gray-level and feature-based identi"cation as well as pose estimation is possible based on this approach (cf. Section 9). Other approaches to object recognition do not allow comparative studies of that kind. Within this work we use the following preprocessing operations: the absolute values of the 2-D discrete Fourier transform (spectrum), the result of Gaussian "ltering, the absolute values of second derivatives (Laplace), the edge strength of pixels computed by the operators due to Nevatia}Babu and Sobel. Finally, we also use binary images, where edge pixels are black and the rest white (edge images).


213

5.3. Optimization-based feature selection and transform The basic idea of feature transforms is that we are looking for mappings which reduce the dimension of feature vectors and optimize optimality criteria related to the cited postulates of pattern recognition systems. The search for the optimal transform requires the restriction to special parametric types of transforms. It seems computationally prohibitive to search for the best transform without any constraints to its properties. For that reason, we restrict the subsequent discussion to linear transforms which map the m-dimensional vectors if of the sample set u"Mif3RmDi"1,2, rN from Rm to the n-dimensional features ic3Rn. The linear transform is obviously completely characterized by the matrix U3RnCm which maps the m-dimensional preprocessed image vector h to the n-dimensional feature vector c, i.e., c"Uh.

(1)

The nm components of the matrix are considered to be the variables of the search process, and thus the search space for suitable transforms is strongly restricted to an nm-dimensional vector space. This makes the search problem tractable and } as we will see in the following subsections } induces optimization problems which can be solved using basic and well-understood techniques of linear algebra. The computation of the optimal linear transform UH makes the use of objective functions necessary which have to be optimized with respect to the parameters, i.e., the components of U. In the following, we de"ne di!erent objective functions

G

RnCmPR, s: (2) i U C s (U), i where i"1, 2 ,2, according to the postulates summarized in Section 5.1. A transform UH is called optimal i with respect to s (U), if it holds i UH"argmax s (U), (3) i i ' presupposed s has to be maximized, and i UH"argmin s (U) (4) i i ' if s has to be minimized. Since scaling of the matrix i U would also a!ect the value of the objective, the mai trices are restricted to those composed of unit length vectors. In the following we use illustrative motivations for di!erent objectives by considering distributions of feature vectors. 5.4. Principal component analysis The most often used linear transform of this type results from the principal component analysis and is the so-called Karhunen}Loe% ve transform (KLT), [10,23].

Fig. 3. KLT and the ADIDAS-problem.

The idea of this transform is based on the reduction of the dimension of original image vectors h using a linear mapping U such that the resulting feature vectors c show pairwise maximum distance. For this transform U, the objective function s (U) thus is the mean squared 1 distance of all sample feature vectors ic"U ih to each other, i.e., 1 N N s (U)" + + (ic!jc)T(ic!jc). 1 N2 i/1 j/1

(5)

The use of KLT provides both advantages and disadvantages: the computation of the optimal linear transform UH with respect to s (U) does not require the classi"ca1 tion of sample vectors. Furthermore, feature vectors resulting from KLT allow the reconstruction of images with minimal mean quadratic error [9]. Problems, however, occur if the distribution of features is such that the principal axes of all classes are parallel to each other. A 2-D example, where we project the features onto the x-axis, is shown in Fig. 3 (ADIDAS-problem, Ref. [24]). Obviously, the projected features will allow no discrimination of these classes. For this situation, the optimal linear mapping related to s will not induce discriminat1 ing features, whereas the projection on the y-axis would. This simple example shows that other objective functions than s (U) seem to be useful or necessary for reduc1 ing the features' dimension and for providing a higher discriminating power. 5.5. Maximizing interclass distance Another plausible optimization criterion, which does not show the disadvantages of KLT, results from the distinction property. Features of one and the same pattern class should have maximum distance to features of the other pattern classes. In contrast to the KLT, however, this transform requires a classi"ed sample set. The original sample data are partitioned, i.e., u"X Q ui, where u "Mif D i"1,2, r N consists of all samples belonging i i i to pattern class ) . Thus the following objective function i

214


can be applied only for those sample sets, where such a labeling is available. Let ic denote the ith sample i vector which belongs to class ) . Of course, the number i r of sample data of each class may be di!erent, i.e., i r Or . The associated objective function based on the i j above motivated criterion is de"ned by 2 K i~1 1 s (U)" + + 2 K(K!1) rr i/2 j/1 i j ri rj ] + + (ic !jc )T (ic !jc ), (6) i j i j i/1 j/1 where K denotes the number of pattern classes. Now we use the classi"ed sample data and also de"ne a criterion which combines the ideas of s and s . For 1 2 each class ) we compute the mean vector k , i i i"1, 2, 2, K, and substitute the feature vectors of s 1 by the mean vectors. The objective s is thus de"ned by 3 K i~1 2 + + (l !l )T (l !l ). (7) s (U)" i j i j 3 K(K!1) i/2 j/1 If we optimize s with respect to the linear transform U, 3 the distance between the class centers is maximized. Consequently, the above ADIDAS-problem can be solved using both s and s . 2 3 5.6. Minimizing intraclass distance The objective functions discussed so far maximize distances of features. In order to take the similarity postulate into account, we de"ne an objective which yields a measure for the density of features belonging to the same pattern class. Features of the same pattern class should have a minimum distance and therefore we suggest to minimize the intraclass distance de"ned by 1 K 1 ri ri s (U)" + + + (ic !jc )T (ic !jc ). (8) 4 i i i i K r2 i/1 i i/1 j/1 The use of this objective function also requires a set of classi"ed training data. The optimal feature transform w.r.t. s results from solving 4 UH"argmin s (U). (9) 4 4 ' The trivial solution U"0 is excluded, because U has to 4 be composed of unit length vectors. As we see later, the matrix UH will be composed by i eigenvectors of a kernel matrix Q(i). In this application the number of sample image vectors if will be much i smaller than the dimension of these vectors. Therefore, the matrix Q(4) will have a fairly large and therefore non-trivial null space. Projection to this null space will minimize the objective with s "0. 4 In this space, as Fig. 4 shows, each class will be represented by a single point. If further feature reduction has to be applied, a proper subspace must be selected to

Fig. 4. Projection to null space.

allow good separation of class points. This can be done by another KLT. Due to the high dimension of the null space and numerical problems in evaluating eigenvectors to eigenvalue zero, we only consider combined objectives which maximize the inter- and minimize the intraclass distance at the same time. 5.7. Combination of inter- and intraclass distance The simplest way of combining the inter- and intraclass distance measure is the use of fractions or linear combinations of s , s , and s , i.e., we could, for instance, 2 3 4 de"ne s (U)"s (U)#hs (U), 5 3 4

(10)

s (U) or s (U)" 2 , (11) 7 s (U) 4 and compute linear transforms using these objectives. Herein, the weighting factor h is a free variable which has to be chosen by the user. The following considerations are restricted to the "rst de"nition s . An experimental 5 comparison of s (Fisher transform) and s can be found 7 1 in Ref. [12]. s (U)"s (U)#hs (U) 6 2 4

6. Optimization of objective functions The optimal linear transforms UH, i"1, 2, 3, 4, 5, with i respect to the introduced objective functions s , s ,2, s 1 2 5 are not obvious considering the complicated sums in Eqs. (5)}(8) and (10). Of course, we could start a brute force exhaustive optimization procedure, but concerning above objectives a simpli"cation of the related optimization tasks results from a reorganization of summations and multiplications. Indeed, all objective functions can be written in the following sum of quadratic forms: m s (U)"2 + uT Q(i)u , (12) i l l l/1


where the kernel matrix Q(i) corresponding to the ith objective function s (U) is implicitly de"ned, and i u , l"1, 2 ,2, m, denote the column vectors of the l transform U, i.e., we obtain U"(uT, uT ,2, uT ) where 1 2 m u 3Rn. l The introduction of kernel matrices shows one crucial advantage: the optimization of the introduced objectives s is reduced to the computation of eigenvectors and i eigenvalues due to the quadratic forms involved in Eq. (12). It is a well-known result from linear algebra that quadratic terms are minimal (resp. maximal) if the vectors u are eigenvectors corresponding to the minimal l (resp. maximal) eigenvalues. The computation of the optimal scatter matrix UH thus i proceeds as follows: 1. we compute the eigenvalues and eigenvectors of the involved kernel matrices Q(i), 2. sort the eigenvalues, 3. de"ne the n rows of the scatter matrix UH to be the i n eigenvectors related to the n eigen values; herein we take the n highest eigenvalues, if the objective function has to be maximized, otherwise we use the vectors corresponding to the smallest eigenvalues. The remaining problem is the explicit de"nition of kernel matrices, and for the implementation of the proposed feature transforms, however, some numerical aspects and computational considerations are required. We prove the validity of Eq. (12) exemplary for s in 3 the Appendix by explicitly computing the kernel matrix Q(3). The technical aspects of computations of other kernel matrices are quite similar and left to the reader. In the following, we present only the "nal kernel matrices related to above objective functions, since these will be required for formalizing the optimization algorithms. 6.1. Kernel matrix of s 1 Elementary algebraic transforms show that using the objective function s the kernel matrix Q(1) is simply the 1 covariance matrix of the sample set, which is de"ned by

A

B

1 r 1 r 2 Q(1)" + jfj fT! + jf r r j/1 j/1 1 r ! + (jf!l) (jf!l)T, r j/1

6.2. Kernel matrices of s and s 2 3 Considering the interclass distance and the related objective function s , we get the explicit kernel matrix 2 1 K 1 ri + jf jfT Q(2)" + i i K r i/1 i j/1 1 K i ! + + K(K!1) i/2 j/1

A

1 ri rj + jf + jfT i j rr i j j/1 j/1

B

1 rj ri # + jf + jfT j i rr i j j/1 j/1

2 K " + (l !lN ) (l !lN )T, i i K(K!1) i/1

(15)

where 1 K 1 ri l " + jf and lN " + l . i i r i K i j/1 i/1

(16)

The reorganization of arithmetic operations in s results 3 in the explicit kernel matrix: K i~1 1 Q(3)" + + (l !l ) (l !l )T. i j i j K(K!1) i/2 j/1

(17)

Obviously, the matrix Q(3) is the kernel matrix of a KLT based on mean vectors instead of feature vectors. In contrast to other kernels, the rank of this matrix is not bounded by the cardinality of the feature set, but by the class number. 6.3. Kernel matrix of s 4 The kernel matrix Q(4) is given by

A

B

1 K 1 ri Q(4)" + + jf jfT!l lT . i i i i K r i/1 i j/1

(18)

This result shows a connection between the kernel matrices of s , s and s . Indeed the identity 2 3 4 Q(4)"Q(2)!Q(3)

(19)

is valid. 6.4. Kernel matrices of s 5 (13)

where l denotes the mean vector of the non-classi"ed sample data, i.e., 1 r l" + jf. r j/1

215

The explicit kernel matrices for linear combinations of objective functions are trivial. Due to the linear nature of involved mappings the kernel matrices are linear combinations of the kernel matrices of its summands Q(5)"Q(2)#hQ(4)"Q(3)#hI Q(4),

(14)

where h and hI denote the weighting factors of Q(4).

(20)

216


7. Computational considerations The direct calculation of the eigenvalues and vectors of Q(i) is computationally prohibitive due to the storage requirements. If we, for instance, assume images of size 128]128, which is a quite low resolution for common computer vision purposes, we get m"16 384 rows and columns for kernel matrices Q(i). If we suppose, for example, that the entries of the matrix are double valued (i.e., eight byte for each entry), this matrix has storage requirements of about 2 GB. This simple numerical example shows that there is a need for more sophisticated methods to compute the optimal linear transforms related to the objective functions s , s , s , s , and s as well 1 2 3 4 5 as associated kernel matrices. 7.1. Implicit computation of eigen vectors The storage requirements can be reduced using a result of singular-value decomposition theory. Let us assume we have to compute the eigenvalues of a matrix Q3RmCm which can be factorized as follows: Q"FFT,

(21)

where F3RmCp, p(m. As already mentioned, the size of the matrix is intractable for the main memory of our computer. We are interested in computing the eigenvectors and eigenvalues, but a straightforward computation is prohibited. Instead of considering Q directly, we de"ne according to Murase and Lindenbaum [25] the implicit matrix Q) "FTF,

(23)

using (22) we thus get FTFu( "jK u( . l l l

(24)

In the next step, we multiply both sides by F, this yields FFT(Fu( )"jK (Fu( ) l l l

(25)

and thus we get Q(Fu( )"jK (Fu( ). l l l

Q(i)"F(i)F(i)T.

(27)

For that reason, the following subsections will derive the required factorizations of involved kernel matrices. 7.2. Reorganization of Q(1) The kernel matrix Q(1) is the covariance matrix of the given sample data, i.e., 1 r Q(1)" + (if!l) (if!l)T. r i/1

(28)

We de"ne F(1)"J2 (1f!l,2,rf!l)3RmCr, r

(29)

and it is obvious that Q(1)"F(1)F(1)T.

(30)

Before we compute the factorization of Q(2), it is advantageous to consider the decomposition of Q(3) and Q(4) (see Eq. (19)). This concrete example shows that there is a trade of between the storage requirements of implicit kernel matrices and the size of sample sets: here we have r"p, i.e., the higher r, the higher is the reliability of resulting models. Higher p-values, however, increase the storage requirements.

(22)

and observe that there is a remarkable relation between the eigenvalues and eigenvectors of Q and Q) . Let u( denote the eigenvectors and jK the eigenvalues of l l the implicit matrix Q) . The eigenvectors and values are de"ned by Q) u( "jK u( ; l l l

This result proves that the eigenvalues and eigenvectors of the kernel matrices Q(i) can be computed with low memory requirements presupposed p;m and matrices can be factorized in the form of [26]

7.3. Reorganization of Q(3) Analogous to Q(1) we get for class centers the decomposition Q(3)"F(3)F(3)T,

(31)

where J2 F(3)" (l !lN ,2, l !lN )3RmCK. 1 K r k

(32)

The scaling factor J2/JK is important if we use the combined distance measures. Otherwise this factor can be neglected. 7.4. Reorganization of Q(4)

(26)

The last equation shows that each eigenvalue of Q is also an eigenvalue of Q) , and the eigenvectors are related by the linear transform F.

The kernel matrix 1 K 1 ri Q(4)" + + (jf !l ) (jf !l )T i i i i K r i/1 i j/1

(33)


217

can also be factorized in the required manner. The similarity to Q(1) is evident, and analogous to Eq. (29) we de"ne the class-dependent matrices

f manifold models as suggested by Murase and Nayar [9], and f Gaussian models.

J2 F" (1f !l ,2, rif !l )3RmCri, i Jr i i i i i

More recent classi"cation methods using support vector machines are omitted [3].

(34)

where i"1, 2 ,2, K. The factor J2/Jr is necessary, i because r varies for di!erent classes ) . The summation i i of matrix products can be written using matrix multiplication, i.e.,

AB

FT 1 1 K 1 Q(4)" + F FT" (F ,2, F ) F i i 1 K K K i/1 FT K 1 " F(4)F(4)T. K

(35)

7.5. Reorganization of Q(2)

8.1. Manifold models Objects have several degrees of freedom. Di!erent rotation angles, for instance, result in di!erent feature vectors. It is suggesting to use parametric models (curves) with respect to these variables. Manifold models result from sample feature vectors by interpolation. Fig. 5 shows 3-D feature vectors and the interpolated manifold model. These manifolds are computed for each object class. The class decision is based on the minimization of the distance between an observed feature vector and the manifold models. The parameter vector associated with the manifold point, which has the lowest distance to the observation, de"nes the pose parameters.

Using Eq. (19) we obviously get Q(2)"F(2)F(2)T"(F(3), F(4)) (F(3), F(4))T.

(36)

7.6. Reorganization of Q(5) The kernel matrix of the combined objective s is 5 Q(5)"Q(4)#hQ(3)"(F(4), JhF(3)) (F(4), JhF(3))T "F(5)F(5)T.

(37)

The weight factor h has to be positive, because of the square root in the de"nition of F(5). The theoretical part has introduced objective functions which are used to compute optimal linear transforms and which are motivated by the basic postulates of pattern recognition. The required linear mapping is e$ciently computed reducing objectives to quadratic forms and solving eigenvalue problems. Related problems with storage requirements of involved computations were solved by the introduction of implicit matrices. In the following section we will compare these transforms and techniques experimentally. Before, it is necessary to de"ne the used models and decision rules the experimental evaluation is based on.

Fig. 5. Example of a manifold model with one degree of freedom (rotation angle) resulting from KLT features. The model corresponds to the second object shown in Fig. 1. The gray-level image is not preprocessed.

8. Models and decision rules The classi"cation of objects is based on the introduced features of the eigenspace. Samples of the training set are represented by points within the eigenspace. Due to the fact that lighting conditions, position and orientation of objects vary, feature vectors di!er in the eigenspace. Here we distinguish two di!erent types of models:

Fig. 6. Clusters of features belonging to four classes shown in Fig. 7. Feature transforms use the combined objective s where 5 h"10~4. Herein, the original image matrix was transformed into the Fourier spectrum.

218


Fig. 7. Industrial objects captured by uncalibrated cameras.

8.2. Gaussian densities Simpler modeling schemes characterize all sample features assigned to one class by a single probability density function. Here we use multivariate Gaussians for modeling and decide for that class with the highest aposteriori probability. Of course, these statistical models do not allow for pose estimation. Therefore, these models are especially useful for those applications or training samples, where no pose information is required or available for model generation. Fig. 6 shows four clusters of features belonging to object classes shown in Fig. 7. Within the chosen probabilistic framework, each cluster is characterized by a 3-D Gaussian density.

9. Experimental results The experimental evaluation provides a comparative empirical study of the introduced transforms UH. Before i we describe detailed results, we give a brief overview of the experimental set up and the used image data. 9.1. Experimental setup and image data The experimental evaluation is done on a HP 9000/735 (99 MHz, 124 MIPS) using 128]128 images. Within the experiments we use two di!erent image databases. To provide the capability of comparing the introduced feature transforms with other methods and di!erent approaches to 3-D object recognition, we discuss some experiments using the standard images of the Columbia University image database.1 We restrict these recognition experiments to the "ve object classes, which were already shown in Fig. 1. For each object 36 training and 36 test views are available. The images show single 3-D objects with homogeneous background rotated by 53. Rotations of 0, 10, 20,2, 3603 are used for training. The

1 See http://www.cs.columbia.edu/CAVE/coil-20.html.

recognition experiments run on images showing rotations 5, 15,2, 3553. Training and test sets are disjoint, and contain images showing objects of varying pose. Occlusion, except self-occlusion, does not occur. For each training and test view the pose parameters, i.e. the single rotation angle, are known. Illumination conditions are constant for all samples. In addition to these idealized images (homogeneous black background) we also consider industrial parts from a real application using an uncalibrated camera.2 We use four objects which are shown in Fig. 7. Of each object 200 di!erent views are available, including also partially occluded objects. Planar rotations and translations as well as lighting are chosen randomly. The set of 2-D views is partitioned into training and test sets of equal cardinalities. In contrast to the above-mentioned image database, the pose parameters are not available. 9.2. Varied parameters and evaluation criterions The computation of features has several degrees of freedom. Within the experiments, we varied the following parameters: f dimension of used features, f di!erent preprocessing methods, and f di!erent objective functions. The basic criteria for experimental evaluations are the recognition rates, errors in pose estimates, and the runtime. The used models are both manifold models as suggested in Ref. [9], which also consider the pose parameters, and simple statistical models. Statistical models assume normally distributed feature vectors for each class, and do not use pose information within the training data. The experiments related to pose estimation accuracy are restricted to manifold models and therefore to images of the Columbia database.

2 These images are available via URL: http://www5.informatik.uni-erlangen.de.


9.3. Pose estimation results We tested the pose estimation accuracy using manifold models. The considered object transforms are rotations around a single coordinate axis. The features are transformed by linear mappings induced by the discussed objective functions. Table 1 summarizes the obtained errors with respect to rotations around the z-axis of the world coordinate system. Obviously, the best results are achieved by the combined objectives. The overall improvement with respect to the standard principal component analysis, however, is minor. Table 2 summarizes the errors based on di!erent preprocessing operations and a subsequent principal component analysis in 10 dimensions. If no bijective mappings are used, we expect a reduction of accuracy. Indeed, the experiments show that the best pose estimates result from the immediate use of the gray-level image. The worst accuracy is obtained by using edge images. These examples prove that the appearance-based approach does not provide reliable pose estimates if segmented images are used. Using images containing lines only decreases the accuracy of pose estimates

219

drastically. Appearance-based pose estimation techniques should not be applied to this type of preprocessed images. 9.4. Recognition results In the following experiments we compare various preprocessing operations and linear transforms with respect to the resulting recognition rates. 9.4.1. Columbia images Using the Columbia images (see Fig. 1, "ve classes) we compare manifold models and statistical models based on simple multivariate Gaussians. The graphs shown in Fig. 8 summarize the recognition results for varying linear transforms and di!erent dimensions of used feature vectors. These experiments prove that the recognition rate is 100% for all transforms, if the dimension of eigen vectors is at least 3 and manifold models are used. For lower-dimensional features, s dominates both with 3 respect to manifold and Gaussian models. The recognition results using combined objectives with di!erent weights are summarized in Fig. 9.

Table 1 Mean errors and deviations in estimated rotation angles based on 10-D feature vectors for s , s and s , and 4-D feature vectors 1 2 5 for s 3

Table 2 Mean error in pose estimates using s as objective function and 1 di!erent preprocessing operations. The chosen dimension of eigenvectors is 10

Objective function

Mean error (deg)

Standard deviation (deg)

Filter

Error (deg)

s 1 s 2 s 3 s (h"10~4) 5 s (h"0.1) 5 s (h"0.5) 5

0.71 0.71 8.45 0.69 0.70 0.67

0.78 0.79 43.32 0.77 0.78 0.74

No "ltering Spectrum Gaussian "ltering Edge detection Laplace Nevatia Sobel

0.70 0.96 0.74 14.84 3.81 2.96 1.73

Fig. 8. Comparison of di!erent linear image transforms using s , s , and s (Columbia images) and di!erent models: manifold models 1 2 3 (left) and Gaussian models (right).

220


Fig. 9. Combined objective s "s #hs , where h"10~4, 10~1, 1, 1 (Columbia images). 5 3 4 2

Fig. 10. Recognition rates using s and di!erent preprocessing operations. 1

Recognition results using di!erent preprocessing operations are summarized in Fig. 10 using objective s and 1 Fig. 11, where we have used s . It is conspicuous that the 3 optimization criterion s combined with the spectrum 3 shows the highest recognition rates independently of the selected model. The main reason for that is the invariance of the spectrum with respect to object translations in the image plane. All examples show that manifold models provide higher recognition rates than Gaussian models. How-

ever, manifold models require pose information within the training samples, probabilistic models using multivariate Gaussians do not. 9.4.2. Industrial objects The next experiments use images, where no pose information is available (see Fig. 7, four classes). Therefore, we only consider probabilistic models and analyze the recognition in the presence of occlusion. The recognition rates do also vary with the dimension of eigenvectors. In


221

Fig. 11. Recognition rates using s and di!erent preprocessing operations. 3

Table 3 Recognition rates using images of industrial objects shown in Fig. 7. The dimension of the eigenspace is 20. The linear transform is based on s , and the columns show recognition rates 1 using no preprocessing, Gaussian "ltering (GF) and segmenting the background (BG) Class

No occlusion

Occlusion

*

GF

BG

*

GF

BG

) 1 ) 2 ) 3 ) 4

25 87 1 57

41 85 0 48

40 80 3 64

10 30 0 80

10 20 0 80

10 30 0 80

Average

43

44

47

30

28

30

contrast to previous experiments, we restrict the dimension of the eigenspace to 10 and 20. Table 3 shows the low recognition rates for the industrial objects based on linear transforms using s , even if di!erent preprocessing 1 operations are used. Obviously, partially occluded objects cannot be classi"ed using the high-dimensional eigenvectors and this approach. Recognition rates are

comparable to random estimates. The use of linear transforms introduced above also does not essentially improve the recognition results. Tables 4 and 5 also show the curse of dimensionality: an increasing dimension of feature vectors does not necessarily increase recognition results. The main reason for low recognition rates is the presence of translations in the image plane. If we detect the object and consider only pixels belonging to the object, we observe a remarkable improvement of recognition rates. We get 100% even in the presence of occlusion. Therefore, we use the spectrum of images (absolute values of the 2-D Fourier transform), which is known to be invariant with respect to translations. These experiments show that rotations do not in#uence the accuracy of recognition in contrast to translations. The segmentation of objects, i.e. the bi-partition of image points into object and background pixels, or usage of Fourier transform for object classi"cation is advantageous for recognition, if no pose information is available within the training data. 9.5. Run time The run time behavior of the complete system is summarized in Tables 6}9. All numbers are based on the

222


Table 4 Recognition rates based on 10-dimensional eigen vectors. The images are preprocessed such that background and object pixels are separated Class

No occlusion

Occlusion

s 1

s 2

s (0.5) 5

s (10~4) 5

s 1

s 2

s (0.5) 5

s (10~4) 5

) 1 ) 2 ) 3 ) 4

90 99 92 87

84 98 93 87

88 98 92 87

61 88 62 62

30 100 60 100

30 100 60 100

30 100 60 100

40 100 40 100

Average

92

90

91

68

72

72

72

70

Table 5 Recognition rates based on 20-dimensional eigenvectors. The images are preprocessed such that background and object pixels are separated Class

No occlusion

Occlusion

s 1

s 2

s (0.5) 5

s (10~4) 5

s 1

s 2

s (0.5) 5

s (10~4) 5

) 1 ) 2 ) 3 ) 4

40 80 3 64

35 80 9 65

47 83 25 63

61 88 62 62

10 30 0 80

10 30 0 90

10 30 0 100

30 40 0 80

Average

47

47

54

68

30

33

35

38

Table 6 Recognition rates using 20-dimensional feature vectors Method

Recognition rate (%) No occlusion

Non-invariant features Separated object/background pixels Spectrum

Table 7 Run time of the learning stage dependent on the dimension of used eigenspaces (180 images)

Occlusion

47 99

30 73

100

100

Columbia image database including 180 training images of size 128]128. Table 6 shows the time required for training using all images of the training set. Most of the time is obviously required for the computation of eigenvectors. Table 8 shows the relation between the time for classi"cation and the dimension of the eigenspace.

10. Summary and conclusions Standard linear feature transforms which are broadly used in pattern recognition and speech recognition are

Dimension of eigenvectors

5 10 20

Computation of eigenvectors (min:s)

Training (ms) Gauss

Manifold

3:34 3:55 4:18

(10 (10 40

(10 (10 (10

Table 8 Run time of eigenvalue computations (10-dimensional eigenspace) dependent on the number of training images Number of images

Time [min:s]

45 90 135 180

0:38 1:37 2:34 3:55

J. Hornegger et al. / Pattern Recognition 33 (2000) 209}224 Table 9 Run time of the classi"cation module. The images are represented as vectors Dimension of eigenspace

5 10 20

Projection

30 60 110

Classi"cation Gauss (ms)

Manifold (ms)

(10 (10 30

560 650 790

successfully applied to solve object recognition and pose estimation problems in the "eld of 3-D computer vision using gray-level images. This paper has summarized various objective functions for the computation of optimal feature transforms: the principal component analysis, the interclass distance, the intraclass distance, and various combinations. We have shown how the associated optimization problems are reduced to the computation of eigenvectors. A twostage re-organization of considered objective functions leads to computational practical solutions: 1. the transform of the objective functions into sums of quadratic forms that reduces the optimization problem to the computation of eigenvectors, and 2. the factorization of kernel matrices into products of matrices and its transposes which induces lower storage requirements for computing eigenvalues and eigenvectors. The experimental evaluation provides a comparison of new types feature transforms. Based on a standard image database, we prove empirically that the best pose estimation results are provided by a transform which maximizes a combination of intra- and interclass distances. The recognition results show highest accuracy, if the distance of class-speci"c mean vectors is maximized. Dependent on the selected dimension of feature vectors we have shown that a dimension of 4 already leads to recognition results of 100% correctness. Instead of manifolds, we have also tested the recognition rates using the assumption of normally distributed feature vectors. Using spectral features which are invariant to translations in the image plane we observed also recognition rates of 100% using industrial objects, where the training set includes no pose information. Considering these results, we conclude that appearance-based object recognition systems can compete with standard geometrical approaches both with respect to recognition rates and run time behavior. The introduction of implicit kernel matrices has reduced storage requirements. The problems which are not yet solved su$ciently are the explicit modeling of occlusion, the analysis of

223

multiple object scenes, and the construction of object models in the presence of background features. The application of the considered transforms to classify and localize objects with heterogeneous background is straightforward using the hierarchical framework introduced in Ref. [15].

Acknowledgements The authors gratefully acknowledge S. Nene, H. Murase and K. Nayar for the friendly admission to use their Software Library for Appearance Matching (SLAM).

Appendix A We consider the criterion s and get the following 3 quadratic form: K 2 + s " 3 K(K!1) i/2 K 2 " + K(K!1) i/2 2 K " + K(K!1) i/2

i~1 + (Ul !Ul )T (Ul !Ul ) i j i j j/1 i~1 + (l !l )T UTU(l !l ) i j i j j/1 i~1 + tr (UTU(l !l ) (l !l )T) i j i j j/1 1 "2tr UTU K(K!1)

C A

K i~1 ] + + (l !l ) (l !l )T i j i j i/2 j/1 "2tr [UTUQ(3)]

BD

r "2 + uTQ(3)u . i i i/1 The kernel matrix for this case thus is [8] 1 K i~1 Q(3)" + + (l !l ) (l !l )T. i j i j K(K!1) i/2 j/1

(38)

(39)

References [1] A.K. Jain, P.J. Flynn (Eds.), Three-Dimensional Object Recognition Systems, Elsevier, Amsterdam, 1993. [2] J. Ponce, Zisserman, M. Hebert (Eds.), Object Representation in Computer Vision, Lecture Notes in Computer Science, vol. 1144, Springer, Heidelberg, 1996. [3] M. Pontil, A. Verri, Support vector machines for 3d object recognition, IEEE Trans. Pattern Anal. Machine Intell. (PAMI) 20 (1998) 637}646. [4] B.D. Ripley, Pattern Recognition and Neural Networks, Cambridge University Press, Cambridge, 1996.

224


[5] C.M. Bishop, Neural Networks for Pattern Recognition, Clarendon Press, Oxford, 1995. [6] E. Trucco, A. Verri, Introductory Techniques for 3-D Computer Vision, Prentice-Hall, New York, 1998. [7] V.N. Vapnik, The Nature of Statistical Learning Theory, Springer, Heidelberg, 1996. [8] H. Niemann, Klassi"kation von Mustern, Springer, Heidelberg, 1983. [9] H. Murase, S.K. Nayar, Visual learning and recognition of 3-D objects from appearance, Int. J. Comput. Vision 14 (1) (1995) 5}24. [10] K. Karhunen, UG ber lineare Methoden in der Wahrscheinlichkeitsrechnung, Ann. Acad. Sci. Fenn. Ser. AI (1947) 37. [11] Y.T. Chien, K.S. Fu, Selection and ordering of feature observations in a pattern recognition system, Inform. Control 12 (1968) 395}414. [12] P.N. Belhumeur, J.P. Hespanha, D.J. Kriegman, Eigenfaces vs. Fisherfaces: recognition using class speci"c linear projection, IEEE Trans. Pattern Anal. Machine Intell. 19 (7) (1997) 711}720. [13] H. Murase, M. Lindenbaum, Spatial temporal adaptive method for partial eigenstructure decomposition of large images, Tech. Report 6527, Nippon Telegraph and Telephone Corporation, March 1992. [14] M. Turk, A. Pentland, Eigenfaces for recognition, J Cognitive Neurosc. 3 (1) (1991) 71}86. [15] H. Murase, S.K. Nayar, Detection of 3D objects in cluttered scenes using hierarchical eigenspace, Pattern Recognition Lett. 18 (5) (1997) 375}384. [16] H. Bischof, A. Leonardis, Robust recovery of eigenimages in the presence of outliers and occlusions, Int. J. Comput. Inform. Technol. 4 (1) (1996) 25}38.

[17] R. Epstein, P.W. Hallinan, A.L. Yuille, 5$2 eigenimages su$ce: an empirical investigation of low-dimensional lighting models, Proceedings of IEEE Workshop on Physics Based Modeling in Computer Vision, Boston, June 1995, pp. 108}116. [18] H. Niemann, Pattern Analysis and Understanding, Springer Series in Information Sciences, vol. 4, Springer, Heidelberg, 1990. [19] O.D. Faugeras, New steps toward a #exible 3d-vision system for robotics, Proceedings of the Eighth International Confernce on Pattern Recognition, Montreal, 1987, pp. 796}805. [20] A.V. Oppenheim, R.W. Schafer, Digital Signal Processing, Prentice-Hall, Englewood Cli!s, NJ, 1975. [21] A.K. Jain, D. Zongker, Feature selection: evaluation, application, and small sample performance, IEEE Trans. Pattern Anal. Machine Intell. 19 (2) (1997) 153}158. [22] W.K. Pratt, The PIKS Foundation C Programmers Guide, Manning, Greenwich, 1995. [23] K. Karhunen, I. Selin (trans.), On linear methods in probability theory. Translation to [10] T-131, The Rand Corporation, August 1960. [24] E.G. Schukat}Talamazzini, Automatische Spracherkennung, Vieweg, Wiesbaden, 1995. [25] H. Murase, M. Lindenbaum, Spatial temporal adaptive method for partial eigenstructure decomposition of large images, IEEE Trans. Image Process. 4 (5) (1995) 620}629. [26] H. Murakami, V. Kumar, E$cient calculation of primary images from a set of images, IEEE Trans. Pattern Anal. Machine Intell. 4 (5) (1982) 511}515.

About the Author*JOACHIM HORNEGGER graduated (1992) and received his Ph.D. degree in Computer Science (1996) at the UniversitaK t Erlangen-NuK rnberg, Germany, for his work on statistical object recognition. Joachim Hornegger was research and teaching associate at UniversitaK t Erlangen}NuK rnberg, a visiting scientist at the Technion, Israel, at the Massachusetts Institute of Technology, USA, and a visiting scholar at Stanford University, USA. His major research interests are 3D computer vision, 3D object recognition and statistical methods applied to image analysis problems. Joachim has taught computer vision and pattern recognition at the UniversitaK t Erlangen-NuK rnberg, Germany, at the University of Seville, Spain, and at Stanford University, USA. He is the coauthor of a monography on pattern recognition and image processing in C##. Currently Joachim is with Siemens Medical Systems working on 3-D reconstruction and a lecturer at the UniversitaK t Erlangen-NuK rnberg, Germany. About the Author*HEINRICH NIEMANN obtained the degree of Dipl.-Ing. in Electrical Engineering and Dr.-Ing. at Technical University Hannover in 1966 and 1969, respectively. During 1966/67 he was a graduate student at the University of Illinois, Urbana. From 1967 to 1972 he was with the Fraunhofer Institut fuK r Informationsverarbeitung in Technik und Biologie, Karlsruhe, working in the "eld of pattern recognition and biological cybernetics. During 1973}1975 he was teaching at Fachhochschule Giessen in the department of Electrical Engineering. Since 1975 he has been Professor of Computer Science at the University of Erlangen-NuK rnberg, since 1988 he is also head of the research group `Knowledge Processinga at the Bavarian Research Institute for Knowledge Based Systems (FORWISS) where he also was on the board of directors for six years. During 1979}1981 he was dean of the Engineering faculty of the University, in 1982 he was program chairman of the 6. International Conference on Pattern Recognition in MuK nchen, Germany, in 1987 he was director of the NATO Advanced Study Institute on `Recent Advances in Speech Understanding and Dialog Systemsa, in 1992 he was Program Chairman of the conference track on `Computer Vision and Applicationsa at the 11. International Conference on Pattern Recognition in The Hague, The Netherlands, and he was program co-chairman at the International Conference on Acoustics, Speech, and Signal Processing 1997 in MuK nchen. His "elds of research are speech and image understanding and the application of arti"cial intelligence techniques in these "elds. He is on the editorial board of Signal Processing, Pattern Recognition Letters, Pattern Recognition and Image Analysis, and Journal of Computing and Information Technology. He is the author or coauthor of 6 books and about 250 journal and conference contributions as well as editor or coeditor of 23 proceedings and special issues. He is a member of ESCA, EURASIP, GI, IEEE, and VDE. About the Author*ROBERT RISACK received his M.Sc. degree in Computer Science (Diplom Informatiker) from the UniversitaK t Erlangen-NuK rnberg, Germany. Since May 1997 Robert has been a Ph.D. student at the Fraunhofer Institut fuK r Informations-und Datenverarbeitung, Karlsruhe. He is working on the design and implementation of computer vision systems.


Adaptive document image binarization J. Sauvola*, M. PietikaK inen Machine Vision and Media Processing Group, Infotech Oulu, University of Oulu, P.O. BOX 4500, FIN-90401 Oulu, Finland Received 29 April 1998; accepted 21 January 1999

Abstract A new method is presented for adaptive document image binarization, where the page is considered as a collection of subcomponents such as text, background and picture. The problems caused by noise, illumination and many source type-related degradations are addressed. Two new algorithms are applied to determine a local threshold for each pixel. The performance evaluation of the algorithm utilizes test images with ground-truth, evaluation metrics for binarization of textual and synthetic images, and a weight-based ranking procedure for the "nal result presentation. The proposed algorithms were tested with images including di!erent types of document components and degradations. The results were compared with a number of known techniques in the literature. The benchmarking results show that the method adapts and performs well in each case qualitatively and quantitatively. ( 1999 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Adaptive binarization; Soft decision; Document segmentation; Document analysis; Document understanding

1. Introduction Most document analysis algorithms are built on taking advantage of the underlying binarized image data [1]. The use of a bi-level information decreases the computational load and enables the utilization of the simpli"ed analysis methods compared to 256 levels of grey-scale or colour image information. Document image understanding methods require logical and semantic content preservation during thresholding. For example, a letter connectivity must be maintained for optical character recognition and textual compression [2]. This requirement narrows down the use of a global threshold in many cases. Binarization has been a subject of intense research interest during the last ten years. Most of the developed algorithms rely on statistical methods, not considering the special nature of document images. However, recent developments on document types, for example documents with mixed text and graphics, call for more specialized binarization techniques. In current techniques, the binarization (threshold selection) is usually performed either globally or locally. * Corresponding author. Tel.: #358-40-5890652. E-mail address: [email protected]." (J. Sauvola)

Some hybrid methods have also been proposed. The global methods use one calculated threshold value to divide image pixels into object or background classes, whereas the local schemes can use many di!erent adapted values selected according to the local area information. Hybrid methods use both global and local information to decide the pixel label. The main situations in which single global thresholds are not su$cient are caused by changes in lumination (illumination), scanning errors and resolution, poor quality of the source document and complexity in the document structure (e.g. graphics is mixed with text). When character recognition is performed, the melted sets of pixel clusters (characters) are easily misinterpreted if binarization labelling has not successfully separated the clusters. Other misinterpretations occur easily if meant to be clusters are wrongly divided. Fig. 1 depicts our taxonomy (called MSLG) and general division into thresholding techniques according to level of semantics and locality of processing used. The MSLG can be applied in pairs, for example (ML), (SL), (MG) and (SG). The most conventional approach is a global threshold, where one threshold value (single threshold) is selected for the entire image according to global/local information. In local thresholding the threshold values


226

J. Sauvola, M. Pietika( inen / Pattern Recognition 33 (2000) 225}236

Fig. 1. Taxonomy of thresholding schemes.

Fig. 2. Examples of document analysis problem types in binarization.

are determined locally, e.g. pixel by pixel, or region by region. Then, a speci"ed region can have &single threshold' that is changed from region to region according to threshold candidate selection for a given area. Multithresholding is a scheme, where image semantics are evaluated. Then, each pixel can have more than one threshold value depending on the connectivity or other semantic dependency related to physical, logical or pictorial contents. Many binarization techniques that are used in processing tasks are aimed at simplifying and unifying the image data at hand. The simpli"cation is performed to bene"t the oncoming processing characteristics, such as computational load, algorithm complexity and real-time requirements in industrial-like environments. One of the key reasons when the binarization step fails to provide the subsequent processing a high-quality data is caused by the di!erent types and degrees of degradation introduced to the source image. The reasons for the degradation may vary from poor source type, the image acquisition process to the environment that causes problems for the image quality directly. Since the degradation is unquestionably one of the main reasons for processing to fail, it is very important to design the binarization technique to detect and "lter possible imperfections from becoming the subject for processing and potential cause of errors for post-processing steps. Most degradation types in document images a!ect both physical and semantic understandability in the document analysis tasks, such as page segmentation, classi"cation and

optical character recognition. Therefore, the result after all the desired processing steps can be entirely unacceptable, just because of the poorly performed binarization. Fig. 2 depicts two types of typical degradation, when dealing with scanned grey-scale document images. In Fig. 2a the threshold &base line' is changing due to illumination e!ect or implanted (designed) entity. Then, each object has a di!erent base level that a!ects the object/ non-object separation decision in selecting threshold(s). In Fig. 2b a general type &stain problem' is presented. In this case, the background and object levels are #uctuating from clear separation to non-clear separation and small level di!erence between object/non-object. The optimal threshold lines are drawn to both images to depict the base line that a successful binarization algorithm should mimic. Fig. 3 presents another type of problem, frequently occurring in scanned document images: more than two di!erent levels are visible in textual areas due to transparency of the next page. Then, a binarization algorithm should cope with at least two di!erent threshold candidates: background-transparent text and background-text. The binarized example presents a correct binarization result. 1.1. Survey on document image binarization techniques The research on binarization techniques originates from the traditional &scene image' processing needs to


227

Fig. 3. Example of good binarization on degraded sample image.

optimize the image processing tasks in terms of image data at hand. While the image types have become more complex the algorithms developed have gained wider theoretical grounds. Current trend seems to move forward image domain understanding based binarization and the control of di!erent source image types and qualities. The state-of-the-art techniques are able to adapt to some degree of errors in a de"ned category, and focus on few image types. In images needing multi-thresholding, the problem seems to be ever harder to solve, since the complexity of image contents, including textual documents has increased rapidly. Some document directed binarization algorithms have been developed. O'Gorman [3] proposes a global approach calculated from a measure of local connectivity information. The thresholds are found at the intensity levels aiming to preserve the connectivity of regions. Liu et al. [4] propose a method for document image binarization focused on noisy and complex background problems. They use grey-scale and run-length histogram analysis in a method called &object attribute thresholding'. It identi"es a set of global thresholds using global techniques which is used for "nal threshold selection utilizing local features. Yang et al.'s [5] thresholding algorithm uses a statistical measurement, called &largest static state di!erence'. The method aims to track changes in the statistical signal pattern, dividing the level changes to static or transient according to a grey-level variation. The threshold value is calculated according to static and transient properties separately at each pixel. Stroke connectivity preservation issues in textual images are examined by Chang et al. in Ref. [6]. They propose an algorithm that uses two di!erent components: the background noise elimination using grey-level histogram equalization and enhancement of grey-levels of characters in the neighbourhood using an edge image composition technique. The &binary partitioning' is made according to a smoothed and equalized histogram information calculated in "ve di!erent steps. Pavlidis [7] presents a technique based on the observation that after blurring a bi-level image, the intensity of original pixels is related with the sign of the curvature of the pixels of the blurred image. This property is used to construct the threshold selection of partial histograms in locations where the curvature is signi"cant. Rosenfeld and Smith [8] presented a global thresholding algorithm to deal with noise problem using an

iterative probabilistic model when separating background and object pixels. A relaxation process was used to reduce errors by "rst classifying pixels probabilistically and adjusting their probabilities using the neighbouring pixels. This process is "nally iterated leading to threshold selection, where the probabilities of the background and the object pixels are increased and will be ruled accordingly to non-object and object pixels. The thresholding algorithm by Perez and Gonzalez [9] was designed to manage situations where imperfect illumination occurs in an image. The bimodal re#ectance distribution is utilized to present grey-scale with two components: re#ectance r and illumination i, used also in homomorphic "ltering. The algorithm is based on the model of Taylor series expansion and uses no a priori knowledge of the image. The illumination is assumed to be relatively smooth, whereas the re#ectance component is used to track down changes. The threshold value is chosen from the probabilistic criterion of occurring twodimensional threshold selection function. This can be calculated in raster-scan fashion. The illumination problem is emphasized in the thresholding algorithm, called &edge level thresholding', presented by Parker et al. in Ref. [10]. Their approach uses the principles that objects provide high spatial frequency while illumination consist mainly of low spatial frequencies. The algorithm "rst identi"es objects using Shen}Castan edge detector. The grey-levels are then examined in small windows for "nding highest and lowest values that indicate object and background. The average of these values are used to determine the threshold. The selected value is then "tted to all pixels as a surface leading the values above to be judged as a part of an object and a value lower than threshold belongs to background. Shapiro et al. [11] introduce a global thresholding scheme, where the independency is stressed in the object/background areas ratio, intensity transition slope, object/background shape and noise-insensitivity. The threshold selection is done by choosing a value that maximizes the global non-homogeneity. This is obtained as an integral of weighted local deviations, where the weight function assign higher standard weight deviation in case of background/object transitions than in homogeneous areas. Pikaz and Averbuch [12] propose an algorithm to perform thresholding for scenes containing distinct

228


objects. The sequence of graphs is constructed using the size of connected objects in pixels as a classi"er. The threshold selection is gained from calculating stable states on the graph. The algorithm can be adapted to select multi-level thresholds by selecting highest stable state candidate in each level. Henstock and Chelberg [13] propose a statistical model-based threshold selection. The weighted sum of two gamma densities, used for decreasing the computational load instead of normal distributions, are "tted to the sum of edge and non-edge density functions using a "ve-parameter model. The parameters are estimated using an expectation maximization-style two-step algorithm. The "tted weighted densities separate the edge pixels from non-edge pixels of intensity images. The enhanced speed entropic threshold selection algorithm is proposed in Ref. [14] by Chen et al. They reduce the image grey-scale levels by quantization and produce a global threshold candidate vector from quantized image. The "nal threshold selection is estimated only from the reduced image using the candidate vector. The reduction in computational complexity is in the order of magnitude of O(G8@3) of the number of grey-scale values, using Onotation. The quality of binarization is su$cient for preliminary image segmentation purposes. Yanowitz and Bruckstein [15] proposed an image segmentation algorithm based on adaptive binarization, where di!erent image quality problems were taken into consideration. Their algorithm aimed to separate objects in illuminated or degraded conditions. The technique uses variating thresholds, whose values are judged by edge analysis processing combined with grey-level information and construction of interpolated threshold surface. The image is then segmented using the gained threshold surface by identifying the objects by post-validation. The authors indicated that validation can be performed with most of the segmentation methods. 1.2. Our approach For document image binarization, we propose a new method that "rst performs a rapid classi"cation of the local contents of a page to background, pictures and text. Two di!erent approaches are then applied to de"ne a threshold for each pixel: a soft decision method (SDM) for background and pictures, and a specialized text bi-

narization method (TBM) for textual and linedrawing areas. The SDM includes noise "ltering and signal tracking capabilities, while the TBM is used to separate text components from background in bad conditions, caused by uneven (il)lumination or noise. Finally, the outcome of these algorithms are combined. Utilizing proper ways to benchmark the algorithm results against ground-truth and other measures is important for guiding the algorithm selection process and directions that future research should take. A well-de"ned performance evaluation shows which capabilities of the algorithm still need re"nement and which capabilities are su$cient for a given situation. The result of benchmarking o!ers information of the suitability of the technique to certain image domains and quality. However, it is not easy to see the algorithm quality directly from a set of performance values. In this paper we use a goal-directed evaluation process with specially developed document image binarization metrics and measures for comparing the results against a number of well-known and well-performed techniques in the literature [16].

2. Overview of the binarization technique Our binarization technique is aimed to be used as a "rst stage in various document analysis, processing and retrieval tasks. Therefore, the special document characteristics, like textual properties, graphics, line-drawings and complex mixtures of their layout-semantics should be included in the requirements. On the other hand, the technique should be simple while taking all the document analysis demands into consideration. Fig. 4 presents the general approach of the binarization processing #ow. Since typical document segmentation and labelling for content analysis is out of question in this phase, we use a rapid hybrid switch that dispatches the small, resolution adapted windows to textual (1) and non-textual (2) threshold evaluation techniques. The switch was developed to cover most generic appearances of typical document layout types and can easily be modi"ed for others as well. The threshold evaluation techniques are adapted to textual and non-textual area properties, with the special tolerance and detection to di!erent basic defect types that are usually introduced to images. The outcome of these techniques represent a threshold value

Fig. 4. Overview of the binarization algorithm.


229

Fig. 5. Interpolation options for binarization computation.

proposed for each pixel, or every nth pixel, decided by the user. These values are used to collect the "nal outcome of the binarization by a threshold control module. The technique also enables the utilization of multi-thresholds region by region of globally, if desired.

3. Adaptive binarization The document image contains di!erent surface (texture) types that can be divided into uniform, di!erentiating and transiently changing. The texture contained in pictures and background can usually be classi"ed to uniform or di!erentiating categories, while the text, line drawings, etc. have more transient properties by nature. Our approach is to analyse the local document image surface in order to decide on the binarization method needed (Fig. 4). During this decision, a &hybrid switching' module selects one of two specialized binarization algorithms to be applied to the region. The goal of the binarization algorithms is to produce an optimal threshold value for each pixel. A fast option is to compute "rst a threshold for every nth pixel and then use interpolation for the rest of the pixels (Fig. 5). The binarization method can also be set to bypass the hybrid switch phase. Then the user can choose which algorithm is selected for thresholding. All other modules function in the same way as in hybrid conditions. The following subsection describes the region type and switching algorithms. The two di!erent binarization algorithms are then discussed in detail. The "nal binarization is performed using the proposed threshold values. This process is depicted in the last subsection.

then scaled between 0 and 1. Using the limits of 10, 15 and 30% of scaled values, the transient di!erence property is de"ned as &uniform', &near-uniform', &di!ering' or &transient'. This coarse division is made according to average homogeneity on the surface. According to these labels, a vote is given to corresponding binarization method that is to be used in a window. The labels &uniform' and &near-uniform' correspond to background and &scene' pictures, and give votes to the SDM. The labels &di!ering' and &transient' give their votes to the TBM method. Selection of a binarization algorithm is then performed as following example rules (1, 2) show: 1. If the average is high and a global histogram peak is in the same quarter of the histogram and transient di!erence is transient, then use SDM. 2. If the average is medium and a global histogram peak is not in the same quarter of the histogram and transient di!erence is uniform, then use TBM. An example result of image partitioning is shown in Fig. 6. The white regions are guided to the SDM algorithm, while the grey regions are binarized with the TBM algorithm. 3.2. Binarization of non-textual components As in soft control applications, our algorithm "rst analyses the window surface by calculating descriptive characteristics. Then, the soft control algorithm is applied to every nth pixel (Fig. 5). The result is a local threshold based on local region characteristics. To ensure local adaptivity of threshold selection, we use two di!erent types of locally calculated features:

3.1. Region analysis and switching Threshold computation is preceded by the selection of the proper binarization method based on an analysis of local image properties. First, the document image is tiled to equal sized rectangular windows of 10}20 pixels wide, corresponding to the resolution that linearly varies between '75 and (300 dpi. Two simple features are then computed for each window; these results are used to select the method. The "rst feature is simply the average grey value of a window. The second feature, &transient di!erence', measures local changes in contrast (Eq. (4)). The difference values are accumulated in each subwindow and

Fig. 6. Example of region (SDM/TBM) selection.

partitioning

for

algorithm


230

&weighted bound' and &threshold di!erence'. The membership function issues, soft decision rules and defuzzi"cation algorithm are presented in the following paragraphs. 3.2.1. Weighted bound calculation Histogram-based analysis schemes and features are often used in binarization methods. In document analysis the histogram is very useful for detecting and di!erentiating domains in physical and logical analysis. We use a new approach developed for local detection and weighting of bounds in grey-scale image texture. A new feature called weighted bound (= ) is introduced and b utilized in the soft control algorithm. The = is used for b characterization of local pixel value pro"les by tracking low, medium and high pixels in a small area. In a given surface area of n]n pixels, where n is a window width gained from the non-overlapping regions analysis tile size (see Section 3.1), three di!erent measures are calculated. The values are collected in a two-dimensional table used to weight and simplify the three envelope curves in soft control membership functions. The measures are minimum, medium and maximum averages given in Eqs. (1)}(3). Minimum average, A .*/ 100@n min (P(i, j)) 100@n A " + , .*/ 100/n k/0

(1)

where P(i, j) is the document image region, and i is the width, and j is the height. n is the static number gained from average window size (see Section 3.1).

Medium average, A

.%$ 100@n med (P(i, j)) 100@n . (2) A " + .%$ 100/n k/0 Maximum average, A .!9 100@n max (P(i, j)) 100@n A " + . (3) .!9 100/n k/0 These values are stored in an n]n]3 table, called a weighted average table (WAT). Using Eqs. (1)}(3), three di!erent histograms are formed where the values are added to their respective bin values (value"bin index). These histograms are then separately partitioned to ten horizontal and three vertical sections, where the number of peaks from histograms are calculated to each section according to sectioning limits. The horizontal borders are set between bins 0 and 255 with a formula int((256/10)Hm), where m"1, 2,2, 9. The number of borders was set to ten. Also a smaller number could be selected, but the penalty is that the original histogram is aliased more. Ten borders equals 25 bins of grey-scale. The two vertical borders are set between 0 and maximum, representing the number of votes calculated for each horizontal bin so that the limits are set to 80% of maximum number of votes and to 40% of the maximum number of votes, respectively. These limits are set according to the tests performed with a large set of images. The higher limit is relatively insensitive to $10% change. Lowering the lower limit brings more votes to medium peak calculation, thus enhancing the envelope curve in bins where a medium peak appears. After the peaks are calculated in a 3]10 table, the weighting is performed (Fig. 7). The result is a = b

Fig. 7. An example of = membership function calculation using A histogram. b .*/


envelope curve that is used in the soft decision process. The three = curves, calculated from A , A and A b .*/ .%$ .!9 are used as membership functions. 3.2.2. Transient diwerence calculation The transient di!erence is aimed at extracting the average amount of variations occurring between the neighbouring pixels (contrast di!erence) in an n]n area, i.e. to follow local surface changes. The di!erences between adjacent pixels are accumulated. The transient di!erence (¹D) of the horizontal and vertical adjacent pixel values is calculated and accumulated. The gained value is then scaled between 0}1 (Eq. (4)). ¸ represents the number of grey-levels in the image. (+n +n D2P(i, j)![P(i!1, j)#P(i, j!1)]D) ¹D" i/1 j/1 . (¸n)2 (4) The ¹D value is used in soft decision making to expose uniform, di!erential and transient area types when calculating the control value for threshold selection. 3.2.3. Membership function generation Two di!erent membership functions are used according to the extracted feature values for a given pixel: weighted bound (= ) and transient di!erence (¹D ). The b m "rst one is calculated dynamically from the image. The transient di!erence uses prede"ned membership functions. Fig. 8 depicts these functions using the ideal functions as = and the actual membership functions b for ¹D . m

231

3.2.4. Soft decision rules and defuzzixcation In the soft decision process, we use nine di!erent rules derived from the feature analysis and membership management. For = these are (LOW, MIDDLE, HIGH), b denoting the local histogram properties. For ¹D we use m (UNIFORM, DIFFERING, TRANSIENT), describing the local region property. The rule set is shown in Fig. 9. As in soft control problems, the rules are expressed with clauses, for example: If = is SP(i, j)T and ¹D is S¹D(i, j)T b m then ¹ (i, j)"S0, 255T. c The current rule set is designed for pictorial and background-type image regions. Using this set the noise and most illumination defects can be adaptively corrected in the processed areas. For defuzzi"cation we use Mamdani's method [17]. The result of the defuzzi"cation is a unique threshold value for each pixel n.

3.3. Binarization of textual components For text binarization we use a modi"ed version of Niblack's algorithm [18]. The idea of Niblack's method is to vary the threshold over the image, based on the local mean, m, and local standard deviation, s, computed in a small neighbourhood of each pixel. A threshold for each pixel is computed from ¹"m#kHs, where k is a user de"ned parameter and gets negative values. This method

Fig. 8. Input and output membership functions: = (ideal), ¹D and ¹ . b m c


232

threshold line that is adapted to original degraded document image. 3.4. Interpolative threshold selection

Fig. 9. Example of soft decision rules for threshold candidate ¹ (i, j). c

does not work well for cases in which the background contains light texture as the grey values of these unwanted details easily exceed threshold values. This results in costly postprocessing as demonstrated in Ref. [19]. In our modi"cation, a threshold is computed with the dynamic range of standard deviation, R. Furthermore, the local mean is utilized to multiply terms R and a "xed value k. This has the e!ect of amplifying the contribution of standard deviation in an adaptive manner. Consider, for example, a dark text on light dirty-looking background (e.g., stains in a bad copy), Fig. 2. The m-coef"cient decreases the threshold value in background areas. This e$ciently removes the e!ect of stains in a thresholded image. In our experiments, we used R"128 with 8-bit gray level images and k"0.5 to obtain good results. The algorithm is not too sensitive to the value of parameter k. Eq. (5) presents the textual binarization formula.

C

¹(x, y)"m(x, y) ) 1#k )

A

BD

s(x, y) !1 R

,

After thresholding guided by the surface type, the "nal thresholds are calculated for background, textual, graphics and line drawing regions. A fast option is to compute "rst a threshold for every nth pixel and then using interpolation for the rest of the pixels. The control algorithm has two modes depending on the value of n. If n"1, the threshold values gained from SDM and TBM algorithms are combined directly. If n'1, threshold values for non-base pixels are calculated using the surrounding threshold values. We have two options to calculate the non-base pixel thresholds: bilinear interpolation and simple averaging. In the interpolation method, the threshold value for a non-base pixel is gained by computing the surrounding base pixels distance to the current one, and using these values as weights, Fig. 11a. This approach gives a more precise, weighted threshold value for each pixel. In the simple averaging method, the average of the surrounding four n pixel threshold candidate values is calculated and used as a "nal threshold for each non-base pixel between the selected base pixels, Fig. 11b. This approach is used to lower the computational load and is suitable for most images, especially for those with random noise and n larger than "ve pixels.

(5)

where m(x, y) and s(x, y) are as in Niblack's formula. R is the dynamic range of standard deviation, and the parameter k gets positive values. Fig. 10 shows an example

4. Experiments The proposed binarization algorithm was tested with the benchmarking technique and various scenarios

Fig. 10. Example of threshold candidate selection of an example scanline.

Fig. 11. Two interpolation choices for threshold selection of non-base pixels.


233

Fig. 12. Visual and numeric results on the comparison algorithms applied to illuminated, textual images.

against several known binarization techniques in the literature [18,20}22]. Using the environment factors (such as di!erent degradations) and available document and test image databases the algorithm results were evaluated and benchmarked against each other, against the ground-truth knowledge by visual and benchmark event(s) evaluation processes. The focus was set on documents with textual content and on multi-content documents, i.e. documents having text, graphics, linedrawings and halftone. The test images were selected from a special database of document image categories, comprising over 1000 categorized document images (e.g. article, letter, memo, fax, journal, scienti"c, map, advertisement, etc.) [23]. The numerical test and results presented were gained using binarization metrics emphasizing the performance in textual image region binarization. Fig. 12 presents an example benchmarking scene performed to a database of 15 textual document images having illumination. Visual results to a sample input image having 20% of centered illumination defect, an example of a ground-truth image map and the results of the proposed and comparison binarization algorithms. The results show good behaviour of Sauvola's, Niblack's and Eikvil's algorithms, when the limit is set to 80% performance, i.e. the limit where the OCR performance drop is less than 10% using Caere Omnipage OCR package [24]. Bernsen su!ered of noise that was introduced to binarized result image, while the Eikvil's threshold ruled some of the darkest areas belong to object pixels. Parker's algorithm adapted poorly to even small changes in lumination, but had su$cient results with relatively &clean' grey-scale document images. The visual tests performed for a synthetic test image database were based on ranking according to di!erent

objectives set for these types of images. The purpose of the synthetic image database is to allow visual analysis of the nature and behaviour of the benchmarking technique in a di!erent kind of situation, e.g. in edge preservation, the object uniformity preservation, in changing/varying background, etc. This is aimed to aid the suitability selection of di!erent algorithm to di!ering environmental conditions in terms of adaptability to changes, shape management, object preservation, homogeneousness of region preservation, and so on. An example of the visual results on synthetic images is shown in Fig. 13. Fig. 13 shows visually the results of our, and comparison, algorithms applied to synthetic grey-scale images having di!erent/di!ering kind of background(s), object(s), line(s), directions and shapes complying with certain simple test setting rules. As the input grey-scale images were synthetically generated, a set of groundtruth images were generated focusing in di!erent areas of interest in measuring the algorithm performance and behaviour. Therefore, the benchmark results are dependent on the selection of the ground-truth set used, i.e. the target performance group the algorithm behaviour. For example, the ground-truth criteria of object uniformity and edge preservation were tested using ground-truth image in Fig. 13a. The object edge and background/ object uniformity was used as a weight criteria, where the Euclidean distance was used as a distance measure between the result and the ground-truth pixel maps. Fig. 13b shows a situation, where the synthetic image has uniformly gliding background from white to black, and thin lines, whose grey-scale value glides on the opposite direction from the background. The test evaluation criterium was set on di!erentiating lines from background and uniformity of the background. Since the results are highly dependent on the target aims of the binarization,

234


Fig. 13. Results on the comparison algorithms applied to the synthetic graphical images.

Fig. 14. Overall benchmarked binarization and example pro"le results on &text only' document database.

the results are presented also visually. By using the criteria of the uniformity and object shape preservation the proposed algorithm behaves robustly compared to other techniques. Since most of the pixels in synthetic images are judged by the soft control method, the threshold between objects and non-object candidates seems very clear. Fig. 14 shows benchmarking results performed with the textual image database with small amounts of clean and mixed illumination and noise types. An example performance pro"le to noise degradation component is shown for all the comparison algorithms. The degree of noise degradation presents the percentage of Gaussian and random noise introduced in the textual image, and the performance using combined pixel map and OCR metrics with equal weight factors. The performance of the proposed and comparison algorithms, excluding Parker's, seems to be su$cient up till 20% noise penetration. The performance pro"le clearly shows that the performance of the comparison algorithms drops between 20 and 30% penetration, while the proposed algorithm tolerated with severe noise, up to 45% having 80% threshold limit for acceptable value. Fig. 15 shows the overall results of the proposed and comparison algorithms with various document categories performed to a large database of document images. The test images comprise simple textual documents with

and without degradation types and degrees, documents with mixed textual and graphical properties, where the bene"ts of the hybrid approach of the proposed algorithm can be clearly seen. The methods of Eikvil and Niblack performed best against the proposed algorithm, but they still su!ered of poor adaptation to various degradation types and, for example, the font size used in the textual parts was combined with the characters. The Bernsen algorithm shows good results on clean document and did tolerate small amount of one defect type. When the degradation was higher, the algorithm's performance decreased rapidly both in visual and numerical evaluation. Parker's algorithm shows su$cient results with clean document images, but the result quality dropped with even small introduction of document with any defect type. The algorithm execution times were not measured in this comparison, where only the quality of the result was benchmarked against the metrics in a weighted (textual, graphics, character) process. The computing times for all the evaluated algorithms were tolerable, for example for utilization as a preprocessing step in optical character recognition engines. One question in performing the benchmarking is the arrangement of parametrization. The proposed algorithm had no parameters to set during testing, while Niblack had one, and Bernsen two, Eikvil


235

Fig. 15. Overall benchmarked binarization results on textual document database.

used "rst Otsu's technique with one parameter and their postprocessing with one parameter, Parker's algorithm had four parameters to set. Each algorithm with parameters that needed manual tuning was computed with di!erent parameters, whose result were evaluated and the best was selected to "nal comparison presented in this paper. When the higher adaptation is required from the algorithm, the number of manually tunable parameters should not exceed two, otherwise the amount of manual work increases too much and cause instability where automated preprocessing is required. The overall results show good evaluated performance to the proposed, Niblack's and Eikvil's algorithms. The difference if these approaches lies in overall adaptability, the need for manual tunability, target document category domain and environment, where the algorithm is utilized, and "nally the threshold performance set for the binarization process. In the latter case the proposed and Niblack's algorithms performance and adaptivity was highest in all test categories in graphical and textual cases.

5. Conclusions Document image binarization is an important basic task needed in most document analysis systems. The quality of binarization result a!ects to subsequent processing by o!ering pre-segmented objects in precise form (object/non-object). In this paper we proposed a new technique to document image binarization, using hybrid approach and taking document region class properties into consideration. Our technique is aimed at generic document types coping also with severe cases of di!erent types of degradation. The result of the quality validation (i.e. benchmarking against other algorithms and ground truth) is an important part of the algorithm development process. The proposed algorithm went over large tests utilizing test image databases having textual, pictorial and synthetically generated document images with

ground-truths and degradations. The results show especially good adaptation into di!erent defect types such as illumination, noise and resolution changes. The algorithm showed robust behaviour in most, even severe, situations in degradation and performed well against the comparison techniques.

6. Summary This paper presents a new algorithm for document image binarization using an adaptive approach to manage di!erent situations in an image. The proposed technique uses rapid image surface analysis for algorithm selection and adaptation according to document contents. The contents is used to select the algorithm type and need for parametrization, if any, and to compute and propose the threshold value for each or every nth pixel (interpolative approach). The document content is used to guide the binarization process: a pictorial content is subjected to a di!erent type of analysis than a textual content. The degradations, such as illumination and noise, are managed within each algorithm structure to e!ectively "lter out the imperfections. The results of the thresholding processes are combined to a binarized image that can either use a fast option, i.e. to compute binarization for every nth pixel and interpolate the threshold value for the in-between pixels, or a pixel by pixel option that computes a threshold value for each pixel separately. The tests were run on a large database of document images having 15 di!erent document types and a number of representative images of each type. Each image was processed with the presence of various amount of di!erent degradation to evaluate the e$ciency of the proposed algorithm. The results were compared to those obtained with some of the best-known algorithms in the literature. The proposed algorithm outperformed its competitors clearly and behaved robustly in di$cult degradation cases with di!erent document types.

236


Acknowledgements The support from the Academy of Finland and Technology Development Centre is gratefully acknowledged. We also thank Dr. Tapio Seppanen and Mr. Sami Nieminen for their contributions.

References [1] J. Sauvola, M. PietikaK inen, Page segmentation and classi"cation using fast feature extraction and connectivity analysis, International Conference on Document Analysis and Recognition, ICDAR '95, Montreal, Canada, 1995, pp. 1127}1131. [2] H. Baird, Document image defect models, Proceedings of the IAPR Workshop on Syntactic and Structural Pattern Recognition, 1990, pp. 38}46. [3] L. O'Gorman, Binarization and multithresholding of document images using connectivity, CVGIP: Graph. Models Image Processing 56 (6) (1994) 496}506. [4] Y. Liu, R. Fenrich, S.N. Srihari, An object attribute thresholding algorithm for document image binarization, International Conference on Document Analysis and Recognition, ICDAR '93, Japan, 1993, pp. 278}281. [5] J. Yang, Y. Chen, W. Hsu, Adaptive thresholding algorithm and its hardware implementation, Pattern Recognition Lett. 15 (2) (1994) 141}150. [6] M. Chang, S. Kang, W. Rho, H. Kim, D, Kim, Improved binarization algorithm for document image by histogram and edge detection, International Conference for Document Analysis and Recognition ICDAR '95, Montreal, Canada, 1995, pp. 636}643. [7] T. Pavlidis, Threshold selection using second derivatives of the gray scale image, International Conference on Document Analysis and Recognition, ICDAR '93, Japan, 1993, pp. 274}277. [8] A. Rosenfeld, R.C. Smith, Thresholding using relaxation, IEEE Trans. Pattern Anal. Mach. Intell. PAMI-3 (5) (1981) 598}606. [9] A. Perez, R.C. Gonzalez, An iterative thresholding algorithm for image segmentation, IEEE Trans. Pattern Anal. Mach. Intell. PAMI-9 (6) (1987) 742}751.

[10] J.R. Parker, C. Jennings, A.G. Salkauskas, Thresholding using an illumination model, ICDAR '93, Japan, 1993, pp. 270}273. [11] V.A. Shapiro, P.K. Veleva, V.S. Sgurev, An adaptive method for image thresholding, Proceedings of the 11th KPR, 1992, pp. 696}699. [12] A. Pikaz, A. Averbuch, Digital image thresholding, based on topological stable-state, Pattern Recognition 29 (5) (1996) 829}843. [13] P.V. Henstock, D.M. Chelberg, Automatic gradient threshold determination for edge detection, IEEE Trans. Image Processing 5 (5) (1996) 784}787. [14] W. Chen, C. Wen, C. Yang, A fast two-dimensional entropic thresholding algorithm, Pattern Recognition 27 (7) (1994) 885}893. [15] S.D. Yanowitz, A.M. Bruckstein, A new method for image segmentation, CVGIP 46 (1989) 82}95. [16] S. Nieminen, J. Sauvola, T. SeppaK nen, M. PietikaK inen, A benchmarking system for document analysis algorithms, Proc. SPIE 3305 Document Recognition V 3305 (1998) 100}111. [17] S.T. Welstead, Neural Network and Fuzzy Logic Applications in C/C##, Wiley, New York, 1994, p. 494. [18] W. Niblack, An Introduction to Image Processing, Prentice-Hall, Englewood Cli!s, NJ, 1986, pp. 115}116. [19] O.D. Trier, A.K. Jain, Goal-directed evaluation of binarization methods, IEEE Trans. Pattern Anal. Mach. Intell. 17 (12) (1995) 1191}1201. [20] L. Eikvil, T. Taxt, K. Moen, A fast adaptive method for binarization of document images, International Conference on Document Analysis and Recognition, ICDAR '91, France, 1991, pp. 435}443. [21] J. Bernsen, Dynamic thresholding of grey-level images, Proceedings of the Eighth ICPR, 1986, pp. 1251}1255. [22] J. Parker, Gray level thresholding on badly illuminated images, IEEE Trans. Pattern Anal. Mach. Intell. 13 (8) (1991) 813}819. [23] J. Sauvola, S. Haapakoski, H. Kauniskangas, T. SeppaK nen, M. PietikaK inen, D. Doermann, A distributed management system for testing document image analysis algorithms, 4th ICDAR, Germany, 1997, pp. 989}995. [24] Caere Ominpage OCR, Users Manual, Caere Corp., 1997.

About the Author*JAAKKO SAUVOLA is a Professor and Director of the Media Team research group in the University of Oulu, Finland, and a member of the a$liated faculty at the LAMP Laboratory, Center for Automation Research, University of Maryland, USA. Dr. Sauvola is also a Research Manager in Nokia Telecommunications, where his responsibilities cover value adding telephony services. Dr. Sauvola is a member of several scienti"c committees and programs. His research interests include computer-telephony integration, media analysis, mobile multimedia, media telephony and content-based retrieval systems. About the Author*MATTI PIETIKAG INEN received his Doctor of Technology degree in Electrical Engineering from the University of Oulu, Finland, in 1982. From 1980 to 1981 and from 1984 to 1985 he was a visiting researcher in the Computer Vision Laboratory of the University of Maryland, USA. Currently, he is a Professor of Information Technology, Scienti"c Director of Infotech Oulu research center, and Director of Machine Vision and Media Processing Group at the University of Oulu. His research interests cover various aspects of image analysis and machine vision, including texture analysis, color machine vision and document analysis. His research has been widely published in journals, books and conferences. He was the editor (with L.F. Pau) of the book `Machine Vision for Advanced Productiona, published by World Scienti"c in 1996. Prof. PietikaK inen is one of the founding Fellows of the International Association for Pattern Recognition (IAPR) and a Senior Member of IEEE, and serves as Member of the Governing Board of IAPR. He also serves on program committees of several international conferences.


Adaptive window method with sizing vectors for reliable correlation-based target tracking Sung-Il Chien*, Si-Hun Sung School of Electronic and Electrical Engineering, Kyungpook National University, 1370 Sankyuk-dong, puk-gu, Taegu 702-701, South Korea Received 5 August 1998

Abstract We propose an adaptive window method that can provide a tracker with a tight reference window by adaptively adjusting its window size independently into all four side directions for enhancing the reliability of correlation-based image tracking in complex cluttered environments. When the size and shape of a moving object changes in an image, a correlator often accumulates walk-o! error. A success of correlation-based tracking depends largely on choosing the suitable window size and position and thus transferring the proper reference image template to the next frame. We generate sizing vectors from the corners and sides, and then decompose the sizing vector from the corner into two corresponding sides. Since our tracker is capable of adjusting a reference image size more properly, stable tracking has been achieved minimizing the in#uence of complex background and clutters. We tested the performance of our method using 39 arti"cial image sequences made of 4260 images and 45 real image sequences made of more than 3400 images, and had the satisfactory results for most of them. ( 1999 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Adaptive window; Sizing vector; Error correction; Target tracking; Correlation; Motion detection; Tracking feature extraction

1. Introduction Correlation-based tracking [1}5] acquires a reference template from the previous frame called a reference image and searches for the most similar area to estimate the target position in the current frame. Although the correlator is said to be robust against the cluttered noise, its real application has some limitations, too. Usually, it is desirable that searching area should be chosen to be small due to its large computation involved. Another problem is its tendency to accumulate walk-o! error especially when the object of interest is changing in size, shape, or orientation from frame to frame.

* Corresponding author. Tel.: #82-53-950-5545; fax: #8253-950-5505 E-mail address: [email protected] (S-I. Chien)

If walk-o! error is accumulated beyond a certain critical point, correlation-based tracking often fails. It is quite important that the size and position of a window should be determined precisely to guarantee that a proper reference image can be transferred to the next frame. In order to increase correlation reliability, the transferred reference image is usually desired to have a high occupancy rate of an object, which means that the window encloses the object properly. For this, it is quite desirable that the window should adjust its size as circumstances of the object have been under change. A concept of an adaptive window could be found also in stereo matching. In the case of stereo matching, the disparity boundaries become sharp for the smaller window but the computed disparity becomes usually noisy. The larger window means that the computed disparity becomes clean but the disparity boundaries can be blurred. Kanade and Okutomi [6] determined the adaptive window size using local intensity and disparity patterns


238

S-I. Chien, S-H. Sung / Pattern Recognition 33 (2000) 237}249

to minimize uncertainty in the disparity computed at each point. Lotti and Giraudon [7] made four noncentered adaptive windows associated to each image point in the thresholded edge image. In correlation-based tracking, the adaptive window studies based on estimating an object size are not much reported in the technical literature. To automatically adapt the reference area, Hughes and Moy [1] designed the edge walking algorithm for the boundary detection to operate on a segmented binary image. This algorithm scans the binary image in a raster fashion looking for illuminated pixels. Once an illuminated pixel has been found, the edge walking algorithm searches for another illuminated pixels connected to the initial pixel. They used a boundary detection method for estimating the size of an object and thus determining the size of window that would enclose the object. Similarly, Montera et al. [2] determined an object region through expanding from the inner point of an object to the outer in the image. To yield the boundary of an object, they searched for the areas where pixel values vary from above the threshold to below the threshold. However, both methods could be di$cult to be applied to an object having internal edges in non-homogeneous cluttered environment. Chodos et al. [3] developed a window algorithm embedded in a tracker, which is able to adjust the track gate size in four directions using the gradient function formed from the correlation weighting function. However, we expect this method to be unsuitable for being applied to a large object moving fast near an observer since it is able to adjust only by one pixel in each direction. An adaptive window without a proper sizing mechanism can hardly accommodate itself to environment variations when a window size is much larger or smaller than an object size or an object size is abruptly changed. To adjust a window size more rapidly and e$ciently, we propose the adaptive window method which is able to control the size continuously with four directional sizing vectors in a complex background. Our method introduces eight sizing vectors estimated from eight districts (four side districts and four corner districts), and decomposes each sizing vector from a corner district into the two orthogonal vectors to estimate the "nal sizing vectors in four side directions.

In the proposed window method, positive di!erence of edges (PDOE) image rather than the conventional edge image is adopted as a basic correlation feature, since its use is found to be quite useful in compressing background components. The detailed description of the PDOE is beyond the viewpoint of this paper. Thus we brie#y introduce the PDOE and the applied correlator in Section 2. In Section 3, we detail the structure and procedure of the proposed window method. Then in Section 4, we provide experimental results using arti"cial and real image sequences. Finally, we include a conclusion in Section 5.

2. Applied image tracker architecture The image tracking block diagram we propose is described in Fig. 1. The overall system is largely divided into the PDOE, the correlator for the main tracking process, the proposed adaptive window block, and the recursive updating loop. First, we acquire the background-reduced image using the PDOE as a tracking feature and then track the object by applying the correlator. Finally, the adaptive window method determines a reference image region tightly enclosing the object to be used in the next frame. For the consecutive tracking, the size and position of the reference image will be suitably updated. 2.1. Positive diwerence of edges (PDOE) for feature extraction Conceptually, the PDOE has spatial and temporal motion components as shown in Fig. 2. We can represent the PDOE image PDOE (x, y) in the nth frame such that n D (x, y)"E (x, y)!E (x, y), n n n~1

(1)

G

(2)

D (x, y) if D (x, y)'0, n PDOE (x, y)" n n 0 otherwise,

where E (x, y) is an edge component at a point (x, y) and n E (x, y) is obtained from the previous frame. We use n~1 the Sobel operator to detect an edge E(x, y). Conventionally, di!erence methods in image processing use an

Fig. 1. Overall tracking block diagram.


Fig. 2. Block diagram of the PDOE.

absolute value of D(x, y). However, the PDOE removes negative components in D(x, y). As shown in Fig. 3, the PDOE can extract single edge component for the target while an absolute di!erence method produces double moving components for a moving target. This is useful for removing the background components and detecting motion components more stably. 2.2. Correlator Correlation-based tracking searches for the best matched point in the current frame to the reference image acquired from the previous frame. Let g(x, y) be a reference image of size m]n and f (x, y) be an image of size M]N to be searched, and we assume that m)M and

239

Fig. 4. Correlation layout for f (x, y) and g(x, y) at point (i, j).

n)N. The elements of the correlation surface R between f (x, y) and g(x, y) in Fig. 4 are given by R(i, j)"+ + f (x, y)g(x!i, y!j), x y

(3)

where i"0, 1, 2 ,2, M!1, j"0, 1, 2 ,2, N!1 and the summation is taken over the image region where f and g overlap. Thus, the matched point (iH, jH) is estimated by (iH, jH)"arg max (R(i, j)). i,j

(4)

This point indicates the position where g(x, y) best matches f (x, y).

Fig. 3. Moving component extraction for two arti"cial noisy images and two real images: (a) previous frame, (b) current frame, (c) absolute di!erence of edge images, and (d) PDOE image.

240


Fig. 5. Overall block diagram of the adaptive window method using sizing vectors.

A correlator is regarded as more robust against complex cluttered environment than a centroid-based tracker based on a center of geometric gravity from the tracking window. However, the walk-o! error often occurs when the situation for a target changes rapidly and a general correlator easily accumulates such error in the process of tracking. Furthermore, it is quite di$cult for a human operator to enclose the target tightly in starting an initial reference window for correlation-based tracking. Therefore, to reduce the error of a correlator, adjusting the size of a reference window adaptively is quite demanded.

3. Proposed adaptive window method Fig. 5 describes an overall block diagram of our proposed adaptive window scheme. The feature set used for adjusting an adaptive window is the PDOE image previously described. Our method needs mainly several steps: the adaptive window setting, the sizing vector estimation from corners and sides, and the window size determination and the reference center point relocation. Here, the adaptive window can expand or shrink independently in four side directions, each side having the sizing magnitude and direction derived from the "nal sizing vector S. In Fig. 5, we present several sizing parameters: S is a side sizing vector estimated from the side, S S is a corner sizing vector from the corner, S and C CH S are the orthogonally decomposed sizing vectors conCV tributed by corner-to-side conversion.

used as a criterion for determining whether pixels in the middle region are part of the object or not. Thus in order to extract more accurate information near the object boundary, the area of the middle region is "xed to be smaller than those of the others. Second, eight overlapped districts consisting of four side districts and four corner districts are then de"ned. The side sizing vector S will act as a dominant paraS meter in "nally determining a sizing direction of the window. A corner district will evaluate the edge distribution to provide a relevant corner sizing vector S that will C be decomposed to the two corresponding side sizing vectors: S and S . For this, the area of a corner is CH CV designed to be similar to that of a side district. Finally, each district has been further divided into three subdistricts: inner (I), middle (M), and outer zones (O), which are not overlapped. Actually, the statistical information from these zones will be used to identify various situations leading to the determination of the suitable sizing vector, which will be detailed in the following Sections. Fig. 6 describes a layout of the left side district of four side districts and a layout of the top right corner district of four corner districts. The remaining side and corner districts will be de"ned similarly.

3.1. Adaptive window setting For adaptively controlling the reference window and relocating the center point, we now construct three regions: inner, middle, and outer regions. First, we de"ne the outside boundary of the middle region as the window boundary given by the previous tracking stage. We design the inner region needed for extracting information within an object boundary and the extended outer region for obtaining useful clues about background information. The extracted information from a middle region will be

Fig. 6. The layout of the left side district and that of the top right corner district.


Fig. 7. Eight reference unit vectors for four sides and four corners.

3.2. Sizing vector estimation in each district 3.2.1. Reference direction of window sizing Now, in order to describe the detailed procedure, we introduce eight reference unit vectors that represent the expanding directions of window movements; ui for four S sides and ui for four corners as shown in Fig. 7 are C de"ned as ui "cos u i#sin u j and ui "cos u i#sin u j, (5) S S S C C C where p u" i S 2

p p and u " i# for i"0, 1, 2, 3. C 2 4

(6)

3.2.2. Direction and magnitude of sizing vector Here, we should estimate a sizing vector by comparing the means evaluated from three zones in each district. The detailed #owchart of determining a sizing vector in a district is shown in Fig. 8, where k , k , and k are I M O

241

means of gray levels in inner, middle, and outer zones, respectively. We heuristically use three conditions to determine a sizing vector. Condition I considers only the case that the mean in the inner zone is smaller than that in the outer zone in a district. Usually, in a normal condition, the mean in the inner zone (whose area is also larger than the area of the outer zone) is larger than that in the outer zone, especially when the PDOE signi"cantly reducing background components is used. Condition I alarms a tracker over an abnormal situation that undesirable strong clutter or the other moving components might exist around the outer zone and thus turns o! the adaptive procedure in this district. Condition II represents the situation of decreasing a window size in which an absolute mean di!erence between the inner zone and the middle zone is larger than that between the middle zone and the outer zone with some marginal value included. Condition III detects the opposite situation. Additionally, for reducing the adjusting sensitivity around the boundary of middle zone, we set a marginal factor a as 20% of k in Conditions II and III. As for the side M district, however, we found that such consideration of the marginal factor is not needed. When we determine the magnitudes of sizing vectors as in Fig. 8, we should consider the two issues. One issue is to assign a weight between the side and the corner based on frequency of occurrence of objects. When the window boundary is located on the border between the object and background components, as often happens in tracking, we found that an object exists more frequently in a side district rather than in a corner district. Thus, we set the corner weighting factor w to 0.7 to balance contribuC tions of a side sizing vector and its related two corner sizing vectors. We also note that the "nal assignment (as in Fig. 8) results in one of three cases: no change occurs in sizing, that is, S "0 or S "0; the window is S C

Fig. 8. Flowchart (a) and three conditions (b) for determining a sizing vector in a district.

242


recommended to expand, that is, S "u or S "w u ; S S C C C the third case is to shrink down the window, that is, S "!u or S "!w u . S S C C C Another consideration is a data loss problem due to a temporary sizing fault. If the window fails to enclose the whole part of an object by lavishly reducing its size, the tracker might lose some valuable information and this leads to quite undesirable situation. On the other hand, increasing a window size is more tolerant of sizing error, since the window still encloses a whole object. We thus make the sizing rate of increase be twice as large as that of decrease and re#ect this di!erence by the sign weighting factor w , which will be detailed in following Section s 3.4. This fact means that our window system is designed to be generous in expanding, but somewhat cautious in shrinking. 3.3. Final sizing vector determination using corner to side decomposition in four side directions We "rst decompose a corner sizing vector S into two C horizontally and vertically decomposed vectors S and CH S . CV S "S #S . (7) C CH CV Eventually, we can determine the "nal sizing vector for each side by performing the vector sum of original side component and the two components from neighboring corners. A "nal sizing vector Si for a horizontal direction i"0, 2, i.e., left and right of a window, is given by Si"Si #Si #S.0$4(i~1) (8) S CH CH and similarly, a "nal sizing vector for the vertical direction i"1, 3, i.e., top and bottom, is given by Si"Si #Si #S.0$4(i~1). (9) S CV CV A conceptual example has been forwarded in Fig. 9. The four "nal sizing vectors having information about the

Fig. 10. Fine tuning of tracking position for correcting the position error: positions before window sizing procedure (left) and after window sizing procedure (right).

magnitudes and directions of four sides have been shown in Fig. 9c. 3.4. Reference window relocation with xnal sizing vector Here, the "nal sizing vectors previously obtained should be converted to the true sizing units in pixels, and for this, the window size of the previous frame should be also considered. First, let the basic sizing weighting factor wi be de"ned as B

A B

w wi "max 1, i , B B

(10)

where the previous window size w in Fig. 10 is given by i

G

m if i"0, 2, w" i n if i"1, 3.

(11)

The parameter B is "xed to 50 in Eq. (10) to properly balance between the magnitude of the sizing vector and the size of the previous window. Still, we do not want wi to drop below 1.0. B Next, we put the origin of coordinates at the center of the window. Then we eventually obtain the window

Fig. 9. Conceptual example of the proposed method for determination of "nal sizing vector: (a) sizing vectors originally estimated from eight districts, (b) decompositions of corner sizing vectors, and (c) the "nal sizing vectors resulted from the sum of side sizing vectors and their decomposed sizing vectors.


sizing magnitude *Si in the ith direction. *Si now de"ned in pixel unit is evaluated by *Si"wi wi DDSiDD, S B

(12)

where a sign weighting factor wi is given by S

G

#2 if Si ' ui '0 S wi " S !1 otherwise.

(13)

Here, wi is employed to avoid a data loss risk referred to S in Section 3.2.2. When a corner weighting factor w is 0.7 C and a horizontal size of the previous window is about 100 pixels, the magnitude of the "nal sizing vector Si with wi is between !2 and #4. We found that the sizing S magnitude *Si ranges about from !4 pixels to #8 pixels. Finally, we now relocate the center point according to the change of window size. New window positions PI i, for i"0, 1, 2, 3, from the origin of coordinates can be simply given by PI i"Pi#sgn(Pi)*Si,

(14)

where

G

sgn(k)"

#1 if k'0, !1 if k(0,

(15)

and Pi is the coordinate value of the previous window position before the window sizing procedure. Fig. 10 illustrates such relocation of a center of the tracking window by using Eq. (14).

4. Experimental results We have applied our proposed method based on four independent sizing vectors to 45 real image sequences made of more than 3400 images and 39 arti"cial image sequences made of 4260 images and obtained satisfactory results for most of them. Now, the aim of this Section is to evaluate of the performance of our tracker with adaptive window sizing. However, when we actually performed tracking experiments based on the "xed size window for many cases in which the size of an object image underwent rapid change, the tracking failed so often that we did not bother to include these experimental results. Hence, we set up two methods designed with only side districts. These methods both lack in some sophisticated details of corner to side decomposition in our "nal tracker. In Method 1, wi information and only the direcB tion of Si are preserved by setting Si "0 in Eqs. (8) and S C (9), and DDSiDD is "xed as 2, which makes the magnitude of the resultant sizing vector comparable to that of the

243

proposed method. In this case, *Si3M!2wi , 0, #4wi N. B B As for Method 2, information about the previous window size (absorbed in wi ) as well as the magnitude of B the "nal sizing vector DDSiDD is further ignored. Here, wi "1, DDSiDD"1, and only the direction information is B just retained, i.e., *Si3M!1, 0, #2N, which means that the tracker can expand by two pixel steps or shrink by one pixel step. The key idea of Method 2 is quite similar to the gate sizing algorithm proposed by Chodos et al. [3]. For objective comparison, the initial position and the initial size of the adaptive window are set to be the same in the experiments. 4.1. Objects in artixcial image sequences Here, we selected arti"cial image sequences with 11 di!erent signal-to-noise ratios (SNRs) for the quantitative evaluation of the proposed method. Then, the SNR of the image and two error measures for evaluating the center position error and the size error of the window will be discussed below. The arti"cial image based on the Markov random "eld (MRF) [8] is constructed by using the stochastic features such as the brightness mean, the standard deviation, and the correlation coe$cient of the target and the background and then is added by the Gaussian noise. The SNR of the generated image is de"ned as Dk !k D B (dB), SNR"20 log T p N

(16)

where Dk !k D is the absolute di!erence of the brightT B ness means between the target and the background and p is the standard deviation of the added Gaussian noise. N We quantify the center position error E and the window P size error E as follows: S E "J(x!x8 )2#(y!y8 )2, P

(17)

DA!AI D E" , S AI

(18)

where the point (x, y) is the target position estimated by a tracker and the point (x8 , y8 ) is the actual target position on the image, which is available at the time of generating an image. A is the area of the adjusted tracking window and AI is the actual area of the known target. Fig. 11 represents part of the test image sequence with a SNR of 0.0 dB. The "nal tracking result is denoted by the white solid rectangle. The target, the non-homogeneously "lled rectangle, is designed to follow in clockwise direction along the boundary of a rhombus in order to produce much variation within the limited size of 256]256 pixels. The size of the moving target also changes slowly from the initial size of 50]50 pixels to the

244


Fig. 11. Sampled tracking example of the proposed method for an arti"cial image sequence using the MRF and Gaussian noise. The SNR considered here is 0.0 dB.

maximum size of 100]100 pixels and then backs to 50]50 pixels. This variation in size is intended to emulate a situation that a target approaches an observer to a certain point and retreats again. In order to evaluate the sizing capability of a tracker, the initial tracking window size is intentionally set to 121]121 pixels much larger than the initial target size. Furthermore, to test how fast a tracker responds to misalignment of the center

of an initial window with that of a target, the center of the tracking window is also initially set to be 20]20 pixels deviated from that of the target. This deviation amounts to E of 28.28. It was found in the test sequence of Fig. 11 P that the tracker could accomplish the correction of sizing and center misalignment at about Frame 23. Fig. 12 describes the evaluating results in terms of the two error measures of E and E for the same sequence of S P


245

Fig. 11. E for each frame is illustrated in Fig. 12a and S E in Fig. 12b. Here, we can "nd that Method 2 is slower P in reducing the window size error (E ) or the center S position error (E ) at the initial portion of tracking. This P is because Method 2 can expand by two pixels or shrink down by one pixel and cannot achieve fast correction of the initial erroneous setting of a window. Now, we consider Method 1 which is, in an aspect of sizing, somewhat similar to the proposed method but lacks in subtle adjustment of the window using corner to side decomposition. It can be concluded from Fig. 12a and b that this method could adjust as fast but shows more oscillatory behavior compared to the proposed method. Table 1 shows the averages of E s over all the frames P using the arti"cial image sequences with varying SNRs Table 2 Average of window size errors over all frames with respect to SNRs under normal conditions with proper initial window size and no center position error Average of window size errors (E s) S

SNR (dB)

Fig. 12. Window size error and center position error evaluation with respect to the separate frames of Fig. 11 (SNR is 0.0 dB): (a) window size error variation and (b) center position error variation.

0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0

Proposed method

Method 1

Method 2

0.37 0.37 0.37 0.34 0.34 0.34 0.35 0.34 0.34 0.33 0.33

0.53 0.45 0.40 0.39 0.40 0.43 0.37 0.36 0.35 0.39 0.36

0.54 0.44 0.43 0.39 0.37 0.38 0.40 0.31 0.32 0.35 0.36

Table 1 Average of center position errors over all frames with respect to SNRs with the large initial window of 121]121 pixels Average of center position errors (E s) P

SNR (dB)

0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0

Initial center position error"0.00



Proposed method

Method 1

Method 2

Proposed method

Method 1

Method 2

Proposed method

Method 1

Method 2

4.72 3.10 3.12 2.60 2.89 2.95 2.80 2.23 2.60 2.86 2.47

8.41 5.55 5.68 5.65 6.02 6.45 5.97 4.44 4.72 4.80 4.45

6.82 3.66 3.71 2.84 2.95 2.13 2.43 1.45 1.62 1.94 2.18

5.26 4.07 4.10 4.42 4.39 4.35 3.98 4.00 3.91 3.93 3.89

14.41 8.66 8.40 6.80 7.59 5.82 7.89 5.52 7.43 6.05 5.65

10.80 8.18 6.31 6.08 7.67 6.87 6.29 5.57 6.18 6.21 5.62

7.22 6.68 6.29 6.10 5.75 5.66 5.57 5.44 5.35 5.68 5.23

10.44 8.13 7.69 7.87 7.38 6.66 6.92 7.71 7.09 7.12 6.70

13.77 13.67 12.25 11.73 11.36 10.84 11.33 9.27 9.84 9.88 9.74

246


Fig. 13(a) and (b). Caption opposite.


247

over all SNRs in Table 2 is 0.35 and is also more e!ective than the other two methods. If we convert this average to the more intuitive pixel unit, we found that our tracker usually keeps a margin of about "ve to six pixels between the window border and the target border as shown in the latter part of Fig. 11. Referring to Tables 1 and 2, we can conclude the proposed method is more robust about the variation of SNR than Methods 1 and 2. 4.2. Objects in real image sequences

Fig. 13. Real image sequence that a helicopter goes over a ridge with complex background: (a) sampled tracking results by the proposed method, (b) window size error and center position error introduced in the tracking window of Methods 1 and 2 due to crossing over the complex background, (c) window size variation in horizontal and (d) in vertical direction.

from 0.0 to 10.0 dB. We classify the initial testing conditions into three initial E s of 0.00, 14.14, and 28.28. P For instance, when the initial E is 28.28 and the P SNR of the image is 0.0 dB as shown in Fig. 11, the proposed method with an average of E s of 7.22 P is the most superior to the other methods under the same condition. The averages of E s were found to P vary signi"cantly according to the initial E s. In fact, P we found that E falls to 2.91 when only averaged from P Frame 23, at which the tracking window enclosed the target properly as described in Fig. 12a, to the last frame. Meanwhile, Table 2 is included to show the sizing performance of the tracker under the normal condition. To simulate this condition, we let the initial E be 0.00 P and the initial window size be slightly bigger but similar to the size of the target. Our proposed method averaged

We evaluated the performance of the proposed method using the real image sequences with various situations acquired from a CCD camera and a frame grabber. A moving object such as an airplane, a "ghter, a helicopter, an automobile, a bike, a human being, and so on is chosen as a target. Usually, these targets move near or through the clutters in complex circumstances without imposing any constraint on their moving patterns. Fig. 13 represents the tracking results for a helicopter going away fast through clutters across a ridge. An initial window was manually chosen to be of larger size than the target. The intentional position error had been included to measure the speed of the window sizing and also to consider the real situation that an operator could not always set the initial window quite tightly. The proposed method could enclose the target properly as in Fig. 13a. From Fig. 13b that are magni"ed three times, however, we found that in Methods 1 and 2 the center position error as well as the window size error had been induced, as shown in Frames 19 and 20, respectively, by including part of clutters inside the tracking window. Here, we found that these two methods were more susceptible to the momentary sizing error due to the complex clutters. Fig. 13c and d describe the vertical and horizontal size variations of the window in this sequence made of 25 frames. Method 1 performed badly in tracking because the severe error had been accumulated about from Frame 16, that is, the point at which the target began to intersect the ridge. This accumulation could be observed in Fig. 13c and d. Method 2 also could not cope with the situation of the fast decreasing target and thus had failed in enclosing the target as desired. Our proposed method was most successful in tracking the indicated target stably by adjusting the window size properly. The tracking results for a vehicle going over the sloping road with huge trees standing nearby are also illustrated in Fig. 14. The wooded forest makes it hard for the tracker to obtain the reliable tracking performance. We tested a case of making the initial tracking window be wide in the horizontal direction as shown in Fig. 14a. Nevertheless, the proposed method could follow the target most correctly by rapidly controlling the window size. The window size errors for Frame 38 and the "nal frame of Methods 1 and 2 are described in Fig. 14b. As for

248


Fig. 14. Real image sequence that a vehicle is running over the inclined road near the forest: (a) sampled tracking results by the proposed method and (b) window size error and center position error of the tracking window occurred in Methods 1 and 2.


249

Method 1, the window size could not be adjusted suitably since the beginning of the sequence. Although the sizing e$ciency of Method 2 was better than that of Method 1, Method 2 was still found to be not so accurate as the proposed method around the end of the sequence.

retreat, and other moving patterns and this history information can be useful for aim-point estimation and other related applications. Since the adaptive window can tightly adjust window size around object, computation time for correlation could be also optimized.

5. Conclusions

References

We have presented an image tracking architecture employing the four-direction adaptive window method with independent sizing vectors for enhancing the performance of correlation-based tracking in the cluttered surroundings. Our method could control the sizing magnitudes fast enough in four directions to reduce the in#uence of the background and increase the occupancy rate of the target. At the onset of tracking, generally a human operator establishes an initial window with its size much larger than the object size. In this case, the tightly enclosing speed of the window with our proposed sizing vectors is found to be over three times faster than that of a window system without the adaptive sizing unit (Method 2). By virtue of the proposed method capable of adjusting the window size more rapidly and properly by generating the sizing vectors in four directions, we could achieve bene"t of minimizing the in#uence of complex background and clutters in correlation-based tracking process. Moreover, this method can more "nely tune the size and position of the reference window after the main correlation-based tracking routine has been terminated. Besides these advantages, we could obtain circumstantial judgment of a moving object: rotation, advance or

[1] D. Hughes, A.J.E. Moy, Advances in automatic electrooptical tracking systems, Proc. SPIE 1697 (1992) 353}366. [2] D.A. Montera, S.K. Rogers, D.W. Ruck, M.E. Oxley, Object tracking through adaptive correlation, Opt. Eng. 33 (1) (1994) 294}302. [3] S.L. Chodos, G.T. Pope, A.K. Rue, R.P. Verdes, Dual Mode Video Tracker, U.S. Patent No. 4849906, 1989. [4] R.L. Brunson, D.L. Boesen, G.A. Crockett, J.F. Riker, Precision trackpoint control via correlation track referenced to simulated imagery, Proc. SPIE 1697 (1992) 325}336. [5] S. Sung, S. Chien, M. Kim, J. Kim, Adaptive window algorithm with four-direction sizing factors for robust correlation-based tracking, Proceedings of the Ninth IEEE International Conference on Tools with Arti"cial Intelligence, 1997, pp. 208}215. [6] T. Kanade, M. Okutomi, A stereo matching algorithm with an adaptive window: theory and experiment, IEEE Trans. Pattern Anal. Mach. Intell. 16 (9) (1994) 920}932. [7] J.L. Lotti, G. Giraudon, Adaptive window algorithm for aerial image stereo, Proceedings of the Twelfth International Conference on Pattern Recognition, 1994, pp. 701}703. [8] A.K. Jain, Fundamentals of Digital Image Processing, Prentice-Hall, Englewood Cli!s, NJ, 1989.

About the Author*SUNG-IL CHIEN recieved the B.S. degree from Seoul National University, Seoul, Korea, in 1977, and the M.S. degree from the Korea Advanced Institute of Science and Technology, Seoul, Korea, in 1981, and Ph.D. degree in Electrical and Computer Engineering from Carnegie Mellon University in 1988. Since 1981, he has been with the School of Electronic and Electrical Engineering, Kyungpook National University, Taegu, Korea, where he is currently a professor. His research interests are pattern recognition, computer vision, and neural networks. About the Author*SI-HUN SUNG recieved the B.S. and M.S. degrees in Electronic Engineering from the Kyungpook National University, Taegu, Korea, in 1995 and 1997, respectively. He is currently working towards the Ph.D degree in Electronic Engineering at the Kyungpook National University as a research assistant. His research interests include the areas of the "eld application of computer and machine vision, image processing, pattern recognition, and neural networks. He is a member of the SPIE, the IEEE, and the Institute of Electronics Engineers of Korea.


Point-based projective invariants TomaH s\ Suk*, Jan Flusser Institute of Information Theory and Automation, Academy of Sciences of the Czech Republic, Pod voda& renskou ve\ z\ n& 4, 182 08 Praha 8, Czech Republic Received 23 January 1998; accepted 3 February 1999

Abstract The paper deals with features of a 2-D point set which are invariant with respect to a projective transform. First, projective invariants for "ve-point sets, which are simultaneously invariant to the projective transform and to permutation of the points, are derived. They are expressed as functions of "ve-point cross-ratios. Then, the invariants for more than "ve points are derived. The algorithm for searching the correspondence between the points of two 2-D point sets is presented. The algorithm is based on the comparison of two projective and permutation invariants of "ve-tuples of the points. The best-matched "ve tuples are then used for the computation of the projective transformation and that with the maximum of corresponding points is used. Stability and discriminability of the features and behavior of the searching algorithm are demonstrated by numerical experiments. ( 1999 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Projective transform; Point set matching; Point-based invariants; Projective invariants; Permutation invariants; Registration; Control points

1. Introduction One of the important tasks in image processing and computer vision is a recognition of objects on images captured under di!erent viewing angles. However, this problem cannot be solved in a general case [1]. Nevertheless, if we restrict ourselves to planar objects only, then the distortion between two frames can be described by projective transform (sometimes called perspective projection) x@"(a #a x#a y)/(1#c x#c y), 0 1 2 1 2 y@"(b #b x#b y)/(1#c x#c y), (1) 0 1 2 1 2 where x and y are the coordinates in the "rst frame and x@ and y@ are the coordinates in the second one. Feature-based recognition of such objects requires features invariant to projective transform (1). Several di!er* Corresponding author. Tel.: #420-2-6605-2231; fax: #420-2-688-4903. E-mail addresses: [email protected] (T. Suk), #[email protected] (J. Flusser)

ent approaches to this problem have been published in recent works. One of them is based on the assumption that the non-linear term of the perspective projection is relatively small and thus the projective transform can be approximated by an a$ne transform. This assumption is true, if the distance from the sensor to the object is much greater than the size of the object. In such cases, various a$ne invariants can be applied such as moment invariants [2,3] or Fourier descriptors [4,5]. However, in some cases the projection cannot be approximated by the a$ne transform and therefore the use of exact projective invariants is required. The invariants, which have been developed for this purpose, can be categorized into two groups: di!erential invariants and point-based ones. Di!erential invariants are applicable only if the object boundary is a smooth continuous curve. A set of invariants based on boundary derivatives up to the sixth order was presented by Weiss [6]. Unfortunately, these invariants are not de"ned for such important curves as straight lines or conics. Weiss's invariants are numerically unstable because of the high-order derivatives. To overcome this di$culty, several improvements were presented [7}9].


252

T. Suk, J. Flusser / Pattern Recognition 33 (2000) 251}261

The second group of invariants is de"ned on point sets [10], on sets composed both from points and straight lines [11,12] and on triangle pairs [13]. A detailed survey of the point-based methods can be found [14]. Another problem to be solved is to establish the correspondence between two point sets, which are projectively deformed. To calculate the invariants, we have to order those sets somehow. The solution when the points are vertices of a polygon has been published [15]. Another solution is to use features also invariant to the order or labeling of the points. Five-point projective and permutation invariants are presented [2,16]. This approach is also used in this paper. The plane transformed by projective transform (1) contains a straight line 1#c x#c y"0, 1 2

(2)

which is not mapped into the second plane (more precisely it is mapped into in"nity) and which divides the plane into two parts. If all elements of our point set lie in one half-plane, then some additional theorems about topology of the set hold for the transform, e.g. the convex hull is preserved during the transform in that case. This fact can be used to derive invariants with lower computational complexity [17]. This paper deals with a general case of the projective transform, when the points can lie in both parts of the plane. The convex hull is not preserved under the transform and all possibilities of the positions of the points must be taken into account. The only assumption is that the points do not lie directly on straight line (2). A projective invariant can be de"ned for at least "ve points. The simplest one is a "ve-point cross-ratio P(1, 2, 3)P(1, 4, 5) . (1, 2, 3, 4, 5)" , P(1, 2, 4)P(1, 3, 5)

(3)

where P(A, B, C) is the area of the triangle with vertices A, B and C. The point No. 1 is included in all four triangles and it is called the common point of the cross-ratio. Reiss [2] proposes to use the median of all possible values of . . A more precise description of the relations between various cross-ratio values under permutations of the given points can be found [16]. After the correspondence between the individual points in both sets has been established, we use them as the control points for image-to-image registration. However, there are often some points having no counterpart in the other image. An approach to solve this problem can be found [18], but that method becomes unstable if the number of the `wronga points increases. The goal of this paper is to derive projective and permutation invariants of point sets. Five-point projec-

tive and permutation invariants are derived in Section 2, they are generalized for more than "ve points in Section 3 and the sets with wrong points are discussed in Section 4. Experiments showing the numerical properties of the invariants as well as their usage for image registration are shown in Section 5.

2. Five-point permutation invariants 2.1. The derivation of the invariants First we derive permutation invariants by the simplest way and then more detailed analysis will be performed. The main principle is to use addition or multiplication (or another symmetric function) of all possible values of the projective invariants over the permutations of the points. The order of terms and factors is only changed during permutations, but the result stays invariant. To obtain permutation invariants, we can employ various functions of cross-ratio (3). Reiss [2] used the function . #. ~1, which is unstable near zero. If some triplet of "ve points in Eq. (3) is collinear, then the function . #. ~1 is in"nite. Thus the more suitable function is t"2/(. #. ~1)"2. /(. 2#1). If . or . ~1 is zero, then the function t is zero. The function . can have only three distinct values during permutations of four points, therefore the functions: F@ (1, 2, 3, 4, 5)"t(1, 2, 3, 4, 5)#t(1, 2, 3, 5, 4) ` #t(1, 2, 4, 5, 3), F@ (1, 2, 3, 4, 5)"t(1, 2, 3, 4, 5) ) t(1, 2, 3, 5, 4) > ) t(1, 2, 4, 5, 3)

(4)

are invariant to the choice of labeling of the last four points, but the point No. 1 must be common at all cross-ratios. To obtain full invariance to the choice of labeling, we must alternate all "ve points as common ones: I 1 2(1, 2, 3, 4, 5)"F 1(1, 2, 3, 4, 5)s F 1(2, 3, 4, 5, 1) s ,s s 2 s s F 1(3, 4, 5, 1, 2)s F 1(4, 5, 1, 2, 3) 2 s 2 s s F 1(5, 1, 2, 3, 4), (5) 2 s where s and s are either sign # or ) . 1 2 The set of n points has 2n degrees of freedom and the projective transform has eight parameters. Therefore, the set can have only m"2n!8

(6)

independent invariants to the projective transform. That is why only two of the four invariants I , I , I and ` >` `> I can be independent. >>


253

2.2. The roots of the invariants Lenz and Meer [16] dealt with the "ve-point projective and permutation invariants in detail. They discovered that if the common point stays the same and the other points are permuted, then the values of the crossratios are 1 1 . ". , . " , . "1!. , . " , 1 2 . 3 4 1!.

. . !1 . " , . " . 5 . !1 6 .

(7)

If we construct a function F(. ), which has the same value for all these values of . (. , . ,2, . ), it is invari1 2 6 ant to the permutation of four points. If we change the common point, we receive another value of the crossratio, let us say p, and the function

AB A B

K(. , p)"F(. )#F(p)#F

A

#F

B

. (p!1) p(. !1)

. . !1 #F p!1 p

(8)

is a "ve-point projective and permutation invariant. As our study implies, if the function F(. ) is constructed as the quotient of two polynomials and . is its root, then each of the values . , . ,2, . must be its root. We can 1 2 6 consider it in the form P (. ) F(. )" 1 , (9) P (. ) 2 P (. )"(. !. )(. !. )(. !. )(. !. ) 1 1 2 3 4 (10) ](. !. )(. !. ) 5 6 and similarly for P (. ) (when all roots di!er from zero). 2 It is advantageous if F(. ) is de"ned for any real . . Thus, P (. ) should not have real roots. Two such invari2 ants are proposed [16] 2. 6!6. 5#9. 4!8. 3#9. 2!6. #2 F " , 14 . 6!3. 5#3. 4!. 3#3. 2!. 3#1

(11)

(. 2!. #1)3 F " . 15 . 6!3. 5#5. 3!3. #1

(12)

where there are the following relations between the roots: b b 1 2 , a "1!a , b "b , b " " 2 1 2 1 3 (a !1)2#b2 a2#b2 1 1 2 2

Theorem 1. If the roots of P (. ) are imaginary, then they 2 lie on the following curves:

(13)

(14)

where a and b , i"1, 2, 3 are real and imaginary parts of i i the roots. The theorem is illustrated in Fig. 1. Proof. If we express P (. ) in the form 2 P (. )"(. !a !b i)(. !a #b i)(. !a !b i) 1 1 1 1 2 2 2 ](. !a #b i)(. !a !b i)(. !a #b i), 2 2 3 3 3 3 i"J!1,

The following Theorem describes the properties of the roots of P (. ). 2

a2#b2"1, 1 1 (a !1)2#b2"1, 2 2 a "1, 3 2

Fig. 1. Illustration of Theorem 1. The circles and the straight line show possible positions of the roots of the invariants in the complex plane (a is real part and b imaginary part).

(15)

then there are 6!"720 possibilities of assignment between a $b i, a $b i, a $b i and . }. in Eq. (7). If 1 1 2 2 3 3 1 6 we use the assignment a #b i". , 1 1 1

(16)

a !b i". , 1 1 2

(17)

a #b i". , 2 2 6

(18)

a !b i". , 2 2 3

(19)

a #b i". 3 3 4

(20)

254


and a !b i". , 3 3 5 then from Eqs. (7) and (16) 1 1 a !b i 1 . " " " 1 2 . a #b i a2#b2 1 1 1 1 1 and from Eq. (17)

(21)

(22)

!b a 1 , 1 . a " !b " (23) 1 a2#b2 1 a2#b2 1 1 1 1 Therefore the "rst two roots must lie on the circle a2#b2"1. From Eq. (7) 1 1 . !1 a !1#b i a2!a #b2#b i 1" 1 1 1 1 (24) . " 1 " 1 6 . a #b i a2#b2 1 1 1 1 1 and

. "1!. "1!a !b i 3 1 1 1 from Eqs. (18), (23) and (24) a2!a #b2 b 1 1"1!a , b " 1 "b a " 1 2 1 2 a2#b2 1 a2#b2 1 1 1 1 and from Eqs. (19) and (26)

(25)

(26)

a "1!a , b "b , (27) 2 1 2 1 therefore the second two roots must lie on the circle (a !1)2#b2"1. From Eq. (7) 2 2 1 1 1!a #b i 1 1 . " " " (28) 4 1!. 1!a !b i (1!a )2#b2 1 1 1 1 1 and

. a #b i a2!a #b2!b i 1 " 1 1 " 1 1 1 1 , (29) . " 5 . !1 a !1#b i (1!a )2#b2 1 1 1 1 1 from Eqs. (20), (23) and (28) 1!a 1 b b 1 2 1 " , b " " a " 3 (1!a )2#b2 2 3 (1!a )2#b2 a2#b2 1 1 1 1 2 2 (30)

that each individual case falls into one of the following categories: 1. The result is some permutation of the previous case. 2. The result is only a "nite set of values, typically J 1$i 3. 2 2 3. The case has no solution. Thus, there is no other solution and the theorem has been proven. h Our invariants have the form in this context 2(1!. ) 2. /(. !1) 2. # # F@ " ` . 2#1 (1!. )2#1 . 2/(. !1)2#1 "2

. 6!3. 5#3. 4!. 3#3. 2!3. #1 , 2. 6!6. 5#11. 4!12. 3#11. 2!6.#2

(!8). 2(1!. )2 F@ " > (1#. 2)(2!2. #. 2)(1!2. #2. 2) !8. 2(1!. )2 " . (32) 2. 6!6. 5#11. 4!12. 3#11. 2!6. #2 The choice of the invariant has one degree of freedom, we can choose one root of the denominator on some curve from Fig. 1, other roots must be de"ned by Theorem 1 and the numerator de"nes the range of values of the invariant. Since both F@ and F@ have ` > the same denominator (roots 1$i, $i and 0.5$0.5i), it is suitable to change one of them. The other can be (. 2!. #1)3 with roots 0.5$i J3. Then, if we want the 2 range of values of the consequential invariants I and `` I from 0 to 1, our invariants will be >` 8 . 2(1!. )2 F " ` 5 (. 2!. #1)3 3. 2(1!. )2 F" > 2. 6!6. 5#11. 4!12. 3#11. 2!6. #2

(33)

and relations to the original invariants will be F "16(1!1/(6!F@ )) and F "!3F@ . ` > 8 > ` 5

and from Eqs. (21) and (29)

2.3. The normalization of the invariants

a2!a #b2 1 b b 1 1" , b " 1 a " 1 " 2 3 (1!a )2#b2 2 3 (1!a )2#b2 a2#b2 1 1 1 1 2 2 (31)

The invariants I "I and I "I corresponding 1 `` 2 >` to the functions F and F utilize the feature space ` > ine!ectively (see Fig. 2). A better result can be reached by the sum and the di!erence (see Fig. 3)

therefore the third two roots must lie on the straight line a "1. 3 2 We cannot investigate all other 719 possibilities because of insu$cient space. The number of cases can be reduced signi"cantly, if we consider mutual relations among the roots only. Thus, we have to deal with 5!"120 cases only. The other 600 cases are just permutations. We treated all of the 120 cases and we proved

I@ "(I #I )/2, I@ "(I !I #0.006) ) 53, (34) 1 1 2 2 1 2 but the best utilization of the given range is reached by subtraction and division by the polynomials (see Fig. 4) 1!I@ #p(I@ ) 2 1. IA "I@ , IA " 1 1 2 d(I@ ) 1

(35)


Fig. 2. Possible range of values of the invariants I , I . It was 1 2 acquired numerically by computing invariants for all combinations of "ve points with integer coordinates from 0 to 511.

255

Fig. 4. Possible range of values of the invariants IA , IA . 1 2

3. Invariants for more than 5ve points There is a number of approaches to the problem of the generalization of the invariants from the previous section to n points (n'5). One of them, yielding good experimental results, consists in summation of powers of the invariants IA , IA over all possible combinations of 5 from 1 2 n points Cn . 5 Theorem 2. I " + IAk (Q), I " + IAk (Q), 1,k 1 2, k 2 Q3Cn5 Q3Cn5 k"1, 2,2n!4

(36)

are projective and permutation invariants of a set of n points.

Fig. 3. Possible range of values of the invariants I@ , I@ . 1 2

Exact coe$cients of the polynomials p(I@ ) and d(I@ ) 1 1 are shown in Appendix B. This normalization is not necessary, but it makes possible to use a simpler classi"er.

Proof. IA and IA are projective invariants (see Eq. (3)) and 1 2 an arbitrary function of invariants is also invariant (if it does not depend on the parameters of the transform), therefore I and I are also projective invariants. 1,k 2,k IA and IA are also 5-point permutation invariants and 1 2 summation over all combinations guarantees permutation invariance of the IA and IA . h 1 2 The number of these invariants is chosen as 2n!8 according to Eq. (6). The computing complexity is approximately (n )¹, i.e. O(n5), where ¹ is the computing 5 complexity of one "ve-point invariant. However, the number of the terms is very high and that is why a

256


normalization of the results is suitable. To preserve the values inside acceptable intervals, we can use

S NA B n

+ IAk(Q), (37) s Q3Cn5 where s"1 or 2. Another, perhaps more sophisticated, normalization is the following. We can consider the "ve-point invariants IA , IA as independent random variables with uniform 1 2 distribution in the interval from 0 to 1. Then the distribution function F (x) of the kth power of the invariant is k F (x)"0 from!R to 0, k F (x)"x1@k from 0 to 1, (38) k F (x)"1 from 1 to R k with the mean value k "1/(1#k) and the variance k p2"k2/(1#k)2(1#2k). The number of terms in sum k (36) is relatively high and the Central Limit Theorem implies that the distribution of the sum is approximately Gaussian, its mean value is the sum of its mean values k and the variance is the sum of the variances p2. The k k given range is the best utilized in case of uniform distribution of the resulting invariants and therefore we can normalize the invariants with Gaussian distribution function I@ "k 1 s,k

5

P

1 x G(x; k, p)" e~(m~k)2@2p2 dm J2pp ~=

(39)

with the mean value k"(n )/(1#k), the variance 5 p2"(n )k2/(1#k)2(1#2k) and the standard deviation 5 p"k/(1#k)J(n )/(1#2k) 5 IA "G(I ; k, p). (40) sk s,k An approximate substitution of the function G is used in practice.

4. Point matching The problem we are dealing with in this section can be formulated as follows. Let us have two-point sets selected from two projectively deformed images. The sets can include some wrong points, i.e. points without a counterpart in the other set. We look for the parameters of the projective transform to register the images. We deal with the methods, which do not consider the image functions, but only the positions of the points. 4.1. Full search algorithm The simplest algorithm is the full search of all possibilities of the correspondence. We can examine each four points from the input image against each four points

from the reference image. If we have n points in the input image and l points in the reference one, we must examine (n )( l )4! possibilities. 4 4 The examination means the computation of the projective transform, the transformation of the points in the input image and judgment of the quality of the transform. We performed this judgment in the following way. The two nearest points are found and removed and again two nearest points from the remaining ones are found. The search is complete when the distance between the nearest points exceeds the suitable threshold. The number of corresponding points is used as the criterion of the quality of the transform. If the number is the same, the distance of the last two points is used. The best transform according to this criterion is used as the solution. The threshold must correspond to the precision of the selection of the points. The threshold 5 pixels proved its suitability in usual situations. If n and l are approximately the same and high, this algorithm has the computing complexity O(n11) and in our experience it is too time consuming. 4.2. Pairing algorithm by means of the projective and permutation invariants We can compute the distance in the feature space between invariants of each "ve points from the input image against invariants of each "ve points in the reference image. Nevertheless, it was found that wrong "ve points often match one another randomly, this false match can be better than the correct one and we must search not only the best match, but also each good match. We carried out experiments with a number of searching algorithms. We consider the following as the best one. We "nd the "rst b best matches and the full search algorithm from the previous section is applied on each pair of "ve tuples corresponding each match. The number b was chosen as (.!9(n, l) ), but this number is not 5 critical. Note: the total number of pairs of "ve tuples is (n )( l ). 5 5 In Ref. [19] the convex hull constraint is proposed. It is based on the assumption that the sets lie in one halfplane of Eq. (2) and that the projective transform preserves the position of points on or inside the convex hull. Then the pairs of "ve-tuples with the di!erent number of points on the convex hull need not be considered. As was written in the introduction, we consider the general case of the projective transform and therefore this constraint is not used. In the same work the idea of partial search is proposed. The authors randomly chose about one-"fteenth of all pairs and tried to search them only. They found the decrease of reliability relatively small. This constraint can be used in our algorithm too, but the following numerical experiment used the general algorithm without this constraint, because the amount of time saved is relatively small.


257

5. Numerical experiments How do we investigate the stability and discriminability of the invariants? Let us imagine the following situation. We have two similar sets of points and we would like to recognize one from the other, but one of them can be distorted by the noise in its coordinates and we would like the noise not to in#uence the recognition. The following numerical experiment was carried out to observe the behavior of invariants (37) and (40) in this situation. Let us have three sets of 11 points. One of them was created by a pseudo-random generator. The point coordinates are uniformly distributed from 0 to 511. The second set was created from the "rst one by moving one point a relatively large distance. The coordinates of the movement are also de"ned by a pseudo-random generator, but with Gaussian distribution. The third set was created from the "rst one by adding small noise to all point coordinates. The noise is independent with zeromean Gaussian distribution and with gradually increasing standard deviation. The standard deviation p of the 1 movement in the second set increased from 0 to 190 with the step 10 and the standard deviation p of the noise in 2 the third set increased from 0 to 9.5 with the step 0.5. The distances d between original and noisy set in the space of the invariants were computed. Since one experiment would not be representative enough, 10, 20, 100 and 1000 experiments were gradually carried out for each standard deviation. A curve of dependency of the distance on the noise standard deviation was acquired as the average of 1000 experiments, because

Fig. 5. The distance between the "rst and second sets (solid line) and between the "rst and third sets (dashed line) in the Euclidean space of the invariants normalized by the average and root as a function of the noise standard deviation.

Fig. 6. The distance between the "rst and second sets (solid line) and between the "rst and third sets (dashed line) in the Euclidean space of the invariants normalized by the Gaussian distribution function as a function of the noise standard deviation.

the average of fewer values was too dependent on the concrete realization of the noise. The result of invariants (37) normalized by the average and root is given in Fig. 5. The scale of the horizontal axis is di!erent for both cases so the curves were as similar as possible, more precisely, the area of the square of the di!erence between them was minimized. The ratio of the scales is 7.36; it means if two sets di!er by only one point, then the distance of the points must be at least approximately 7.36 times greater than the noise standard deviation to be possible to distinguish both sets. In another words, if the ratio of the standard deviations is 7.36 and their value is such that the dashed line is under the solid one, the sets will be recognized correctly. If the ratio increases, the limit of correct recognition increases too, but if the noise standard deviation is greater than approximately 9, i.e. about 2% of the coordinate range, then the sets cannot be distinguished at all, because the dashed line is always above the solid one. The results of invariants (40) normalized by the Gaussian distribution function are given in Fig. 6. The result is similar to the previous case, the ratio of the scales is 11.18, that means a little bit worse discriminability. The main di!erence is the scale on the vertical axis, which is about twice larger. It means these invariants utilize the given range better. The second experiment demonstrates using the pairing algorithm by means of the projective and permutation invariants. A cut of a Landsat Thematic Mapper image of north-east Bohemia (surroundings of the town Trutnov) from 29 August 1990 (256]256) was used as the

258


Fig. 7. The satellite image used as reference one (] the control points with counterparts in the input image, # the points without counterparts).

reference image (see Fig. 7) and an aerial image from 1984 (180]256) with a relatively strong projective distortion was used as the input one (see Fig. 8). Sixteen points were selected in the input image (see their coordinates in Table 1), 18 points were selected in the reference one (see their coordinates in Table 2) and 10 points in both images had counterparts in the other image (numbers 1}10 in the input correspond to numbers 9}18 in the reference). The "rst (18)"8568 best matches 5 was examined and the 476th one was correct. All 10 pairs of control points were found, the distance of the tenth pair was 2.23 pixels. The result is shown in Fig. 9. The "nal parameters of the transform were computed from all 10 control points by means of the least-square method. The deviations on the control points were from 0.17 to 1.71 pixels, the average was 0.94 pixels. The time of the search of the best matches was about an hour and a half on the workstation HP 9000/700 and the time of the examination of all 8568 matches was about two hours and a half, but the correct 476th match was found in 8 min. The method supposes plane point sets, that is satis"ed on these images only approximately. In our experience, if the height of the #ight is signi"cantly greater than altitude di!erences between hills and valleys, then the in#uence of the terrain causes only small perturbations of point coordinates. Owing to the robustness of the algorithm, we can handle those cases satisfactorily.

Fig. 8. The aerial image used as the input one (] the control points with counterparts in the reference image, # the points without counterparts). Table 1 The coordinates of the points marked by ] and # in the input image in Fig. 8 No. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

x

y

35 233 104 253 16 202 73 130 55 176 172 152 47 72 126 114 113 155

42 106 166 147 243 235 39 40 111 67 197 215 122 86 118 146 181 182

T. Suk, J. Flusser / Pattern Recognition 33 (2000) 251}261 Table 2 The coordinates of the points marked by ] and # in the reference image in Fig. 7 No. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

x

y

22 161 117 64 8 50 90 61 35 96 65 14 35 145 139 161

26 27 189 214 31 15 52 73 114 142 143 116 153 149 51 75

259

suitable, because then they can be used for recognition in Euclidean feature space without any additional weights. The normalization by the Gaussian distribution function is suitable in case of less noise for better distinguishing of the sets. We can use the invariants also for registration of the images by means of control points. If the a$ne and simpler transforms can be used for approximation of the distortion between the images, other methods are suitable. In case of strong projective distortion between the images, the described algorithm is one of the possible solutions of the task. The minimum number of corresponding pairs is six and correspondence between point sets with less pairs of points cannot be found principally. In case of six corresponding pairs only once the wrong correspondence was found during tens of experiments, in case of more than six corresponding pairs no error was found. It means that in case of more than six corresponding pairs the hope of a successful result is very high.

7. Summary

Fig. 9. The registered image.

6. Conclusion The roots of the polynomials in the "ve-point projective and permutation invariants have one degree of freedom. We can choose one of them and the others must lie symmetrically on certain curves. The normalization of these invariants is suitable for improving numerical stability of following computations with them. The normalization of the invariants for more than "ve points is also

The paper deals with features of a point set which are invariant with respect to a projective transform. First, projective invariants for "ve-point sets, which are simultaneously invariant to the projective transform and to permutation of the points, are derived. They are expressed as functions of "ve-point cross-ratios. The roots of the polynomials in the "ve-point projective and permutation invariants have one degree of freedom. We can choose one of them and the others must lie symmetrically on certain curves. The normalization of these invariants is suitable for improving the numerical stability of following computations with them. The invariants for more than "ve points are derived. The normalization of the invariants for more than "ve points is also suitable, because then they can be used for recognition in Euclidean feature space without any additional weights. The normalization by the Gaussian distribution function is suitable in case of less noise for better distinguishing of the sets else the normalization by the average and the root should be used. The algorithm for searching the correspondence between the points of two 2-D point sets is presented. The algorithm is based on the comparison of two projective and permutation invariants of "ve-tuples of the points. The best-matched "ve-tuples are then used for the computation of the projective transformation and that with the maximum of corresponding points is used. Stability and discriminability of the features and behavior of the searching algorithm are demonstrated by numerical experiments.

260


Acknowledgements

References

This work has been supported by the Grant Nos. 102/98/P069 and No. 102/96/1694 of the Grant Agency of the Czech Republic.

[1] J.B. Burns, R.S. Weiss, E.M. Riseman, The non-existence of general-case view-invariants, in: J.L. Mundy, A. Zisserman (Eds.), Geometric Invariance in Computer Vision, MIT Press, Combridge, MA, 1992, pp. 120}131. [2] T.H. Reiss, Recognition planar objects using invariant image features, Lecture Notes in Computer Science, vol. 676, Springer, Berlin, 1993. [3] J. Flusser, T. Suk, Pattern recognition by a$ne moment invariants, Pattern Recognition 26 (1993) 167}174. [4] C.C. Lin, R. Chellapa, Classi"cation of partial 2-D shapes using Fourier descriptors, IEEE Trans. Pattern Anal. Mach. Intell. 9 (1987) 686}690. [5] K. Arbter, W.E. Snyder, H. Burkhardt, G. Hirzinger, Application of a$ne-invariant Fourier descriptors to recognition of 3-D objects, IEEE Trans. Pattern Anal. Mach. Intell. 12 (1990) 640}647. [6] I. Weiss, Projective invariants of shapes, Proceedings of the Image Understanding Workshop, Cambridge, MA, USA, 1988, pp. 1125}1134. [7] C.A. Rothwell, A. Zisserman, D.A. Forsyth, J.L. Mundy, Canonical frames for planar object recognition, Proceedings of the Second ECCV, Springer, Berlin, 1992, pp. 757}772. [8] A.M. Bruckstein, R.J. Holt, A.N. Netravali, T.J. Richardson, Invariant signatures for planar shape recognition under partial occlusion, Proceedings of the 11th International Conference on Pattern Recognition, The Hague, The Netherlands, 1992 pp. 108}112. [9] I. Weiss, Di!erential invariants without derivatives, Proceedings of the 11th International Conference on Pattern Recognition, The Hague, The Netherlands, 1992, pp. 394}398. [10] P. Meer, I. Weiss, Point/line correspondence under 2D projective transformation, Proceedings of the 11th International Conference on Pattern Recognition, The Hague, The Netherlands, 1992, pp. 399}402. [11] T.H. Reiss, Object recognition using algebraic and di!erential invariants, Signal Process. 32 (1993) 367}395. [12] D. Forsyth, J.L. Mundy, A. Zisserman, C. Coelho, A. Heller, C. Rothwell, Invariant descriptors for 3-D object recognition and pose, IEEE Trans. Pattern Anal. Mach. Intell. 10 (1987) 971}991. [13] S. Linnainmaa, D. Harwood, L.S. Davis, Pose determination for a three-dimensional object using triangle pairs, IEEE Trans. Pattern Anal. Mach. Intell. 5 (1988) 634}647. [14] J.L. Mundy, A. Zisserman (Eds.), Geometric Invariance in Computer Vision, MIT Press, Combridge, MA, 1992. [15] T. Suk, J. Flusser, Vertex-based features for recognition of projectively deformed polygons, Pattern Recognition 29 (1996) 361}367. [16] R. Lenz, P. Meer, Point con"guration invariants under simultaneous projective and permutation transformations, Pattern Recognition 27 (1994) 1523}1532. [17] N.S.V. Rao, W. Wu, C.W. Glover, Algorithms for recognizing planar polygonal con"gurations using perspective images, IEEE Trans. Robotics Automat. 8 (1992) 480}486.

Appendix A Sometimes a task on how to save and load information about combinations to and from a memory may be required to be solved. We have got the combinations of k elements from n and we can save this information in the following way: a"0 for i :"0; i (n 1 1 for i :"i #1; i (n 2 1 2 F for i :"i #1; i (n k k~1 k Mm[a]:"information (i , i ,2, i ) 1 2 k a:"a#1N When information about k-tuple (i , i ,2, i ) is re1 2 k quired, we need to compute the address a from the k-tuple. If we sort the indices by size so it holds i (i (2(i , 1 2 k then this address can be computed as

AB

a"

n

!1

k

A

BA B

n k k~j`1 # + + (!1)m`1 k!j!m#1 j/1 m/0

i #m j . m (41)

Appendix B p(I@ )"10.110488 ) IA 6!27.936483 ) I@5 1 1 1 #31.596612 ) I@4!16.504259 ) I@3 1 1 !0.32251158 ) I@ #3.0473587 ) I@ 1 1 !0.66901966. If I@ (0.475 then 1 d(I@ )"17.575974 ) I@4!16.423212 ) I@3#9.111527 ) I@2 1 1 1 1 !0.43942294 ) I@ #0.016542258 1 else d(I@ )"3.9630392 ) I@4!13.941518 ) I@3#21.672754 ) I@2 1 1 1 1 !17.304971 ) I@ #5.6198814. 1

T. Suk, J. Flusser / Pattern Recognition 33 (2000) 251}261 [18] P.J. Besl, N.D. McKay, A method for registration of 3-D shapes, IEEE Trans. Pattern Anal. Mach. Intell. 14 (1992) 239}256. [19] P. Meer, S. Ramakrishna, A. Lenz, Correspondence of coplanar features through P2-invariant representations,

261

Applications of Invariance in Computer Vision, Lecture Notes in Computer Science, vol. 825, Springer, Berlin 1993, pp. 473}492.

About the Author*TOMAD S[ SUK was born in Prague, Czech Republic, on April 30, 1964. He received the M.Sc. degree in Technical Cybernetics from the Czech Technical University, Prague, Czech Republic in 1987 and the Ph.D. degree in Computer Science from the Czechoslovak Academy of Sciences in 1992. Since 1987 he has been with the Institute of Information Theory and Automation, Academy of Sciences of the Czech Republic, Prague. His current research interests include image pre-processing, geometric invariants and remote sensing. He has authored or coauthored more than 20 scienti"c publications in these areas.

About the Author*JAN FLUSSER was born in Prague, Czech Republic, on April 30, 1962. He received the M.Sc. degree in Mathematical Engineering from the Czech Technical University, Prague, Czech Republic in 1985 and the Ph.D. degree in Computer Science from the Czechoslovak Academy of Sciences in 1990. Since 1985 he has been with the Institute of Information Theory and Automation, Academy of Sciences of the Czech Republic, Prague. Since 1995 he has been holding the position of a head of the Department of Image Processing. Since 1991 he has also been a$liated with the Faculty of Mathematics and Physics, Charles University, Prague, where he gives courses on Digital Image Processing. His current research interests include digital image processing, pattern recognition and remote sensing. He has authored or coauthored more than 30 scienti"c publications in these areas. Dr Flusser is a member of the Pattern Recognition Society, the IEEE Signal Processing Society, the IEEE Computer Society and the IEEE Geoscience and Remote Sensing Society.


Boundary detection by contextual non-linear smoothing Xiuwen Liu!,",*, DeLiang L. Wang!,#, J. Raul Ramirez" !Department of Computer and Information Science, The Ohio State University, 2015 Neil Avenue, Columbus, OH 43210, USA "Center for Mapping, The Ohio State University, 1216 Kinnear Road, Columbus, OH 43212, USA #Center for Cognitive Science, The Ohio State University, 1961 Tuttle Park Place, Columbus, OH 43210, USA Received 7 April 1998; received in revised form 16 November 1998; accepted 18 February 1999

Abstract In this paper we present a two-step boundary detection algorithm. The "rst step is a nonlinear smoothing algorithm which is based on an orientation-sensitive probability measure. By incorporating geometrical constraints through the coupling structure, we obtain a robust nonlinear smoothing algorithm, where many nonlinear algorithms can be derived as special cases. Even when noise is substantial, the proposed smoothing algorithm can still preserve salient boundaries. Compared with anisotropic di!usion approaches, the proposed nonlinear algorithm not only performs better in preserving boundaries but also has a non-uniform stable state, whereby reliable results are available within a "xed number of iterations independent of images. The second step is simply a Sobel edge detection algorithm without non-maximum suppression and hysteresis tracking. Due to the proposed nonlinear smoothing, salient boundaries are extracted e!ectively. Experimental results using synthetic and real images are provided. ( 1999 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Nonlinear smoothing; Contextual information; Anisotropic di!usion; Edge detection; Boundary detection

1. Introduction One of the fundamental tasks in low-level machine vision is to locate discontinuities in images corresponding to physical boundaries between a number of regions. A common practice is to identify local maxima in local gradients of images } collectively known as edge detection algorithms. The Sobel edge detector [1] consists of two 3]3 convolution kernels, which respond maximally to vertical and horizontal edges respectively. Local gradients are estimated by convolving the images with the two kernels, and thresholding is then applied to get rid of noisy responses. The Sobel edge detector is computationally e$cient but sensitive to noise. To make the estimation of gradients more reliable, the image can be convolved with a low-pass "lter before estimation and two

* Corresponding author. Tel.: #1-614-292-7402; fax: #1614-688-0066. E-mail address: [email protected] (X. Liu)

in#uential methods are due to Marr and Hildreth [2] and Canny [3]. By convolving the image with a Laplacian of Gaussian kernel, the resulting local maxima, which are assumed to correspond to meaningful edge points, are zero-crossings in the "ltered image [2]. Canny [3] derived an optimal step edge detector using variational techniques starting from some optimal criteria and used the "rst derivative of a Gaussian as a good approximation of the derived detector. Edge points are then identi"ed using a non-maximum suppression and hysteresis thresholding for better continuity of edges. As noticed by Marr and Hildreth [2], edges detected at a "xed scale are not su$cient and multiple scales are essentially needed in order to obtain good results. By formalizing the multiple scale approach, Witkin [4] and Koenderink [5] proposed Gaussian scale space. The original image is embedded in a family of gradually smoothed images controlled by a single parameter, which is equivalent to solving a heat equation with input as the initial condition [5]. While Gaussian scale space has nice properties and is widely used in machine vision [6], a major limitation is that Gaussian smoothing inevitably blurs edges and


264

X. Liu et al. / Pattern Recognition 33 (2000) 263}280

other important features due to its low-pass nature. To overcome the limitation, anisotropic di!usion, which was proposed by Cohen and Grossberg [7] in modeling the primary visual cortex, was formulated by Perona and Malik [8]:

ing spatial regularization [10] and edge-enhancing anisotropic di!usion [11], the general framework remains the same. As shown by You et al. [12], anisotropic di!usion given by Eq. (1) is the steepest gradient descent minimizer of the following energy function:

LI "div (g(DD+IDD)+I). Lt

E(I)"

(1)

Here div is the divergence operator, and g is a nonlinear monotonically decreasing function and +I denotes the gradient. By making the di!usion conductance dependent explicitly on local gradients, anisotropic di!usion prefers intra-region smoothing over inter-region smoothing, resulting in immediate localization while noise is reduced [8]. Because it produces visually impressive results, anisotropic di!usion generates much theoretical as well as practical interest (see Ref. [9] for a recent review). While many improvements have been proposed, includ-

P

f (DD+IDD) d)

(2)

X

with f @(DD+IDD) . g(DD+IDD)" DD+IDD Under some general conditions, the energy function given by Eq. (2) has a unique and trivial global minimum, where the image is constant everywhere, and thus interesting results exist only within a certain period of di!usion. An immediate problem is how to determine the termination time, which we refer to as the termination

Fig. 1. An example with non-uniform boundary gradients and substantial noise. (a) A noise-free synthetic image. Gray values in the image: 98 for the left &[' region, 138 for the square, 128 for the central oval, and 158 for the right &]' region. (b) A noisy version of (a) with Gaussian noise of p"40. (c) Local gradient map of (b) using the Sobel operators. (d)}(f) Smoothed images from an anisotropic di!usion algorithm [13] at 50, 100, and 1000 iterations. (g)}(i) Corresponding edge maps of (d)}(f), respectively, using the Sobel edge detector.


problem. While there are some heuristic rules on how to choose the stop time [10,11], in general it corresponds to the open problem of automatic scale selection. As in Gaussian scale space, a "xed time would not be su$cient to obtain good results. Another problem of anisotropic di!usion is that di!usion conductance is a deterministic function of local gradients, which, similar to non-maximum suppresion in edge detection algorithms, makes an implicit assumption that larger gradients are due to true boundaries. When noise is substantial and gradients due to noise and boundaries cannot be distinguished based on magnitudes, the approach tends to fail to preserve meaningful region boundaries. To illustrate the problems, Fig. 1a shows a noise-free image, where the gradient magnitudes along the central square change considerably. Fig. 1b shows a noisy version of Fig. 1a by adding Gaussian noise with zero mean and p"40, and Fig. 1c shows its local gradient magnitude obtained using Sobel operators [1]. While the three major regions in Fig. 1b may be perceived, Fig. 1c is very noisy and the strong boundary fragment is barely visible. Fig. 1d}f show the smoothed images by an anisotropic di!usion algorithm [13] with speci"ed numbers of iterations. Fig. 1g}i show the edge maps of Fig. 1d}f, respectively, using the Sobel edge detection algorithm. While at the 50th iteration the result is still noisy, the result becomes meaningless at the 1000th iteration. Even though the result at the 100th iteration is visually good, the boundaries are still fragmented and it is not clear how to identify a `gooda number of iterations automatically. These problems to a large extent are due to the assumption that local maxima in gradient images are good edge points. In other words, due to noise, responses from true boundaries and those from noise are not distinguishable based on magnitude. To overcome these problems, contextual information, i.e., responses from neighboring pixels, should be incorporated in order to reduce ambiguity as in relaxation labeling and related methods [14}17]. In general, relaxation labeling methods use pair-wise compatibility measure, which is determined based on a priori models associated with labels, and convergence is not known and often very slow in numerical simulations [18]. In this paper, by using an orientation-sensitive probability measure, we incorporate contextual information through the geometrical constraints on the coupling structure. Numerical simulations show that the resulting nonlinear algorithm has a non-uniform stable state and good results can be obtained within a "xed number of iterations independent of input images. Also, the oriented probability measure is de"ned on input data, and thus no a priori models need to be assumed. In Section 2, we formalize our contextual nonlinear smoothing algorithm and show that many nonlinear smoothing algorithms can be treated as special cases. Section 3 gives some theoretical results as well as numerical simulations regarding the stability and convergence

265

of the algorithm. Section 4 provides experimental results using synthetic and real images. Section 5 concludes the paper with further discussions.

2. Contextual nonlinear smoothing algorithm 2.1. Design of the algorithm To design a statistical algorithm, with no prior knowledge, we assume a Gaussian distribution within each region. That is, given a pixel (i , j ) and a window 0 0 R 0 0 at pixel (i , j ), consisting of a set of pixel locations, (i , j ) 0 0 we assume that

G

H

1 (I !k )2 R , P(I 0 0 , R)" exp ! (i0, j0) (3) (i , j ) 2p2 J2pp R R where I is the intensity value at pixel location (i, j). To (i, j) simplify notation, without confusion, we use R to stand for R 0 0 . Intuitively, P(I 0 0 , R) is a measure of com(i , j ) (i , j ) patibility between intensity value at pixel (i , j ) and 0 0 statistical distribution in window R. To estimate the unknown parameters of k and p , consider the pixels in R R R as n realizations of Eq. (3), where n"DRD. The likelihood function of k and p is [19] R R 1 n ¸(R; k , p )" R R J2pp R 1 ]exp ! + (I !k )2 . (4) (i, j) R 2p2 R (i, j)|R By maximizing Eq. (4), we get the maximum likelihood estimators for k and p : R R 1 k( " + I , (5a) R n (i, j) (i, j)|R 1 J + (I !k( )2. (5b) p( " (i, j) R R Jn (i, j)|R To do a nonlinear smoothing, similar to selective smoothing "lters [20,21], suppose that there are M windows R(m), where 1)m)M, around a central pixel (i , j ). Here these R(m)'s can be generated from one or 0 0 several basis windows through rotation, which are motivated by the experimental "ndings of orientation selectivity in the visual cortex [22]. Simple examples are elongated rectangular windows (refer to Fig. 6), which are used throughout this paper for synthetic and real images. The probability that pixel (i , j ) belongs to R(m) can be 0 0 estimated from Eqs. (3), (5a) and (5b). By assuming that the weight of each R(m) should be proportional to the probability, as in relaxation labeling [14,15], we obtain an iterative nonlinear smoothing "lter:

A

A

B

+ P(It , R(m))k( t (m) R It`1 " m (i0, j0) (i0, j0) + P(It 0 0 , R(m)) m (i , j )

B

(6)

266


A problem with this "lter is that it is not sensitive to weak edges due to the linear combination. To generate more semantically meaningful results and increase the sensitivity even to weak edges, we apply a nonlinear function on weights, which is essentially same as anisotropic di!usion [8]: + g(P(It 0 0 , R(m)))k( t (m) (i , j ) R . It`1 " m (7) (i0, j0) + g(P(It 0 0 , R(m))) (i , j ) m Here g1 is a nonlinear monotonically increasing function. A good choice for g is an exponential function, which is widely used in nonlinear smoothing and anisotropic diffusion approaches: g(x)"exp(x2/K)

(8)

Because k(m) is a linear combination of random variables with a Gaussian distribution, k(m) has also a Gaussian distribution with the same mean and a standard deviation given by 1 p (m)" p (m). k JDR(m)D R

This provides a probability measure of how likely that the M windows are sampled from one homogenous region. Given a con"dence level a, for each pair of windows R(m1) and R(m2) we have Dk(m1)!k(m2)D

AS

)min Here parameter K controls the sensitivity to edges [23]. Eq. (7) provides a generic model for a wide range of nonlinear algorithms, the behavior of which largely depends on the sensitivity parameter K. When K is large, Eq. (7) reduces to the equally weighted average smoothing "lter. When K is around 0.3, g is close to a linear function in [0, 1] and Eq. (7) then reduces to Eq. (6). When K is a small positive number, Eq. (7) will be sensitive to all discontinuities. No matter how small the weight of one window can be, theoretically speaking, if it is nonzero, when tPR, the system will reach a uniform stable state. Similar to anisotropic di!usion approaches, the desired results will be time-dependent and the termination problem becomes a critical issue for autonomous solutions. To overcome this limitation, we restrict smoothing only within the window with the highest probability similar to selective smoothing [20,21]: (9) mH" max (P(I 0 0 , R(m))). (i , j ) 1xmxM The nonlinear smoothing through Eq. (9) is desirable in regions that are close to edges. By using appropriate R(m)'s, Eq. (9) encodes discontinuities implicitly. But in homogenous regions, Eq. (9) may produce arti"cial block e!ects due to intensity variations. Under the proposed statistical formulation, there is an adaptive method to detect homogeneity. Based on the assumption that there are M R(m) windows around a central pixel (i , j ), where 0 0 each window has a Gaussian distribution, consider the mean in each window as a new random variable: 1 k(m)" DR(m)D

+ I . (i, j) (i, j)|R(m)

(10)

1 Because the probability measure given by (1) is inversely related to gradient measure used in most non-linear smoothing algorithms, (8) is an increasing function instead of a decreasing function in our method.

(11)

log(1/a) p( (m1), DR(m1)D R

S

B

log(1/a) p( (m2) . DR(m2)D R

(12)

If all the pairs satisfy Eq. (12), the M windows are likely from one homogenous region with con"dence a. Intuitively, under the assumption of a Gaussian distribution, when we have more samples, i.e., the window R(m) is larger, the estimation of the mean is more precise and so the threshold should be smaller. In a region with a larger standard deviation, the threshold should be larger because larger variations are allowed. The nonlinear smoothing algorithm outlined above works well when noise is not very large. In cases when signal to noise ratio is very low, the probability measure given in Eq. (3) would be unreliable because pixel values change considerably. This problem can be alleviated by using the mean value of pixels sampled from R which are close to the central pixel (i , j ), or along a certain direc0 0 tion to make the algorithm more orientation sensitive. To summarize, we obtain a nonlinear smoothing algorithm. We de"ne M oriented windows which can be obtained by rotating one or more basis windows. At each pixel, we estimate parameters using Eqs. (5a) and (5b). If all the M windows belong to a homogenous region according to Eq. (12), we do the smoothing using all the M windows. Otherwise, the smoothing is done only within the most compatible window given by Eq. (9). 2.2. A generic nonlinear smoothing framework In this section we will show how to derive several widely used nonlinear algorithms from the nonlinear smoothing algorithm outlined above. Several early nonlinear "lters [20,21] do the smoothing in a window where the standard deviation is the smallest. These "lters can be obtained by simplifying Eq. (3) to 1 C P(I 0 0 , k( , p( )" (i , j ) J2pp(

(13)

where C is a constant. Then the solution to Eq. (9) is the window with the smallest deviation. Recently, Higgins


and Hsu [24] extended the principle of choosing the window with the smallest deviation for edge detection. Another nonlinear smoothing "lter is the gradientinverse "lter [25]. Suppose that there is one window, i.e., M"1, consisting of the central pixel (i , j ) itself only, 0 0 the estimated deviation for a given pixel (i, j) in Eq. (5b) now becomes p( "DI !I 0 0 D. (i, j) (i , j )

(14)

Eq. (14) is a popular way to estimate local gradients. Using Eq. (13) as the probability measure, Eq. (6) becomes exactly the gradient inverse nonlinear smoothing "lter [25]. Smallest Univalue Segment Assimilating Nucleus (SUSAN) nonlinear smoothing "lter [26] is proposed based on SUSAN principle. It is formulated as =(i , j , di, dj) + It 0 0 , It`1 " (di, dj)E(0,0) (i0`di, j0`dj) (i0, j0) + =(i , j , di, dj) (di, dj)E(0,0) 0 0

(15)

where

A

B

r2 (It !It 0 0 )2 (i , j ) . =(i , j , di, dj)"exp ! ! (i0`di, j0`dj) 0 0 2p2 ¹2 Here (i , j ) is the central pixel under consideration, and 0 0 (di, dj) de"nes a local neighborhood. Essentially, it integrates Gaussian smoothing in spatial and brightness domains. The parameter ¹ is a threshold for intensity values. It is easy to see from Eq. (15) that the weights are derived based on pair-wise intensity value di!erences. It would be expected that the SUSAN "lter performs well when images consist of relatively homogenous regions and within each region noise is smaller than ¹. When noise is substantial, it fails to preserve structures due to the pair-wise di!erence calculation, where no geometrical constraints are incorporated. This is consistent with the experimental results, which will be discussed later. To get the SUSAN "lter, we de"ne one window including the central pixel itself only. For a given pixel (i, j) in its neighborhood, Eq. (3) can be simpli"ed to

G

H

(I !k( )2 R , P(I , R)"C exp ! (i, j) (i, j) ¹2

If we have four singleton regions, Eq. (17) is essentially a simpli"ed version of Eq. (7) with an adaptive learning rate.

3. Analysis 3.1. Theoretical results One of the distinctive characteristics of the proposed algorithm is that it requires spatial constraints among responses from neighboring locations through coupling structure as opposed to pair-wise coupling structure. Fig. 2 illustrates the concept using a manually constructed example. Fig. 2a shows the oriented windows in a 3]3 neighborhood, and Fig. 2b shows the coupling structure if we apply the proposed algorithm to a small image patch. The directed graph is constructed as follows. There is a directed edge from (i , j ) to (i , j ) if and 1 1 0 0 only if (i , j ) contributes to the smoothing of (i , j ) 1 1 0 0 according to Eqs. (12) and (9). By doing so, the coupling structure is represented as a directed graph as shown in Fig. 2b. Connected components and strongly connected components [27] of the directed graph can be used to analyze the temporal behavior of the proposed algorithm. A strongly connected component is a set of vertices, or pixels here, where there is a directed path from any vertex to all the other vertices in the set. We obtain a connected component if we do not consider the direction of edges along a path. In the example shown in Fig. 2b, all the black pixels form a strongly connected component and so do all the white pixels. Also, there are obviously two connected components. Essentially our nonlinear smoothing algorithm can be viewed as a discrete dynamic system, the behavior of which is complex due to spatial constraints imposed by coupling windows and adaptive coupling structure by probabilistic grouping. We now prove that a constant region satisfying certain geometrical constraints is a stable state of the smoothing algorithm. Theorem. If a region S of a given image I satisxes

(16)

where C is a scaling factor. Because now k( is I 0 0 , R (i , j ) Eq. (6) with the probability measure given by Eq. (16) is equivalent to Gaussian smoothing in the brightness domain in Eq. (15). Now consider anisotropic di!usion given by Eq. (1). By discretizing Eq. (1) in image domain with four nearest-neighbor coupling [13] and rearranging terms, we have It`1 "gt It #j+ g(P(It , R(m)))k( t (m). (i, j) (i, j) (i, j) (i, j) R m

267

(17)

(18a) (i , j )3S and (i , j )3SNI 1 1 "I 2 2 (i , j ) 1 1 2 2 (i , j ) ∀(i, j)3SN& m R(m) -S, (18b) (i, j) then S is stable with respect to the proposed algorithm. Proof. Condition (18a) states that S is a constant region and the standard deviation is zero if R(m) is within S according to Eq. (5b). Consider a pixel (i , j ) in S. Inequal0 0 ity (12) is satis"ed only when all R(m)'s are within S. In this case, the smoothing algorithm does not change the intensity value at (i , j ). Otherwise, R(mH) according to Eq. (9) 0 0 must be within S because there exists at least one such window according to Eq. (18b) and thus the smoothing

268


Fig. 2. Illustration of the coupling structure of the proposed algorithm. (a) Eight oriented windows and a fully connected window de"ned on a 3]3 neighborhood. (b) The resulting coupling structure. There is a directed edge from (i , j ) to a neighbor (i , j ) if and 1 1 0 0 only if (i , j ) contributes to the smoothing of (i , j ) according to Eqs. (12) and (9). Filled circles represent black pixels, empty circles 1 1 0 0 represent white pixels, and hatched circles represent gray pixels. Ties in Eq. (9) are broken according to left-right and top-down preference of the oriented windows in (a).

algorithm does not change the intensity value at (i , j ) 0 0 also. So S is stable. h A maximum connected component of the constructed graph is stable when its pixels are constant and thus maximum connected components of the constructed graph are a piecewise constant stable solution of the proposed algorithm. For the image patch given in Fig. 2b, for example, a stable solution is that pixels in each of the two connected components are constant. The noise-free image in Fig. 1a is also a stable solution by itself, as we will demonstrate through numerical simulations later on.

It is easy to see from the proof that any region which satis"es conditions (18a) and (18b) during temporal evolution will stay unchanged. In addition, due to the smoothing nature of the algorithm, a local maximum at iteration t cannot increase according to the smoothing kernel by Eq. (12) or Eq. (9), and similarly, a local minimum cannot decrease. We conjecture that any given image approaches an image that is almost covered by homogenous regions. Due to the spatial constraints given by Eq. (18b), it is not clear if the entire image converges to a piece-wise constant stable state. Within each resulting homogenous region, Eq. (18b) is satis"ed and thus the region becomes stable. For pixels near boundaries,


269

Fig. 3. Temporal behavior of the proposed algorithm with respect to the amount of noise. Six noisy images are obtained by adding zero-mean Gaussian noise with p of 5, 10, 20, 30, 40 and 60, respectively, to the noise-free image shown in Fig. 1a. The plot shows the deviation from the ground truth image with respect to iterations of the noise-free image and six noisy images.

corners, and junctions, it is possible that Eq. (18b) is not uniquely satis"ed within one constant region, and small changes may persist. The whole image in this case attains a quasi-equilibrium state. This is supported by the following numerical simulations using synthetic and real images. While there are pixels which do not converge within 1000 iterations, the smoothed image as a whole does not change noticeably at all. The two maximum strongly connected components in Fig. 2b satisfy condition (18b). Both of them are actually uniform regions and thus are stable. Gray pixels would be grouped into one of the two stable regions according to pixel value similarity and spatial constraints. 3.2. Numerical simulations Because it is di$cult to derive the speed of convergence analytically, we use numerical simulations to demonstrate the temporal behavior of the proposed algorithm. Since smoothing is achieved using equally weighted average within selected windows, the algorithm should converge rather quickly in homogenous regions. To obtain quantitative estimations, we de"ne two measures similar to variance. For synthetic images, where a noise-free image is available, we de"ne the deviation from the ground truth image as

S

D " (I)

+ + (I !Igt )2 i j (i, j) (i, j) . DID

(19)

Here I is the image to be measured and Igt is the ground truth image. The deviation gives an objective measure of how good the smoothed image is with respect to the true image. To measure the convergence, we de"ne relative variance for image I at time t:

S

. A set function g de"ned on B is a fuzzy measure if it satis"es the following three axioms: 1. Boundary conditions: g(0)"0, g(>)"1.

T. Pham, M. Wagner / Pattern Recognition 33 (2000) 309}315

2. Monotonicity: g(A))g(B) if ALB, andA, B3B. 3. Continuity: lim g(A )"g(lim A ) if A 3B and i?= i i?= i i MA N is monotone (an increasing sequence of measuri able sets). A g -fuzzy measure is also proposed by Sugeno [11] j which satis"es another condition known as the j-rule (j'!1): g(AXB)"g(A)#g(B)#jg(A)g(B), A, BL>, and AWB"0. It is noted that when j"0, the g -fuzzy measure j becomes a probability measure [14]. In general, the value of the constant j can be determined by the properties of the g -fuzzy measure as follows. j Let >"My , y ,2, y N. If the fuzzy density of the 1 2 m g -fuzzy measure is de"ned as a function j g : y 3>P[0, 1] such that g "g (My N), i"1, 2,2, m, i i j i then the g -fuzzy measure of a "nite set can be obtained j as [15] g (>) j n m~1 n " + g #j + + g 1g 2#2#jm~1g g 2g . i i i 1 2 m i/1 i1/1 i2/i1`1 (7) Provided that jO0, Eq. (7) can be rewritten as

C

D

1 m g (>)" < (1#jg )!1 . (8) j i j i/1 With boundary condition g(>)"1, the constant j can be determined by solving the following equation: m j#1" < (1#jg ). (9) i i/1 It has been proved [16] that for a "xed set of g , i 0(g (1, there exists a unique root of j'!1, and i jO0, using Eq. (9). And also from Eq. (9) it can be seen that if the values of g are known, then j can be cali culated.

311

> are rearranged to make this relation hold), the Sugeno integral can be computed by

P

m f (y)"g ( ) )"max [min( f (y ), g(A ))], i i i/1 A

(11)

where A "My , y ,2, y N, and g(A ) can be recursively i i i`1 m i calculated in terms of the g -fuzzy measure as [14] j g(A )"g #g(A )#jg g(A ), 1(i)m. i i i~1 i i~1

It can be seen that the above de"nition is not a proper extension of the usual Lebesgue integral, which is not recovered when the measure is additive. In order to overcome this drawback, the so-called Choquet integral was proposed by Murofushi and Sugeno [17]. The Choquet integral of f with respect to a fuzzy measure g is de"ned as follows:

P

m f (y) dg( ) )" + [ f (y )!f (y )]g(A ) i i~1 i A i/1

(13)

in which f (y )"0. 0 To help further understand the concepts of fuzzy measures and fuzzy integrals, let us consider a simple example as follows [18,19]: Let >"My , y , y N, and 1 2 3 given that the fuzzy densities are g "0.34, 1 g "0.32, g "0.33. Using Eq. (9), we obtain the quad2 3 ratic equation 0.0359j2#0.3266j !0.001"0. The parameter j can be obtained by taking the unique root j'!1, which gives j"0.0305. Also using Eq. (9), we can calculate all the fuzzy measures on the power subsets of >, whose values are shown in Table 1. Suppose that f (y )"0.6, f (y )"0.7, and f (y )"0.1. Thus, we need 1 2 3 to rearrange the elements in >, which yields g " 1 0.33, g "0.34, and g "0.32 in order to satisfy f (y )" 2 3 1 0.1(f (y )"0.6(f (y )"0.7. Using Eq. (11), the 2 3 Sugeno integral is computed as: max[min(0.1, 1), min(0.6, 0.66), min(0.7, 0.32)]"0.6. Using Eq. (13), the Choquet integral is obtained as (0.1!0)(1.0)# (0.6!0.1)(0.66)#(0.7!0.6)(0.32)"0.462.

3.2. Fuzzy integral Let (>, B, g) be a fuzzy measure space and f : >P[0, 1] be a B-measurable function. A fuzzy integral over AL> of the function f with respect to a fuzzy measure g is de"ned by

P

f (y) " g( ) )e" sup [min(a, g(f ))], (10) a A a|*0, 1+ where f is the a level set of f, f "My : f (y)*aN. a a The fuzzy integral in Eq. (10) is called the Sugeno integral. When >"My , y , 2, y N is a "nite set, and 1 2 n 0)f (y ))f (y )2)f (y ))1, (if not, the elements of 1 2 n

(12)

Table 1 g measures on the power set of > j Subset A

g (A) j

0 My N 1 My N 2 My N 3 My , y N 1 2 My , y N 2 3 My , y N 1 3 My , y , y N 1 2 3

0 0.34 0.32 0.33 0.6633 0.6532 0.6734 1

312


4. Fuzzy-fusion based normalization It has been mentioned in the foregoing sections that the a priori probability of an utterance given that it is from one of the cohort speakers is assumed to be equal in the conventional similarity normalization methods, we use the concept of the fuzzy measure to calculate the grades of similarity or closeness between each cohort speaker model and the client model, i.e. the fuzzy density, and the multi-attributes of these fuzzy densities. The "nal score for the normalization of the cohort speakers can then be determined by combining all of these fuzzy measures with the corresponding likelihood values using the Choquet integral. We express the proposed model in mathematical terms as log ¸(X)"log p(XDS)!log F(XDS@),

(14)

where F(XDS@) is the fuzzy integral of the likelihood values of an utterance X coming from the cohort speaker set S@"MS : b"1, 2,2, BN with respect to the b fuzzy measures of speaker similarity. It is de"ned as follows: B F(XDS@)" + [p(XDS )!p(XDS )]g(Z DS), b b~1 b b/1

(15)

where p(XDS ) has been previously de"ned, Z " b b MS , S , 2, S N, g(Z DS) is the fuzzy measure of Z b b`1 B b b given that the true speaker is S, p(XDS )"0, and the 0 relation 0)p(XDS ))p(XDS ),2, p(XDS ) holds, other1 2 B wise the elements in S@ need to be rearranged. From the previous presentation of the fuzzy measure and the fuzzy integral, it is noticed that the key factor for the fuzzy fusion process is the fuzzy density. If the fuzzy densities can be determined then the fuzzy measures can be identi"ed, which make it ready for the operation of the fuzzy integral. For the fusion of similarity measures, we consider the fuzzy density as the degree of similarity or closeness between the acoustic model of a cohort speaker and that of the client, i.e. the greater the value of the fuzzy density is, the closer the two models are. Therefore, we de"ne the fuzzy density as g(S DS)"1!exp(!aEvo !vo E2), b b S

(16)

where a is a positive constant, E ) E2 is the Euclidean norm which indicates the root-mean-square averaging process, vo is the mean code vector of a cohort speaker S , and vo b b S is the mean code vector of the client speaker S. It is reasonable to assume that some acoustic models of a cohort speaker, say S , may be more similar to those 1 of the client speaker S than those of another cohort speaker, say S . However, some other acoustic models of 2 S may be more similar to those of S than those of S . 2 1 Since the mean code vectors are globally generated from the codebooks including all di!erent utterances of the

speakers, we therefore introduce the constant a in Eq. (16) for each cohort speaker in order to "ne-tune the fuzzy density with respect to the Euclidean distance measure. At present we select the values of a by means of the training data and will further discuss this issue in the experimental section.

5. Experiments 5.1. Measure of performance One of the most common performance measures for speaker veri"cation systems is the equal error rate (EER) which applies an a posteriori threshold to make the false acceptance error rate equal to the false rejection error rate. If the score of an identity claim is above a certain threshold then it is veri"ed as the true speaker, otherwise the claim is rejected. If the threshold is set high then there is a risk of rejecting a true speaker. On the contrary, if the threshold is set low then there is a risk of accepting an impostor. In order to balance the trade-o! between these two situations, the threshold is selected at a level which makes the percentage of the false acceptance error and the false rejection error equal based on the distributions of client and impostor scores. Thus, the EER o!ers an e$cient way for measuring the degree of separation between the client and the impostor models. Using the EER as an indicator of system performance means that the smaller the EER is, the higher the performance is. 5.2. The database The commercial TI46 speech data corpus is used here for the experiments. The TI46 corpus contains 46 utterances spoken repeatedly by 8 female and 8 male speakers, labeled f1}f8 and m1}m8, respectively. The vocabulary contains a set of 10 computer commands: Menter, erase, go, help, no, rubout, repeat, stop, start, yesN. Each speaker repeated the words 10 times in a single training session, and then again twice in each of 8 testing sessions. The corpus is sampled at 12,500 samples/s and 12 bits/sample. The data were processed in 20.48 ms frames at a frame rate at 125 frames/s. The frames were Hamming windowed and preemphasized with k"0.9. Fortysix mel-spectral bands of a width of 110 mel and 20 mel-frequency cepstral coe$cients (MFCC) were determined for each frame. In the training session, each speaker's 100 training tokens (10 utterances]1 training session]10 repetitions) were used to train the speaker-based VQ codebook by clustering the set of all the speakers' MFCC into codebooks of 32, 64 and 128 codewords using the LBG algorithm [20].


5.3. The results The veri"cation was tested in the text-dependent mode. Since both the geometric mean and the fuzzy fusion methods operate on the principle of integration and depend on the size of the cohort set, we therefore compare the performances of these two methods. This is a closed set test as the cohort speakers in the trainig are the same as those in the testing. For the purpose of comparison and due to a limited number of speakers, we select for each claimed speaker a cohort set of three (same gender) whose acoustic models are closest to the claimed model. In the testing mode, each cohort speaker's 160 test tokens (10 utterances]8 testing sessions]2 repetitions) are tested against each claimed speakers' 10-word models. To identify the fuzzy densities for the cohort speakers, we select the values of a by means of the training data. The range of a was speci"ed to be from 1 to 50, and a unit step size was applied in the incremental trial process. It was observed that using di!erent values of a for di!erent speakers could give more reduction in the equal error rates. However, as an intial investigation we chose the same value for each gender set, that is a"10 for the

Table 2 Equal error rates (%EERs) for the 16 speakers using geometric mean (GM) fuzzy fusion (FF) based normalization methods GM

FF

Codebook size

Codebook size

Speaker

32

64

128

32

64

128

f1 f2 f3 f4 f5 f6 f7 f8 m1 m2 m3 m4 m5 m6 m7 m8

4.17 5.98 9.90 0.00 1.78 6.67 7.38 12.76 3.07 4.17 7.03 10.77 2.70 8.43 7.18 1.83

3.01 1.19 5.66 0.00 1.78 3.01 4.32 9.73 3.05 1.28 7.00 8.28 2.44 7.44 5.88 3.01

2.40 1.79 3.67 0.00 0.59 1.80 3.61 9.22 3.06 1.22 6.32 7.90 1.80 6.53 4.83 2.40

1.80 1.19 7.79 0.00 1.19 2.41 6.48 10.05 3.03 3.14 6.87 8.29 1.62 7.53 6.86 1.80

1.19 0.60 3.70 0.00 0.60 0.59 4.00 8.22 3.03 1.22 6.85 6.89 0.63 5.47 4.88 1.21

1.19 1.20 2.33 0.00 0.00 0.00 2.30 7.62 2.43 1.22 5.92 6.91 1.19 4.72 3.65 1.19

Female

6.08

3.66

2.89

3.50

2.48

1.92

Male

5.65

4.80

4.17

4.89

3.86

3.40

Average

5.87

4.23

3.53

4.20

3.17

2.66

313

female cohort set and a"1 for the male cohort set. As a result, Table 2 shows the mean equal-error rates for the 16 speakers with three codebook sizes of 32, 64 and 128 entries. For the veri"cation of the female speakers, using the fuzzy fusion the average EERs are reduced to (6.08!3.50)"2.58%, (3.66!2.48)"1.18%, (2.89! 1.92)"0.97% for the codebook sizes of 32, 64 and 128, respectively, For the model of the male speakers, the average EERs are reduced to (5.65!4.89)"0.76%, (4.80!3.86)"0.94%, (4.17!3.40)"0.77% for the codebook sizes of 32, 64 and 128, respectively, The total average EER reductions in both models for the three codebook sizes of 32, 64 and 128 are (5.87!4.20)" 1.67%, (4.23!3.17)"1.06%, (3.53!2.66)"0.87%, respectively. Through these results, it can be seen that the speaker veri"cation system using the fuzzy fusion is more favorable than using the geometric mean method.

6. Conclusions A fusion algorithm based on the fuzzy integral has been proposed and implemented in the similarity normalization for speaker veri"cation. Then the experimental results show that the application of the proposed method is superior to that of the conventional normalization. The key di!erence between the two methods is that the assumption of equal a priori probabilities is not necessary for the fuzzy integral-based normalization due to the concept of the fuzzy measure. In fact, applications of fuzzy measures and fuzzy integrals have been attracting great attention among researchers in the "eld of pattern recognition [21}24]. Two useful aspects of fuzzy measures are that the importance and interaction of features are taken into account, and fuzzy integrals serve as a basis for modeling these representations [25]. For this problem of speaker recognition, we interpret the importance of features as the similarity between the acoustic models of cohort and client speakers. There are three kinds of interaction: redundancy, complementarity, and independency. The "rst interaction is meant by that the scoring of the cohort models do not increase signi"cantly if the joint similarity is not greater than the sum of individual similarities. The second type is the converse, that is the scoring is increased signi"cantly when the joint similarity is greater than the sum of individual similarities. The last type indicates that each similarity measure contributes to the total scoring process. The complexity involves in the proposed method is the determination of the fuzzy densities and the computation of the fuzzy integrals that require more computational e!ort than the conventional method. However, the di!erence in computer running time between the two methods was found to be negligible. One important issue arising here for further investigation is the optimal identi"cation

314


of the fuzzy densities, which can o!er #exibility and have great e!ect in the fuzzy fusion. At present, the fuzzy densities were determined based on a rough estimate of the values for a using a small range of integers. One convenient and promising method for "nding such a solution is the optimizing process of the genetic algorithms which are random-search algorithms based on the principles of natural genetics, and have attracted great attention as function optimizers. Using genetic algorithms, the fuzzy densities can be identi"ed in such a way that the error for the training data is minimized in the least-squares sense. Some typical similar problems in data fusion which have been sucessfully tackled by genetic algorithms can be found in Ref. [26,27].

References [1] S. Furui, An overview of speaker recognition technology, Proceedings of Workshop on Automatic Speaker Recognition, Identi"cation and Veri"cation, Martigny, Switzerland, 1994, pp. 1}9. [2] J.P. Campbell, Speaker recognition: a tutorial, Proc. IEEE 85 (1997) 1437}1462. [3] G.R. Doddington, Speaker recognition evaluation methodology } an overview and perspective, Proceedings of Workshop on Speaker Recognition and its Commercial and Forensic Applications (RLA2C), Avignon (France), 1998, pp. 60}66. [4] B.S. Atal, E!ective of linear prediction characteristics of speech wave for automatic speaker identi"cation and veri"cation, J. Acoust. Soc. Am. 55 (1974) 1304}1312. [5] S. Furui, Cepstral analysis techniques for automatic speaker veri"cation, IEEE Trans. Acoust. Speech Signal Process. 29 (1981) 254}272. [6] A.L. Higgins, L. Bahler, J. Porter, Speaker veri"cation using randomnized phrase prompting, Digital Signal Processing 1 (1991) 89}106. [7] T. Matsui, S. Furui, Concatenated phoneme models for text variable speaker recognition, Proceedings of IEEE International of Conference Acoustics, Speech, and Signal Processing, Minneapolis, USA, 1993, pp. 391}394. [8] G. Gravier, G. Chollet, Comparison of normalization techniques for speaker veri"cation, Proceedings of Workshop on Speaker Recognition and its Commercial and Forensic Applications (RLA2C), Avignon, France, 1998, pp. 97}100. [9] C.S. Liu, H.C. Wang, C.H. Lee, Speaker veri"cation using normalization log-likelihood score, IEEE Trans. Speech Audio Process. 4 (1996) 56}60. [10] L.A. Zadeh, Fuzzy sets, Inform. and Controls 8 (1965) 338}353.

[11] M. Sugeno, Fuzzy measures and fuzzy integrals } a survey, in: M.M. Gupta, G.N. Saridis, B.R. Gaines (Eds.), Fuzzy Automata and Decision Processes, North-Holland, Amsterdam, 1977, pp. 89}102. [12] Z. Wang, G.J. Klir, Fuzzy Measure Theory, Plenum Press, New York, 1992. [13] M. Grabisch, II.T. Nguyen, E.A. Walker, Fundamentals of Uncertainty Calculi with Applications to Fuzzy Inference, Kluwer Academic Publishers, Dordrecht, Netherland, 1995. [14] G. Banon, Distinction between several subsets of fuzzy measures, Fuzzy Sets and Systems 5 (1981) 291}305. [15] K. Leszczynski, P. Penczek, W. Grochulski, Sugeno's fuzzy measure and fuzzy clustering, Fuzzy Sets and Systems 15 (1985) 147}158. [16] H. Tahani, J.M. Keller, Information fusion in computer vision using the fuzzy integral, IEEE Trans. Systems Man Cybernet 20 (1990) 733}741. [17] T. Murofushi, M. Sugeno, An interpretation of fuzzy measure and the Choquet integral as an integral with respect to a fuzzy measure, Fuzzy Sets and Systems 29 (1989) 201}227. [18] T.D. Pham, H. Yan, A kriging fuzzy integral, Inform. Sci. 98 (1997) 157}173. [19] S.B. Cho, On-line handwriting recognition with neuralfuzzy method, Proc. IEEE FUZZ-IEEE/IFES'95, Yokohama, Japan, 1995, pp. 1131}1136. [20] Y. Linde, A. Buzo, R.M. Gray, An algorithm for vector quantization, IEEE Trans. Commun. 28 (1980) 84}95. [21] J.M. Keller, P. Gader, H. Tahani, J.H. Chiang, M. Mohamed, Advances in fuzzy integration for pattern recognition, Fuzzy Sets and Systems 65 (1994) 273}283. [22] M. Grabisch, J.M. Nicolas, Classi"cation by fuzzy integral: performance and tests, Fuzzy Sets and Systems 65 (1994) 255}271. [23] S.B. Cho, J.H. Kim, Combining multiple neural networks by fuzzy integral for robust classi"cation, IEEE Trans. Systems Man Cybernet 25 (1995) 380}384. [24] Z. Chi, H. Yan, T. Pham, Fuzzy Algorithms with Applications in Image Processing and Pattern Recognition, World Scienti"c, Singapore, 1996. [25] M. Grabisch, The representation of importance and interaction of features by fuzzy measures, Pattern Recognition Lett. 17 (1996) 567}575. [26] A.L. Buczak, R.E. Uhrig, Information fusion by fuzzy set operation and genetic algorithms, Simulation 65 (1995) 52}66. [27] T.D. Pham, H. Yan, Fusion of handwritten numeral classi"ers based on fuzzy and genetic algorithms, Proceedings North America Fuzzy Information Processing Society (NAFIPS)'97, New York, USA, 1997, pp. 257}262.

About the Author*TUAN D. PHAM received the B.E. degree (1990) in Civil Engineering from the University of Wollongong, the Ph.D. degree (1995) in Civil Engineering, with a thesis on fuzzy-set modeling in the "nite element analysis of engineering problems, from the University of New South Wales. From 1994 to 1995, he was a senior systems analyst with Engineering Computer Services Ltd, and from 1996 to early 1997 he was a post-doctoral fellow with the Laboratory for Imaging Science and Engineering in the Department of Electrical Engineering at the University of Sydney. From 1997 to 1998 he held a research fellow position with the Laboratory for Human-Computer Communication in the Faculty of Information Sciences and Engineering at the University of Canberra, and he is


315

now a lecturer in the School of Computing in the same Faculty. He is a co-author of 2 monographs, author and co-author of over 40 technical papers published in popular journals and conferences. His main research interests include the applications of computational intelligence and statistical techniques to pattern recognition, particularly in image processing, speech and speaker recognition. Dr. Pham is a member of the IEEE.

About the Author*MICHAEL WAGNER received a Diplomphysiker degree from the University of Munich in 1973 and a Ph.D. in Computer Science from the Australian National University in 1979 with a thesis on learning networks for speaker recognition. Dr. Wagner has been involved in speech and speaker recognition research since and has held research and teaching positions at the Technical University of Munich, National University of Singapore, University of Wollongong, University of New South Wales and the Australian National University. He was the Foundation President of the Australian Speech Science and Technology Association from 1986 to 1992 and is currently a professor and head of the School of Computing at the University of Canberra. Dr. Michael Wagner is a fellow of IEAust and a member of ASSTA, ESCA and IEEE.


E!ect of resolution and image quality on combined optical and neural network "ngerprint matching C.L. Wilson*, C.I. Watson, E.G. Paek Information Technology Laboratory, National Institute of Standards and Technology, Stop 8940, Gaithersburg, MD 20899-8940, USA Received 20 July 1998; accepted 3 February 1999

Abstract This paper presents results on direct optical matching, using Fourier transforms and neural networks for matching "ngerprints for authentication. Direct optical correlations and hybrid optical neural network correlation are used in the matching system. The test samples used in the experiments are the "ngerprints taken from NIST database SD-9. These images, in both binary and gray-level forms, are stored in a VanderLugt correlator (A. VanderLugt, Signal detection by complex spatial "ltering, IEEE Trans. Inform. Theory IT-10 (1964) 139}145). Tests of typical cross correlations and autocorrelation sensitivity for both binary and 8 bit gray images are presented. When Fourier transform (FT) correlations are used to generate features that are localized to parts of each "ngerprint and combined using a neural network classi"cation network and separate class-by-class matching networks, 90.9% matching accuracy is obtained on a test set of 200,000 image pairs. These results are obtained on images using 512 pixel resolution. The e!ect of image quality and resolution are tested using 256 and 128 pixel images, and yield accuracy of 89.3 and 88.7%. The 128-pixel images show only ridge #ow and have no reliably detectable ridge endings or bifurcations and are therefore not suitable for minutia matching. This demonstrates that Fourier transform matching and neural networks can be used to match "ngerprints which have too low image quality to be matched using minutia-based methods. Since more than 258,000 images were used to test each hybrid system, this is the largest test to date of FT matching for "ngerprints. Published by Elsevier Science Ltd. Keywords: Fingerprints; Matching; Optical correlation; Neural networks; Image quality

1. Introduction This paper presents data on inked "ngerprint images matched with optical and hybrid optical neural network correlators. The matching method is tested on an authentication application. The inked "ngerprint images are rolled prints scanned at 20 pixels/mm on a 4 cm]4 cm area of a "ngerprint card. We study matching of the inked "ngerprints using global optical correlations [1] and partial optical correlation features and a system of neural classi"cation and matching networks [2]. Images of three di!erent resolutions are tested to determine the

* Corresponding author. Tel.: #301-975-2080; fax: #301975-5287. E-mail address: [email protected] (C.L. Wilson) 0031-3203/99/$20.00 Published by Elsevier Science Ltd. PII: S 0 0 3 1 - 3 2 0 3 ( 9 9 ) 0 0 0 5 2 - 7

e!ect of image resolution and quality on matching accuracy. For images of inked rolled "ngerprints, even after core alignment and correction for rotation, optical matching of most prints is successful for matching the original image and rejecting other "ngerprints, but fails on second copies of inked rolled images because plastic pressure distortions and image size variation are too large to allow global Fourier transform (FT) matching. Detailed computer simulations show that global optical matching uses the "ne-grained phase-plane structure of the Fourier transform of the "ngerprints to produce strong optical correlations. This "ne-grained structure is very sensitive to pressure and plastic distortion e!ects which then dominate in correlations of static "ngerprints. The "ne grained local variations in "ngerprints can be compensated for by calculating optical correlations on

318

C.L. Wilson et al. / Pattern Recognition 33 (2000) 317}331

smaller zones of the "ngerprints. A training set was derived from disk two volume 1 of SD-9 and the testing set from disk one volume 1 of SD-9 [3]. This database contains "ngerprints of widly varing quality which are representative of the US population; the "ngerprints were taken from an unbiased sample of the "ngerprint searches submitted to the FBI. Since all "ngerprints on disk one were tested against each other, a total of 258,444 tests were performed in each experiment. This is the largest FT-based matching experiment reported to date. In our experiments, two 4]4 matrices of correlations on zones of the "ngerprint are used to produce a total of 32 features. One set of correlations is computed with the local zone grid centered on the core and one set is computed with the core in the center of the grid just above and to the left of grid center. Features were extracted from the correlation data using correlation peak height, correlation peak width, and correlation area. These features are combined using two types of neural networks. The "rst network is used to classify the "ngerprints [4}6]. This "ngerprint classi"cation network works directly with the "ngerprint image. After each "ngerprint is classi"ed, class-by-class matching networks are trained for each class. These two networks function in a way similar to the binary decision networks discussed in Ref. [7]. For this particular problem, the network training is strongly dependent on regularization and pruning for accurate generalization [2]. The advantage of the combined optical neural network method is its insensitivity to image resolution and quality. The experiments presented in this paper were done with three di!erent image sizes. Initial results were obtained with image samples with 512]512 pixels on each side, sampled at 20 pixels/mm. These images were downsampled to 256]256 and 128]128 using averaging of the gray levels to achieve sampling rates of 10 and 5 pixels/mm. The full matching test was then performed for three combinations of extracted features and for images of each size. As we will discuss in Section 3, analysis of ridge spacing data on the test "ngerprints shows that the Nyquist sampling limit of two pixels for each ridge and valley occurs at the 256 pixel level. The 128]128 images were sampled at half the Nyquist level and are of low quality with few clear ridge endings and/or bifurcations. As we discuss in Section 5, the accuracy of the hybrid matching method decreases with image resolution but remains usable even for 128]128 images. In Section 2 we describe the direct optical correlation experiment. In Section 3 we present an analysis of ridge frequency data and its e!ect on image quality. In Section 4 we discuss combining optical and neural network methods. In Section 5 we present the results of the hybrid system and in Section 6 we draw some conclusions about the di!erence in correlations of real time and rolled inked "ngerprints.

2. Global optical correlations In the global optical matching experiment, images from NIST Special Database 9 (SD-9) [3] are core aligned using the method discussed in Ref. [4] and cropped to "t the 640]480 pixel "eld of the pattern recognition system. Two hundred reference "ngerprints and second rollings (inked images taken at a di!erent time) are available for autocorrelation and cross-correlation experiments. When binary "nger prints are used the method used is based on that presented in Ref. [5]. Fig. 1 shows a schematic diagram of the optical pattern recognition system. It is based on the conventional VanderLugt correlator [1]. The target "ngerprint image is loaded on a spatial light modulator (SLM), and is Fourier transformed by a lens. The resulting Fourier spectrum is interfered with a reference beam to record a Fourier transform hologram. After recording is "nished, if an arbitrary input "ngerprint is presented on the SLM the correlation of the input and the target appears in the correlation output plane. Although the spatial heterodyning technique, often called joint transform correlator [8], has many advantages for real-time applications [9,10] and was used in most of the recent "ngerprint recognition experiments [11}16], the VanderLugt correlator was adopted in this experiment. This is because the VanderLugt correlator does not require a fast SLM with high resolution and the large space bandwidth product (SBP) available from holographic recording materials provide a high degree of freedom to accommodate various distorted versions of a target that are simultaneously compared with an input. Also, since the information is recorded in the form of a di!raction pattern (hologram) instead of a direct image, it can be used on a credit card or an ID card for security purposes without need for further encoding. Finally, the VanderLugt correlator is better suited for spatial "ltering to increase signal-to-noise ratio (SNR). The critical positioning tolerance problem of the VanderLugt correlator can be greatly relaxed by using in situ recording materials, such as thermoplastic plates, as were used in this experiment. In this case, once the system is aligned, new holographic "lters can be generated with no fear of misalignment. In the global correlation experiment, "ngerprint images are generated from the NIST "ngerprint database [3]. In the real-time correlation experiment, images are generated by a live-scan "ngerprint scanner (Identicator Technology, Gatekey Plus ver. 4.1).1 An electrically 1 Certain commercial equipment may be identi"ed in order to adequately specify or describe the subject matter of this work. In no case does such identi"cation imply recommendation or endorsement by the National Institute of Standards and Technology, nor does it imply that the equipment identi"ed is necessarily the best available for the purpose.


319

Fig. 1. Diagram of the optical pattern recognition system.

Fig. 2. Histograms of peak correlations for gray (a) and binary (b) "ngerprint images.

addressable liquid crystal SLM (Kopin, LVGA kit, 14 mm diagonal)1 is used as an input device. The SLM is mounted on a rotational stage to facilitate precise rotational tolerance measurements. Holographic "lters are recorded on a thermoplastic plate (Newport Corp. HC-300)1 that allows fast nonchemical processing, high di!raction e$ciency and high sensitivity. Although the recording process cannot be achieved in real time (close to 1 min.), the time-consuming comparison of an input with many other images in a large database can be done very fast, once a hologram is made. A 10 mW HeNe laser with a ND 2 "lter was used as a light source, and so only 0.1 mW was used to see correlation output, due to the high light e$ciency of the system.

The system is also equipped with real-time in situ monitoring of an input image, its Fourier transform, and the correlation output. These monitoring parts, combined with a frame grabber and other analytic tools, permit real-time quantitative analyses and accurate characterization of every stage of the system operation. The correlator system is capable of shift-invariant pattern recognition over a broad range of input positions and has high SNR due to accurate alignment using an interferometer and a microscope. Fig. 2 shows a histogram of peak correlations for gray (a) and binary inputs (b). For each of 20 randomly chosen "ngerprints, an individual holographic "lter was fabricated and tested against the 200 "ngerprints in the NIST database stored in the control computer. Therefore each plot involves 4000 correlations. Each peak

320


correlation value was obtained by taking the maximum value in the correlation plane. In case of gray inputs shown in (a), all 20 autocorrelations peak at the maximum value (152). Cross-correlations distribute in a Gaussian shape with a full-width half-minimum (FWHM) of around 15, and the maximum at 60. For binary inputs shown in (b), all autocorrelations peak at the maximum value, as in gray inputs. However, in this case, cross correlations are signi"cantly reduced to zero except for the few cases which were found to be from the correct "ngerprints of the other rolling. For both gray and binary inputs, the autocorrelations are well separated from the cross correlations to permit perfect 100% recognition for correct "ngerprints (without considering distortions). The exact mechanism for the signi"cant increase in SNR for binary inputs is not completely understood. However, several previous works [17,18] support the experimental results. Such a high SNR of binary inputs can be e$ciently used to make a composite "lter to permit tolerance against distortion.

3. Fingerprint image characteristics In this section we present data on "ngerprint ridge pitch and frequency and the e!ect of image sampling frequency on image quality. The standard sampling frequency for "ngerprint data for images of inked "ngerprints is 500 pixels/in. or 19.7 pixels/mm, approximately 20 pixels/mm. Live-scan system designed for law enforcement applications use this sampling rate but live-scan systems designed for veri"cation applications are using lower sampling rates down to approximately 5 pixels/mm. The constraint on image quality that ridge frequency values imposed is important for both minutia matching methods and for Fourier transform methods. For minutia matching methods, the ridge structure of the "ngerprint must be sampled with su$cient frequency to allow the ridge and valley structure to be accurately detected. This is observed to be approximately two points for each ridge and two points for each valley as expected from basic Nyquist sampling theory. In the FT case, the frequency of sampling is important because it e!ects the sensitivity of the correlation to plastic distortion. Near the center of the "ngerprint ridge and valley position do not vary much with pressure but at the edges "ngerprint ridges may be displaced by a full ridge width e!ectively interchanging the ridge and valley positions. Smaller ridge pitch values for an equally elastic "nger will increase this e!ect. Typical e!ects of elastic distortion are shown in Fig. 14. In previous optical "ngerprint correlation studies [11}16], decreasing image size and sampling frequency has decreased sensitivity to rotational alignment and plastic distortion.

3.1. Ridge pitch variation In a collection of "ngerprints, two kinds of variation of ridge spacing are of interest for matcher evaluation. First we have variations in ridge pitch within individual "ngerprints, and second we have variations in ridge pitch across samples of "ngerprints, such as the volume 1 disk 1 of NIST database 9. The variations discussed here were measured by performing FTs for each "ngerprint. The power spectrum of the FT of the "ngerprint was then sampled over angles from 0 to n at 257 di!erent radial distances and a histogram of relative power vs spatial frequency was produced for each "ngerprint. The average values of these histograms over each class for male and female subjects was also produced. The class sample sizes re#ect the natural class occurence rates; sample sizes for arches and tented arches are about 1/19 the size of those for loops and whorls. The variations in ridge spacing for two individual "ngerprints are shown in Figs. 3 and 5. Both "ngerprints have sharp peaks in their power spectrum in the typical ridge pitch range between 0.4 and 0.6 mm and have minimum ridge spacing of about 0.2 mm and maximum ridge spacing of about 0.8 mm. The peak power of the two prints are near the limits for peak power observed in the special database 9 sample. The "ngerprint with the smaller ridge pitch has a power distribution skewed toward smaller ridge pitch values, see Fig. 3, and the "ngerprint with the larger ridge pitch has a power distribution skewed toward larger ridge pitch (see Fig. 5). The "ngerprints measured to obtain the two distributions are shown in Figs. 4 and 6 respectively. Both images have a scale of 19.7 pixels/mm and the ridge pitch shown in Fig. 4 is, as expected from the distributions, 2/3 of that shown in Fig. 6. Fig. 6 also demonstrates that using FT power to measure ridge pitch is robust enough to work

Fig. 3. Relative power of the FT as a function of ridge pitch for "ngerprint with small pitch.


Fig. 4. Example "ngerprint with narrow, 0.4 mm, ridge pitch.

Fig. 5. Relative power of the FT as a function of ridge pitch for "ngerprint with large pitch.

well on a "ngerprint with a poor quality image. Examination of the "ngerprint images illustrated that the larger ridges are near the crease at the bottom of the images and the smaller ridges are near the "nger tip. This is true for all of the "ngerprints tested. As the number of "ngerprints used in the calculation of the power spectra vs ridge pitch is increased, the distributions become smooth and approach skewed Gaussian form. This is illustrated in Fig. 7 for males with "ngerprints classi"ed as whorls and by Fig. 8 for females with "ngerprints classi"ed as whorls. In these distributions, the range of ridge pitches in each distribution, 0.2}1.0 mm, is larger than the variations in maximum ridge pitch between males and females, and larger than the variation in maximum ridge pitch between classes. The distributions for other classes of "ngerprints have similar shapes

321

Fig. 6. Example "ngerprint with wide, 0.6 mm, ridge pitch.

Fig. 7. Relative power of the FT as a function of ridge pitch for males for "ngerprint which are classed whorls.

Fig. 8. Relative power of the FT as a function of ridge pitch for females for "ngerprint which are classed whorls.

322


Fig. 9. Example "ngerprint sampled at 20 pixels/mm.

and ranges. These measurements show that ridge pitch variations in individual "ngerprints and in classes of "ngerprints are always larger than variations in most likely ridge pitch across groups of "ngerprints or between sexes. 3.2. Image quality The e!ects of sampling frequency on image quality are shown in Figs. 9}11 for a single "ngerprint sampled at 20, 10, and 5 pixels/mm. Based on the FT power spectra presented in the previous section we would expect some part of the "ngerprint, usually above the core of the "ngerprint, to be just adequately sampled at the 20 pixel/mm sampling rate, however most of the "ngerprint would be adequately sampled. In Fig. 9 the region of narrow ridge spacing between 1 and 3 o'clock above and to the right of the core are just adequately sampled. When the sampling rate is reduced to 10 pixels/mm, as in Fig. 10, some of the minutiae in this region are di$cult to detect because of blurring. Most of other sections of the "ngerprint has clearly de"ned ridge structure. From the class FT power distribution, such as Fig. 7, we would expect that only a few percent of the ridges would be obscured by this 2-to-1 down-sampling and this is the case.

When the sampling rate is further reduced to 5 pixels/ mm, as in Fig. 11, we see a large reduction in image quality. At 5 pixels/mm we would expect, from Fig. 7, that about 40% of the "ngerprint would be sampled with a resolution less than the expected Nyquist limit. In Fig. 11 most of the minutia locations are blurred, and ridge locations in the area above and to the right of the core are lost at this level of resolution. From the combined e!ect of FT ridge pitch analysis and from the examples given above we conclude that on most "ngerprints, the e!ect of sampling at 10 pixels/mm will be small although some part of many "ngerprints will be undersampled. Sampling at 5 pixels/mm will make minutiae detection either uncertain or impossible, and will make detection of ridges di$cult in a signi"cant part of most "ngerprints.

4. Combined optical and neutral system Direct global correlation of "ngerprints for matching has a signi"cant failure rate caused by the elasticity of "ngerprints. Two rollings of the same print can vary signi"cantly, as seen by computing their Fourier transforms, because of the stretching variations which occur


323


when rolling a "ngerprint. Fig. 12 shows the correlation of two rollings of the same print that have been rotationand-translation-aligned based on the ridge structure around the core. It is clearly seen that the "ngerprints correlate (indicated by the dark gray pixels) around the core, but away from the core the patterns have di!erent amounts of elastic distortion. Since the elastic distortion problem is local, a method of local correlation can be used to lower the average distortion in small subregions of the "ngerprint. 4.1. Optical features A solution to elastic distortion that occurs in di!erent rollings of the same "ngerprint is partitioning the images into tiles and comparing the data within each of the tiles using FT-based methods. For this work, each image was partitioned into 4]4 tiles twice so that each tile contained one-sixteenth of the total image area. One partition had the core located in the center of the image, as de"ned by the "ngerprint core, and the second partition had the core shifted away from the center so that the new center was located at the corner of one of the "rst set of partitions. This double partitioning allowed for overlap of data (speci"cally data on the edge of the tiles). Since

the neural network is allowed to prune any data that is not needed, excess overlap in the features can be removed during network training. Fig. 13a and b show the core location for each 4]4 partition. In NIST database 9 two rollings of each "ngerprint are present; these "ngerprint sets labeled "le prints ( f (n)) and search print (s(n)). After partitioning, each f (n) and s(m) pairs are compared by correlating the corresponding tiles (32 tiles) for each print and extracting features from the correlations as inputs to the neural network. The features used are the central correlation peak height, correlation peak area and correlation peak width. Fig. 14 shows two print tiles from a matched pair and the corresponding correlation output. The correlation peak data is extracted by taking a cross section (perpendicular to the "ngerprint ridge direction) of the peak at the maximum correlation value. Only the central peak, from the maximum values to the "rst minimum on each side of the curve, is used for extracting features. The peak height is the di!erence between the maximum value and the lowest of the two minimum values. The peak width is measured between the two minimums and peak area is the area under the peak curve from the maximum-to-minimum values. The correlation is computed in the Fourier domain by taking the Fourier transform of the partitions and

324



computing the inverse Fourier transform of their product, using the complex conjugate of the "rst (Eq. (1)). f (n)"s(m)"F~1[FH[ f (n)]]F[s(m)]].

(1)

Each f (n) and s(m) vector has 32 values for each peak feature (i.e. height, area and width) (n"1, 2,2, 900 and m"1, 2,2, 900): peak} features[ f (n) "s(m) ] 1 1 peak} features[ f (n) "s(m) ] 2 2 F peak} features[ f (n) "s(m) ] 31 31 peak} features[ f (n) "s(m) ] 32 32 Automated feature detection procedures were applied to NIST Special Database 9 Vol 1, where disk 2 was used as training data and disk 1 was used as testing data. For this partitioning technique to be e!ective, the images need to be rotationally and translationally aligned about the cores of the two "ngerprints being compared. This alignment was accomplished over a large set of data using an automated technique. There are three steps in the automated alignment, "lter/binarize image, detect core location, and determine alignment.

Filtering, binarization, and core detection are done using methods previously developed and discussed in detail in Ref. [6]. The only addition is that the binarized "ngerprint is median "ltered using a 3]3 window to help smooth noise in the ridge data and improve correlation performance. The alignment step uses 128]128 segments that are centered about the core of the "ngerprints being aligned. The correlation of the segments is computed while rotating the second segment over a range of angles. The angle which produces the largest correlation is used for rotation alignment. Since two prints can have signi"cant angular displacements the alignment is actually done in two stages. Stage one uses an angular step size of 13 over a range of $153 and stage two a step size of 0.23 over a range of $13 from the angle determined in the "rst stage. Since the correlation computed by Eq. (1) is translation independent, translation alignment is accomplished by using the peak correlation location from the second stage of the angular alignment. The amount that the peak correlation is o! center of the 128]128 segment determines how much the second print needs to be shifted to achieve translational alignment with the "rst. The feature extraction procedure results in a total of 96 features for each pair of "ngerprints compared. In SD-9, each "ngerprint has one print in the test set that matches


325

Fig. 12. Correlation of two rollings of the same print. (Dark gray indicates correlated ridges, white and light gray indicate uncorrelated ridges.

matching prints had very narrow, well-de"ned peaks and non-matching prints had broader #at peaks. This led to the next neural network in which correlation peak area was added as an input feature (64 features). Signi"cant improvements were obtained in matching error rates (shown later in the paper). Since the shape of the peak curves vary, an exact correlation could not be made between peak area, height and width. The "nal network tested used all 96 features (correlation peak maximum, area, and width) and showed a smaller improvement over the 64 feature network. Fig. 13. Shows image partitioning and the corresponding feature number.

4.2. Neural network matching

and several thousand which do not match. Only those prints which do not match but are of the same class are included in the training set. The previously developed neural network classi"er [6] is used for this screening process. The 96 features were used in three di!erent neural networks. The "rst network only used the maximum correlation values as the features (32 features). Because of partitioning the prints, a main source of error resulted from non-matching prints having a maximum peak value in the ranges of matching prints. The di!erence was that

The matching networks discussed in this section were trained using a dynamically modi"ed scaled conjugate gradient method presented in Ref. [2]. In Ref. [2], we demonstrated that performance equal to or better than probabilistic neural networks (PNN) [19] can be achieved with a single three-layer multi-layer perceptron (MLP) by making fundamental changes in the network optimization strategy. These changes are: (1) neuron activation functions are used which reduce the probability of singular Jacobians; (2) successive regularization is used to constrain volume of the weight space being minimized;

326


Fig. 14. Original matched pair and corresponding correlation.

(3) boltzmann pruning is used [20] to constrain the dimension of the weight space; and (4) prior class probabilities are used to normalize all error calculations so that statistically signi"cant samples of rare but important classes can be included without distortion of the error surface. All four of these changes are made in the inner loop of a conjugate gradient optimization iteration [21] and are intended to simplify the training dynamics of the optimization. In this work we found that the e!ect of the sinusoidal activation, 1 above, was not useful, but that pruning, 3 above, and regularization, 2 above, were essential to good generalization. Since the distribution of match and

do not match classes was highly unequal, the e!ect of prior weights, 4 above, was also very important. The optimal regularization factor for all runs was found to be 0.001 and the optimum pruning temperature was found to be 0.005.

5. Hybrid results Five di!erent experiments were performed. In each experiment all "ngerprints from disk one of volume one of SD9 were used as a test sample. All "ngerprints from disk two of volume one of SD9 were used for network


training. A total of 258,444 tests were performed in each experimental test sequence. The experiments were designed to test the e!ect of di!erent methods of FT peak feature extraction and the e!ect of image resolution on accuracy. The feature extraction experiments used the three methods described above to obtain features from the local FT data. All images were sampled at 20 pixels/mm. In the "rst of these experiments the correlation peak height was used as the feature. In the second experiment the area under the correlation peak was used as an additional feature. In the third set of experiments, the width of the correlation peak was added to the feature set. For each feature set the Karhunen}Loe`ve (K}L) transformation was used, as in Ref. [7], to reduce the dimensionality of the feature set. Before K}L transformation, these three experiments had feature vector lengths of 32, 64 and 96-elements. In the second set of experiments, the 96-element feature vectors including correlation peak height, peak area, and peak width were extracted for sets of images which were sampled at 20, 10, and 5 pixels/mm. The "rst set of images in the resolution experiments was the same as the set of images from the third experiment. Each of the local features sets discussed above was separated into testing and training samples both by class and as a global (all class) set. The training sets were used to calculate global and class-by-class covariance matrices and eigenvectors and to calculate K}L transform [22] features for all of the testing and training sets. The e!ect of the K}L transform was to reduce the feature set sizes from 32 to 13, from 64 to 58 and from 96 to 58 or 59. When the eigenvectors of the K}L transform were examined for the peak-based 32-element feature set, the primary source of variation was found to be in 12 zones near the center of the two feature grids. The "rst eigenvector of each of the transforms was approximately 40 times larger than the 13th eigenvector, indicating that only about 13 statistically independent features were computed from the training sets. No large di!erence in K}L transform characteristics were seen between global and class-by-class data sets. When the eigenvectors of the K}L transform were examined for the combined peak-and-area-based 64-element feature set, most elements, 58 of the 64, made a signi"cant contribution to the sample variance. Increasing the feature set width to 96-elements by adding the correlation peak width did not increase the number of useful eigenvalues. The transformed feature vectors were still 58-elements long. We can therefore conclude that peak width and peak area are highly correlated. The K}L transformed features were used to train neural networks for both global and class-by-class matching for each of the "ve data sets. The networks were trained using regularization to bound weight size and pruning to restrict the number of active weights in the network to a size. Network size, pruning, and regularization were

327

adjusted empirically to provide reasonable generalization. The criterion used to test generalization accuracy was the comparison of the test and training matching errors. 5.1. Correlation peak features The basic network size was 13-24-2 network with 386 weights including bias weights. Twenty-four hidden units were needed to provide adequate coverage of the various combinations of interconnections during pruning. A sigmoidal activation function was used for the hidden nodes. With this network size and these training parameters, a typical functioning network has approximately 150 weights and has a accuracy of 62}71%. The results of this process are given in Table 1. These results can be signi"cantly improved by using PCASYS or some other classi"cation method to test only prints of the same class for matching. Assuming the PCASYS accuracy of 99% correct classi"cation at 20% rejects and a natural distribution of classes would allow the results given above to be improved to 84.3% matching accuracy. If a perfect classi"er were available, then the combined accuracy would be 90.3%. This model assumes that each print is classi"ed or rejected by PCASYS. The rejected prints are matched with the All network given in the top line of the table. All other prints are matched by the network selected by its PCASYS class. All prints misclassi"ed by PCASYS are assumed to be mismatched. The process of calculating the results shown in Table 1 involved training runs in which both the regularization and pruning were systematically varied to determine the correct network size and the appropriate dynamics for training. As discussed in Ref. [20], network size is an indication of the amount of information that can be transferred between the training sample and the network without learning random noise patterns. In Table 1, all of the "nal networks had a potential weight space size of Table 1 Results of training and testing for global and class-by-class neural network matching using 13 K}L features. Features were taken from the correlation peak in each subregion. Images contained 512 pixels on each axis. Combining all networks with pattern classi"cation yields 85.28% accuracy. All networks had a 13-24-2 structure with 386 weights Class

Train

Test

Wts. pruned

Test set size

All Arch Left loop Right loop Tented arch Whorl

70.2 71.8 72.1 72.5 75.4 72.0

65.2 64.9 62.6 71.1 67.3 68.5

285 229 240 209 247 275

258,444 1681 73,984 68,121 1089 113,569

328


386 weights. Larger networks were found to have poorer testing error than networks of this size. The pruning temperature was varied to produce similar testing and training errors for each class and for the global class. As the Table shows, this produced weight reductions from 209 to 285 weights, leaving 101}187 non-zero weights. The small network size and large pruning ratio for acceptable generalization with training set of up to 100,000 samples show that the noise in the features used in the training is at a level where larger network sizes are not useful, because all of the information needed for generalization is learned by these small networks. All of the pruning experiments require that some small amount of regularization be used to constrain the volume of weight space [2]. This allows the discriminant surfaces to remain in the part of the training space which is heavily populated by the data. All of these runs were done in the 13-feature K}L space, but numerous test pruning and regularization runs were made in the original 32-feature space. Similar e!ective weight spaces were found in the full 32-feature space, about 150 weights. The 13-feature data set was selected for additional testing to save on computation time during training.

from 64 to 58 features. For some classes, such as left loops, this results in greatly improved accuracy, from 62.2 to 84.6%, and a reduced number of weights, from 146 to 90. For other classes the network does not reduce in size or improve in accuracy. For tented arches the accuracy decreases from 75.4 to 74.1% and the number of active weights increases from 139 to 320. Since the training data has a natural class distribution, the data indicates that the classes with relatively small sample sizes, arch and tented arch did not improve with more complex features but the classes with larger training set sizes, loops and whorls, improved an average of 10%. 5.3. Correlation peak, peak area, and peak width features

The basic network size was a 58-24-2 network with 722 weights, including bias weights, for class networks and 59-48-2 for the all class network. A sigmoidal activation function was used for the hidden nodes. With this network size, from Table 2 we see that a typical functioning network has approximately 100}300 weights and has an accuracy of 74}84% for class networks. The global (All) class network had 298 weights and an accuracy of 76%. This experiment di!ers from the peak feature experiment in that the combined feature set is only reduced

The basic network size was 57, 58, 59-48-2 network with 1442, 1466, 1490 weights, including bias weights, for the various networks. A sigmoidal activation function was used for the hidden nodes. With this network size, from Table 3 we see that a typical functioning network has approximately 250}530 weights and has a accuracy of 74}84% for class networks. The global (All) class network had 298 weights and an accuracy of 76%. This experiment di!ers from the two previous experiments in that the required network size has 48 hidden nodes for all of the networks and pruning on these networks with the training set sizes used is substantially less e!ective than it was with peak and area features. The less e!ective pruning of the network doubles the number of weights from 100}300 to 250}530. This shows that, even after feature correlations are removed by the K}L transform, various complex feature combinations are available that are detected in the network training. As in the previous experiment, the classes with relatively small sample sizes, arch and tented arch, did not improve with more complex features, but the classes with larger

Table 2 Results of testing and training for global and class-by-class neural network matching using 56}58 K}L features. All class networks had 24 hidden nodes while the all network had 48 hidden nodes. Features used were based on peak correlation and area under the correlation peak. Images contained 512 pixels on each axis. Combining all networks with pattern classi"cation yields 89.95% accuracy. Typical class networks had a 58-24-2 structure

Table 3 Results of testing and training for global and class-by-class neural network matching using 58 K}L features. All networks had 48 hidden nodes. Features used were based on peak correlation, the width of the correlation peak, and area under the correlation peak. Images contained 512 pixels on each axis. Combining all networks with pattern classi"cation yields 90.9% accuracy

Class

Train

Test

Wts. pruned

Total wts.

Test set size

Class

Train

Test

Wts. pruned

Total wts.

Test set size


78.3 78.2 79.1 80.6 86.8 79.3

76.0 75.0 84.6 80.0 74.1 80.7

1144 587 608 576 402 589

1442 746 698 722 722 722

258,444 1681 73,984 68,121 1089 113,569


78.3 80.1 82.2 82.6 96.0 84.8

76.0 76.0 84.9 79.9 74.5 82.7

1144 1135 1276 1251 963 1235

1442 1490 1466 1466 1490 1490

258,444 1681 73984 68,121 1089 113,569

5.2. Correlation peak and peak area features


329



Class

Train

Test

Wts. pruned

Total wts.

Test set size

Class

Train

Test

Wts. pruned

Total wts.

Test set size


78.3 76.2 77.6 76.9 80.1 77.5

76.0 71.3 81.9 76.0 72.0 79.4

1144 1276 1358 1343 1276 1381

1442 1466 1442 1442 1466 1466

258,444 1681 73,984 68,121 1089 113,569


78.3 79.0 79.0 76.6 84.7 77.6

76.0 77.9 81.7 77.5 76.6 73.8

1144 1315 1384 1430 1360 1433

1562 1562 1514 1514 1514 1514

258,444 1681 73,984 68,121 1089 113,569

training set sizes, loops and whorls, improved an average of 10% over simple peak features.

Table 6 Accuracy of match for di!erent features and sample resolutions Feature type

Sample resolution (pixels/mm)

Accuracy (%)

Peak Peak#Area Peak#Area#Width Peak#Area#Width Peak#Area#Width

20 20 20 10 5

85.28 89.95 90.9 89.3 88.66

5.4. Scan resolution of 10 pixels/mm In this experiment the image scanning resolution was reduced from 20 to 10 pixels/mm. The basic network size was 57, 58-48-2 network with 1442, 1466 weights, including bias weights, for the various networks. The network size is similar to the 20 pixel/mm experiment. A sigmoidal activation function was used for the hidden nodes. With this network size, from Table 4 we see that a typical functioning network has approximately 85}190 weights and has an accuracy of 71}81% for class networks. The global (All) class network had 302 weights and an accuracy of 76 %. The two main e!ects of lower image resolution are to increase pruning e!ectiveness and to decrease accuracy. The required network size is still about 1466 weights but the number of weights that are useful has decreased by about a factor of two and the class networks have been trained to a corresponding lower accuracy. The data needed to generate more complex weights set is missing in the lower resolution data. 5.5. Scan resolution of 5 pixels/mm In this experiment the image scanning resolution was reduced from 10 to 5 pixels/mm. The basic network size was 62, 60-48-2 network with 1562, 1514 weights, including bias weights, for the various networks. A sigmoidal activation function was used for the hidden nodes. With this network size, from Table 5 we see that with these training parameter a typical functioning network has approximately 88}247 weights and has an accuracy of 74}81% for class networks. The global (All) class network had 418 weights and an accuracy of 76%.

Further reduction in image resolution from 10 to 5 pixels/mm has had two e!ects. The K}L transform yields 60}62 features instead of 58}59 features which results in a small increase in initial network size. This larger network is then pruned to about the same size as in the 10 pixel/mm case and the "nal accuracy of the matching is reduced by less than 1%. 5.6. Summary In Table 6 the global matching accuracy of all "ve matching experiments are compared. The largest improvement in accuracy occurs when the peak area is added to the peak height; this improves matching accuracy from 85 to 89%. Adding the peak width provides another 1% increase in accuracy but the peak width and area are su$ciently correlated that major improvements are not possible. Decreasing the image resolution decreases matching accuracy. The reduction from 20 to 10 pixels/mm reduces matching accuracy from 90.9 to 89.3%. The most surprising result is that, by using FT-based features, the 5 pixel/ mm image can still be used to train an 88.66% accurate matcher which is only 0.64% lower than the 10 pixel/mm case. This clearly shows that the FT-based features can

330


be used for matching on images which have too low quality to provide clear minutia. The various operations in the hybred matching system have limiting speeds which vary over many orders of magnitude. The optical feature extraction occurs at the speed of light which allows a correlation peak to be processed in about 10 ns. The neural network used here is small and can combine the features to generate a match on a typical PC at a rate of 1000 matches/s. The speed limitation of the system is in the input and output of the data to the optical system. The SLM requires about 50 ms to form an image and the output camera is limited to NTSC video rates of 60 frames/s. 6. Conclusions We have compared optical and combined optical-neural network methods for rolled inked "ngerprint image matching. For static inked images, direct global optical correlation of inked images made at di!erent times has very low reliability, although cross correlations and auto correlations of the original inked images are good. This di$culty can be accounted for by the plastic deformation of the "ngerprint during rolling. Combining zonal optical features with neural networks for classi"cation and matching can yield reliable matching with an accuracy of 90.9%. This result was achieved using a neural classi"cation network described elsewhere [4}6] and three components of the local FT correlation to drive class-by-class matching networks. The information content analysis of the features, both from the dimension of the K}L transform features and the generalization error analysis, show that the information transfer from the training data to the classi"cation network is as high as the noise level of the features will allow for each K}L transform, feature set, and image resolution. The method used to achieve this optimal training is discussed in Ref. [2]. In principle, direct combination of multiple real-time images in a holographic matched "lter can allow for greater stored information content in the matching process. This will be the subject of further study. References [1] A. VanderLugt, Signal detection by complex spatial "ltering, IEEE Trans. Inform. Theory IT- 10 (1964) 139}145. [2] C.L. Wilson, J.L. Blue, O.M. Omidvar, Training dynamics and neural network performance, Neural Networks 10 (5) (1997) 907}923. [3] C.I. Watson, Mated "ngerprint card pairs, Technical Report Special Database 9, MFCP, National Institute of Standards and Technology, February 1993.

[4] C.L. Wilson, G.T. Candela, P.J. Grother, C.I. Watson, R.A. Wilkinson, Massively parallel neural network "ngerprint classi"cation system, Technical Report NISTIR 4880, National Institute of Standards and Technology, July 1992. [5] C.L. Wilson, G.T. Candela, C.I. Watson, Neural-network "ngerprint classi"cation, J, Arti"cial Neural Networks 1 (2) (1994) 203}228. [6] G.T. Candela, P.J. Grother, C.I. Watson, R.A. Wilkinson, C.L. Wilson, PCASYS } a pattern-level classi"cation automation system for "ngerprints, Technical Report NISTIR 5647, National Institute of Standards and Technology, 1995. [7] C.L. Wilson, P.J. Grother, C.S. Barnes, Binary decision clustering for neural network based optical character recognition, Pattern Recognition 29 (3) (1996) 425}437. [8] C.S. Weaver, J.W. Goodman, Technique for optically convolving two functions, Appl. Opt. 5 (1966) 1248}1249. [9] F.T.S. Yu, X.J. Lu, A real-time programmable joint transform correlator, Opt. Commun. 52 (1984) 10}16. [10] J.L. Horner, Optical processing or security and anticounter"eting, IEEE LEOS"9296 Proceedings, Boston, vol. 1, 18}21 November 1996, pp. 228}229. [11] Y. Petillot, L. Guibert, J.-L. de Bougrenet de la Tocnaye, Fingerprint recognition using a partially rotation invariant composite "lter in a FLC joint transform correlator, Opt. Commun. 126 (1996) 213}219. [12] J. Podolfo, H. Rajenbach, J.-P. Huignard, Performance of a photorefractive joint transform correlator for "ngerprint identi"cation, Opt. Engng 34 (1995) 1166}1171. [13] B. Javidi, J. Wang, Position-invariant two-dimensional image correlation using a one-dimensional space integrating optical processor: application to security veri"cation, Opt. Engng 35 (1966) 2479}2486. [14] T.J. Grycewicz, B. Javidi, Experimental comparison of bianry joint transform correlators used for "ngerprint identi"cation, Opt. Engng. 35 (1996) 2519}2525. [15] F.T. Gamble, L.M. Frye, D.R. Grieser, Real-time "ngerprint veri"cation system, Appl. Opt. 31 (1992) 652}655. [16] K.H. Fielding, J.L. Horner, C.K. Makekau, Optical "ngerprint identi"cation by binary joint transform correlation, Opt. Engng. 30 (1991) 1958}1961. [17] J.L. Horner, H.O. Bartelt, Two-bit correlation, Appl. Opt. 24 (1985) 2889}2893. [18] D. Psaltis, E.G. Paek, S.S. Venkatesh, Optical image correlation with a binary spatial light modulator, Opt. Engng 23 (1984) 698}704. [19] D.F. Specht, Probabilistic neural networks, Neural Networks 3 (1) (1990) 109}118. [20] O.M. Omidvar, C.L. Wilson, Information content in neural net optimization, J. Connection Sci. 6 (1993) 91}103. [21] J.L. Blue, P.J. Grother, Training feed forward networks using conjugate Gradients. in: Conference on Character Recognition and Digitizer Technologies, vol. 1661, SPIE, San Jose, February 1992, pp. 179}190, [22] P.J. Grother, Cross validation comparison of NIST OCR databases, in: D.P. D'Amato (Ed.), Conference on Character Recognition Technologies, Vol. 1906, SPIE, San Jose, 1993, pp. 296}307.


331

About the Author*C.L. WILSON has worked in various areas of computer modeling ranging from semiconductor device simulation, for which he received a DOC gold medal in 1983, and computer aided design to neural network pattern recognition at Los Alamos National Laboratory, AT & T Bell Laboratories and for the last 19 years NIST. He is currently the manager of the Visual Image Group in the Information Access and User Interface Division. His current research interest are in application of statistical pattern recognition, neural network methods and dynamic training methods for iamge recognition, image compression, optical information processing systems and in standards used to evalute recognition systems. About the Author*CRAIG I. WATSON received his B.S. in Electrical Engineering Technology from the University of Pittsburgh, Johnstown, in 1991 and his M.S. in Electrical Engineering from The Johns Hopkins University in 1997. He has worked with the Visual Image Processing Group at the National Institute of Standards and Technology for the past 7 years. His work has included image processing, image compression and automated "ngerprint classi"cation. Currently, he is working with holographic storage and pattern matching using optical correlators. About the Author*EUNG GI PAEK received his B.Sc. degree in Physics from Seoul National University in 1972 and the M.Sc. and Ph.D. degrees in Physics from Korea Advanced Institute of Science in 1976 amd 1979, respectively. From 1979 to 1981, he worked at the Agency for Defense Development in Korea. In April 1982, he joined the California Institute of Technology as a postdoctoral fellow and later as a senior research fellow for a period of "ve years. In early 1987, he joined the Physical Science and Technology group of Bellcore (Bell Communications Research in Red Bank, NJ) as a permanent member of technical sta! and a principal investigator. During his stay at Bellcore for seven years, he had investigated various applications of photonic devices interacting with device and material scientists. He later joined Rockwell Science Center to contribute to the DARPA holographic storage project for a year until he moved to NIST (National Institute of Standards and Technology in Gaithersburg, MD) in March, 1995. Currently he is a physicist in Information Technology Laboratory of NIST. His current interest includes biometrics, volume holographic storage and optical neutral networks with various photonic devices such as surface-emitting microlasers. Recently, he has also been actively involved in dense WDM-based optical telecommunications and RF photonics. He is a fellow of both SPIE (International Society for Optical Engineering) and the OSA (Optical Society of America). He also serves the Optical Society of America as a Topical Editor of Optics Letters.


Bayes empirical Bayes approach to unsupervised learning of parameters in pattern recognition Tze Fen Li* Department of Applied Mathematics, National Chung Hsing University, Kuo-Kuang Road, Taichung 40227, Taiwan, Republic of China Received 29 April 1998; accepted 10 March 1999

Abstract In the pattern classi"cation problem, it is known that the Bayes decision rule, which separates k classes, gives a minimum probability of misclassi"cation. In this study, all parameters in each class are unknown. A set of unidenti"ed input patterns is used to establish an empirical Bayes rule, which separates k classes and which leads to a stochastic approximation procedure for estimation of the unknown parameters. This classi"er can adapt itself to a better decision rule by making use of unidenti"ed input patterns while the system is in use. The results of a Monte Carlo simulation study with normal distributions are presented to demonstrate the favorable estimation of unknown parameters for the empirical Bayes rule. The percentages of correct classi"cation is also estimated by the Monte Carlo simulation. ( 1999 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Classi"cation; Empirical Bayes; Pattern recognition; Stochastic approximation

1. Introduction A pattern recognition system in general consists of feature extractor and classi"er. The function of feature extractor is to extract or measure the important characteristics from the input patterns. Let x denote the measurement of the signi"cant, characterizing features. This x will be called an observation. The function performed by a classi"er is to assign each input pattern to one of several possible pattern classes. The decision is made on the basis of feature measurements supplied by the feature extractor in a recognition system. This general approach has been applied to many research areas: speech and speaker recognition, "ngerprint identi"cation, electrocardiogram analysis, radar and sonar signal detection, weather forecasting, medical diagnosis and reading handprinted characters and numerals. Since the measurement x of a pattern may have a variation or

* Tel.: #04-287-3028; fax: #04-287-3028

noise, a classi"er may classify an input pattern to a wrong class. The classi"cation criterion is usually the minimum probability of misclassi"cation. Essentially, there are two di!erent approaches to solving the classi"cation problem. One approach is to "nd a Bayes decision rule which separates classes based on the present observation X and minimizes the probability of misclassi"cation [1}3]. To apply this approach, one needs su$cient information about the conditional density function f (x D u) of X given class u and the prior probability p(u) of each class. Otherwise, the conditional density function and the prior probability have to be estimated through a set of past observations (or a training set of sample patterns). On the other hand, if very little is known about statistical properties of the pattern classes, a discriminant function D(X, h , 2, h ) can be 1 m used. A learning procedure and an algorithm are designed to learn the unknown parameters h of the disi criminant function. This is the so-called nonparametric procedure. The most common nonparametric procedures are the k-mean and the k-nearest-neighbor classi"ers [1]. After learning, the function is used to separate pattern


334

T.F. Li / Pattern Recognition 33 (2000) 333}340

classes [2,4}7]. For this approach, it is not easy to de"ne the exact functional form and the parameters of the discriminant function, which gives the minimum probability of classi"cation. In this study, the "rst approach is applied to solving k-class pattern problems: all parameters of the conditional density function f (x D u) are unknown, where u denotes one of k classes, and the prior probability of each class is unknown. A set of n unidenti"ed input patterns is used to establish a decision rule, called an empirical Bayes (EB) decision rule, which is used to separate k classes. After learning the unknown parameters, the EB decision rule will make the probability of misclassi"cation arbitrarily close to that of the Bayes rule when the number of unidenti"ed patterns increases. The problem of learning from unidenti"ed samples (called unsupervised learning or learning without a teacher) presents both theoretical and practical problems [2]. In fact, without any prior assumption, successful unsupervised learning is indeed unlikely. Our classi"er, after unsupervised learning the unknown parameters, can adapt itself to a better and more accurate decision rule by making use of the unidenti"ed input patterns after the system is in use. The results of a Monte Carlo study with normal distributions are presented to demonstrate the favorable estimation of the unknown parametrs for the EB decision rule.

2. Empirical Bayes decision rules for classi5cation

P

h f (x D m ) dx, i c i

R(h)"inf R(h, d). d|D

(2)

A decision rule d which satis"es Eq. (2) is called the h Bayes decision rule6 with respect to the prior probability vector h"(h , h ,2, h ) and given by [7] 1 2 k d (x)"c if h f (x D m )'h f (x D m ) for all jOi. (3) h6 i i i j j In the EB decision problem [8}10], the past observations (u , X ), m"1, 2,2, n, and the present observation m m (u, X) are i.i.d., and all X are drawn from the same m conditional densities, i.e., f (x D u ) with p(u "c )"h . m m m i i The EB decision problem is to establish a decision rule based on the set of past observations X " n (X , X ,2, X ). In a pattern recognition system 1with 1 2 n unsupervised learning, X is a set of unidenti"ed input n patterns. The decision 1rule can be constructed using X to select a decision rule t (X ) which determines n n 1 n whether the present observation 1 X belongs to c . Let i m"(m ,2, m ). Then the risk of t , conditioned on X , is 1 k n 1 n R(h, t (X ))*R( h ) and the overall risk of t is n 1 n n n n R (h, t )" R(h , t (x )) < p(x D h, m) < dx , (4) m n n n n m m/1 m/1

P

where p(x D h, m) is the marginal density of X with m m respect to the prior distribution of classes, i.e., p(x D h, m)"+ k h f (x D m ). Let m i/1 i m i S"Mh, mN

Let X be the present observation which belongs to one of k classes c , i"1, 2,2, k. Consider the decision probi lem consisting of determining whether X belongs to c . i Let f (x D u) be the conditional density function of X given u, where x denotes one of k classes and let h , i i"1, 2,2, k, be the prior probability of c with i + k h "1. In this study, both the parameters of f (x D u) i/1 i and the h are unknown. Let d be a decision rule. A simple i loss model is used such that the loss is 1 when d makes a wrong decision and the loss is 0 when d makes a correct decision. Let h"M(h , h ,2, h ); h '0, + k h "1N be 1 2 k i i/1 i the prior probabilities. Let R(h, d) denote the risk function (the probability of misclassi"cation) of d. Let ! , i"1, 2,2, k, be k regions separated by d in the doi main of X, i.e., d decides c when X3! . Let m denote all i i i parameters of the conditional density function in class c , i"1,2, k. Then i k R(h, d)" + i/1

be denoted by

(1)

!i

where !c is the complement of ! . Let D be the family of i i all decision rules which separate k pattern classes. For h "xed, let the minimum probability of misclassi"cation

(5)

de"ne a parameter space of prior probabilities h and the i parameters m representing the ith class, i"1,2, k. Let i ' be a probability distribution on the parameter space S. An EB decision rule t is said to be the Bayes EB rule with n respect to ' if it minimizes

P

RK (', t )" R ( h , t ) d'( h, m). n n n n

(6)

Similar approaches to constructing EB decision rules can be found in the recent literature [8}12]. From Eqs. (1) and (4), Eq. (6) can be written as RK (', t ) n n

P P CP D

k " + i/1

C ci,n

n f (x D m )h < p(x D h, m) i i m m/1

n dU(h , m) dx < dx , m m/1

(7)

where, in the domain of X, ! , i"1, 2,2, k, are k rei,n gions, separated by t (X ) and hence they depend on the n 1 n past observations X . The Bayes EB rule which minim1 n in the same way as for the Bayes izes Eq. (7) can be found

T.F. Li / Pattern Recognition 33 (2000) 333}340

rule (3) and is given by

P

n tK (x )(x)"c if f (x D m )h < p(x D h , m) d'(h , m) n n i i i m m/1 n ' f (x D m )h < p(x D h , m) d'( h , m) (8) j j m m/1 for all jOi. In applications, we let the parameters m , i"1,2, k, be i bounded by "nite numbers M . Let o'0 and d'0. i Consider the subset S of the parameter space S de"ned by 1 S1"M(n o, n o,2, n o, n d, n d,2, n d); 1 2 k k`1 k`2 2k k integer n , + n o"1, D n d D )M , i i i i i/1 i"k#1,2, 2kN, (9)

P

where (n o,2, n o) are prior probabilities and 1 k (n d,2, n d) are the parameters of k classes. Let ' be k`1 2k a "nite discrete distribution on the parameter space S with equal mass on S . The boundary for class i relative 1 to another class j as separated by Eq. (8) can be represented by the equation (10) E[ f (x D m )h D x ]"E[ f (x D m )h D x ], j j n i i n where E[ f (x D m )h D x ] is the conditional expectation of i i n f (x D m )h given X "x with the conditional probability n i i n function of (h , m)1 given X "x equal to n 1 n ),

A A

B

n2 M (>)! SM (>)2 , M (Z)"e~V1 (Y) M M 32

B

1 n M (>) SM (>)! sN (Z)"e~V1 (Y) c! M SM (>)3 , 4n 384

PII of original article: S0031-3203(97) 00030-8 0031-3203/99/$20.00 ( 1999 Published by Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 1 - 3 2 0 3 ( 9 9 ) 0 0 1 9 3 - 4

Pattern Recognition 33 (2000) 350

Erratum

&&Shape-based retrieval: a case study with trademark image databases'' by A.K. Jain, A. Vailaya Pattern Recognition 31(9) pp. 1369}1390 (1998) The equations for M and M on p. 1376 were printed incorrectly and should appear as follows: 5 7 M "(k #k ) (k !3k ) [(k #k )2!3(k #k )2] 5 30 12 30 12 30 12 21 03 #(3k !k ) (k #k ) [3(k #k )2!(k #k )2]. 21 03 21 03 30 12 21 03 M "(3k !k ) (k #k ) [(k #k )2!3(k #k )2] 7 21 03 30 12 30 12 21 03 !(k !3k ) (k #k ) [3(k #k )2!(k #k )2]. 30 12 21 03 30 12 21 03

PII of original article: S0031-3203(97) 00131-3 0031-3203/99/$20.00 ( 1999 Published by Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 1 - 3 2 0 3 ( 9 9 ) 0 0 1 9 4 - 6

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

Recommend Documents