No title

Pattern Recognition 33 (2000) 1405}1410 On the independence of rotation moment invariants Jan Flusser* Institute of Inf...

Author: Elsevier

28 downloads 240 Views 7MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Pattern Recognition 33 (2000) 1405}1410

On the independence of rotation moment invariants Jan Flusser* Institute of Information Theory and Automation, Academy of Sciences of the Czech Republic, Pod voda& renskou ve\ z\ n& 4, 182 08 Prague 8, Czech Republic Received 28 August 1998; accepted 19 May 1999

Abstract The problem of the independence and completeness of rotation moment invariants is addressed in this paper. First, a general method for constructing invariants of arbitrary orders by means of complex moments is described. As a major contribution of the paper, it is shown that for any set of invariants there exists a relatively small basis by means of which all other invariants can be generated. The method how to construct such a basis and how to prove its independence and completeness is presented. Some practical impacts of the new results are mentioned at the end of the paper. 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Moment invariants; Complex moments; Independence; Completeness

1. Introduction Moment invariants have become a classical tool for object recognition during the last 30 years. They were "rstly introduced to the pattern recognition community by Hu [1], who employed the results of the theory of algebraic invariants [2,3] and derived his seven famous invariants to the rotation of 2-D objects:

"k #k ,

"(k !k )#4k ,

"(k !3k )#(3k !k ),

"(k #k )#(k #k ),

"(k !3k )(k #k )((k #k ) !3(k #k ))#(3k !k )(k #k ) ;(3(k #k )!(k #k )),

"(k !k )((k #k )!(k #k )) #4k (k #k )(k #k ),

* Tel.: #420-2-6605-2357; fax: #420-2-6641-4903. E-mail address: #[email protected] (J. Flusser).

"(3k !k )(k #k )((k #k ) !3(k #k ))!(k !3k )(k #k ) ;(3(k #k )!(k #k )),

(1)

where

k " NO

(x!x )N(y!y )O f (x, y) dx dy A A \ \

(2)

is the central moment of the object f (x, y) and (x , y ) are A A the coordinates of the object centroid. Hu also showed how to achieve the invariance to scaling and demonstrated the discriminative power of these features in the case of recognition of printed capital characters. Since then, numerous works have been devoted to the various improvements and generalizations of Hu's invariants and also to its use in many application areas. Dudani [4] and Belkasim [5] described their application to aircraft silhouette recognition, Wong and Hall [6], Goshtasby [7] and Flusser and Suk [8] employed moment invariants in template matching and registration of satellite images, Mukundan [9,10] applied them to estimate the position and the attitude of the object in 3-D space, Sluzek [11] proposed to use the local moment invariants in industrial quality inspection and many

0031-3203/00/$20.00 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 1 - 3 2 0 3 ( 9 9 ) 0 0 1 2 7 - 2

1406

J. Flusser / Pattern Recognition 33 (2000) 1405}1410

authors used moment invariants for character recognition [5,12}15]. Maitra [16] and Hupkens [17] made them invariant also to contrast changes, Wang [18] proposed illumination invariants particularly suitable for texture classi"cation, Van Gool [19] achieved photometric invariance and Flusser et al. [20,21] described moment invariants to linear "ltering. Several papers studied recognitive and reconstruction aspects, noise tolerance and other numerical properties of various kinds of moment invariants and compared their performance experimentally [5,22}27]. Moment invariants were shown to be also a useful tool for geometric normalization of an image [28,29]. Large amount of e!ort has been spent to "nd the e!ective algorithms for moment calculation (see [30] for a survey). Recently, Flusser and Suk [31] and Reiss [32] have corrected some mistakes in Hu's theory and have derived the invariants to a$ne transform. In the contrast to a large number of applicationoriented works, only few attempts to derive invariants from moments of orders higher than three have been done. Li [33] and Wong [34] presented the systems of invariants up to the orders nine and "ve, respectively. Unfortunately, none paid attention to the mutual dependence/independence of the invariants. The invariant sets presented in their papers are algebraicly dependent. There is also a group of papers [14,35,36] that use Zernike moments to construct rotation invariants. Their motivation comes from the fact that Zernike polynomials are orthogonal on a unit circle. Thus, Zernike moments do not contain any redundant information and are more convenient for image reconstruction. However, Teague [35] showed that Zernike invariants of second and third orders are equivalent to the Hu's ones when expressing them in terms of geometric moments. He presented the invariants up to eight order in explicit form but no general rule as to the way to derive them is given. Wallin [36] described an algorithm for a formation of moment invariants of any order. Since Teague [35] as well as Wallin [36] were particularly interested in the reconstruction abilities of the invariants, they did not pay much attention to the question of independence. However, the independence of the features is a fundamental issue in all the pattern recognition problems, especially in the case of a high-dimensional feature space. The bene"t of this paper is twofold. First, a general scheme how to derive moment invariants of any order is presented. Secondly, we show that there exist relatively small set (basis) of the invariants by means of which all other invariants can be expressed and we give an algorithm for its construction. As a consequence of this, we show that most of the previously published sets of rotation moment invariants including Hu's system (1) are dependent. This is really a surprising result giving a new look at Hu's invariants and possibly yielding a new interpretation of some previous experimental work.

2. A general scheme for deriving invariants There are various approaches to the theoretical derivation of moment-based rotation invariants. Hu [1] employed the theory of algebraic invariants, Li [33] used the Fourier}Mellin transform, Teague [35] and Wallin [36] proposed to use Zernike moments and Wong [34] used complex monomials which also originate from the theory of algebraic invariants. In this paper, we present a new scheme, which is based on the complex moments. The idea to use the complex moments for deriving invariants was already described by Mostafa and Psaltis [24] but they concentrated themselves to the evaluation of the invariants rather than to constructing higher-order systems. In comparison with the previous approaches, this one is more transparent and allows to study mutual dependence/independence of the invariants easily. It should be noted that all the above approaches di!er from each other formally by the mathematical tools and the notation used but the general idea behind them is common and the results are similar or even equivalent. Complex moment c of the order (p#q) of an inteNO grable image funciton f (x, y) is de"ned as

c " NO

(x#iy)N(x!iy)Of (x, y) dx dy, \ \

(3)

where i denotes the imaginary unit. Each complex moment can be expressed in terms of geometric moments m as NO

N O p c " NO k I H

q j

(!1)O\ H iN>O\I\H m

. I>HN>O\I\H (4)

In polar coordinates, Eq. (3) becomes the form

c " NO

p

rN>O>e N\OFf (r, h) dr dh.

(5)

It follows immediately from Eq. (5) that c "cH (the NO ON asterix denotes a complex conjugate). The following lemma describes an important rotation property of the complex moments. Lemma 1. Let f be a rotated version (around the origin) of f, i.e. f (r, h)"f (r, h#a) where a is the angle of rotation. Let us denote the complex moments of f as c . Then NO c "e\ N\O?c . NO NO

(6)

Using Eq. (5), the proof of Lemma 1 is straightforward. It can be seen immediately that "c " is invariant to NO rotation for any p and q. However, the moment magnitudes do not generate a complete set of invariants. In the following theorem, we propose a better approach to the construction of rotation invariants.


Theorem 1. Let n*1 and let k , p and q ; i"1,2n, be G G G non-negative integers such that L k (p !q )"0. G G G G Then L I" cIG NGOG G is invariant to rotation.

on I if and only if there exists a function F of k variables containing only the operations multiplication, involution with an integer (positive or negative) exponent and complex conjugation, such that J"F(I , 2, I ). I

(7)

The proof of Theorem 1 follows immediately from Lemma 1. According to Theorem 1, some simple examples of rotation invariants are c , c c , c c , etc. As a rule, most invariants (7) are complex. If we want to have real-valued features, we only take the real and imaginary parts of each of them. To achieve also translation invariance (or, equivalently, invariance to the rotation around an arbitrary point), we only use the central coordinates in the de"nition of the complex moments (3). It can be seen that the Hu's invariants (1) are nothing else than particular representatives of the general form (7):

"c ,

"c c ,

"c c ,

"c c ,

"Re(c c ),

"Re(c c ),

"Im(c c ).

1407

De5nition 2. Let k'1 and let I"+I , 2, I , be a set I of rotation invariants of the type (7). The set I is said to be dependent if and only if there exists k )k such that I depends on I!+I ,. The set I is said to be indepenI I dent otherwise. According to this de"nition, +c c , c c ,, +c c , c c , c c c , and +c c , c c , are the examples of the dependent invariant sets. De5nition 3. Let I be a set of rotation invariants of the type (7) and let B be its subset. B is a basis of I if and only if E B is independent, E Any element of I!B depends on B (this property is called completeness). Now we can formulate the central theorem of this paper that tells us how to construct an invariant basis above a given set of moments. Theorem 2. Let M be a set of the complex moments of any orders (not necessarily of all moments), let MH be a set of their complex conjugates and let c 3M6MH such that NO p !q "1 and c O0. Let I be a set of all rotation NO invariants created from the elements of M6MH according to (7). Let B be constructed as follows:

(8)

3. Independence and completeness of the sets of invariants In this section, our attention will be paid to the construction of a basis of the invariants. Theorem 1 allows us to construct an in"nite number of the invariants for any order of moments, but only a few of them are mutually independent. By the term basis we intuitively understand the smallest set by means of which all other invariants can be expressed. The knowledge of the basis is a crucial point in all the pattern recognition problems because it provides the same discriminative power as the set of all invariants at minimum computational costs. To formalize this approach, we introduce the following de"nitions "rst. De5nition 1. Let k*1, let I"+I ,2, I , be a set of I rotation invariants of the type (7) and let J be an invariant of the same type. Invariant J is said to be dependent

3B). (∀p, q"p*qc 3M6MH)('(p, q),c cN\O NO NO ON Then B is a basis of I. For the proof of Theorem 2 see Appendix A.

4. Some consequences of Theorem 2 In this section, we highlight some consequences of Theorem 2 that are of practical importance. 4.1. On the dependence of the previously published invariants We show that the previously published systems of rotation invariants including the famous Hu's one are dependent. This fact has not been reported in the literature yet. Using Eq. (8) and assuming c O0, it is easy to prove that

# c c c c .

"c c " "

(c c )

1408


Moreover, the Hu's system is incomplete. There are two third-order invariants } c c and c c } that are independent of + ,2, ,. Li [33] published a set of invariants from moments up to the ninth order. Unfortunately, his system includes the Hu's one as its subset and therefore it also cannot be a basis. Wong [34] presented a set of 16 invariants from moments up to the third order and a set of `more than 49a invariants from moments up to the fourth order. It follows immediately from Theorem 2 that a basis of the third-order invariants has only six elements and a basis of the fourth-order invariants has 11 elements (these numbers relate to the real-valued invariants). Thus, most of Wong's invariants are dependent. The systems of invariants published by Teague [35] and Wallin [36] are also dependent. We can, however, obtain an independent system just by removing all skew invariants. The proof of completeness is given in Ref. [36] but this term is in that paper de"ned as a possibility to recover all the moments up to the given order from the set of invariants. 4.2. An explicit construction of the third-order and fourth-order bases In this section, we present the bases of low-order invariants that have been constructed according to Theorem 2 and that we recommend to use for 2-D object recognition. E Second and third orders: t "c " , t "c c " , t "Re(c c )" , t "Im(c c ) "k ((k #k )!(k #k )) !(k !k )(k #k )(k #k ), t "Re(c c )" , t "Im(c c )" . E Fouth order: t "c , t "Re(c c ), t "Im(c c ), t "Re(c c ), t "Im(c c ).

In the case of third-order and fourth-order invariants, the bases are determined unambiguously. However, there are various possibilities as to apply Theorem 2 when constructing higher-order bases. The di!erence is in the choice of the indices p and q . Although it is not strictly required, it is highly desirable always to choose p and q as small as possible, because low-order moments are less sensitive to noise than the higher-order ones.

5. Skew invariants In this section, we investigate the behavior of the rotation invariants under re#ection. The invariants, which do not change their values under re#ection are traditionally called the true invariants while the others are called skew invariants [1] or pseudoinvariants [35]. Skew invariants distinguish between the mirrored images of the same object that is useful in some applications but may be undesirable in other cases. In the following text we show which invariants of those introduced in Theorem 1 are skew and which are the true ones. Let us consider an invariant of the type (7) and let us investigate its behavior under the re#ection across an arbitrary line. Due to the rotation and shift invariance, we can restrict ourselves to the re#ection across the x-axis. Without loss of generality, we consider the invariants from the basis only. Let fN (x, y) be a re#ected version of f (x, y), i.e. fN (x, y)"f (x, !y). It follows from Eq. (3) that c "cH . NO NO Thus, it holds for any basic invariant '(p, q) "cH (cH )N\O"'(p, q)H. '(p, q)"c cN\O NO ON NO O N This indicates that the real parts of the basic invariants are true invariants. On the other hand, the imaginary parts of them are skew invariants, because they change their signs under re#ection.

6. Summary and conclusion In this paper, the problem of the independence and completeness of the rotation moment invariants was discussed. Although the moment invariants have attracted a signi"cant attention of pattern recognition community within the last thirty years, they have not been studied from this point of view as yet. A general method how to derive the rotation invariants of any order was described "rst. Then the theorem showing what the basis of the invariants looks like was formulated and proven. This is the major theoretic result of the paper. Finally, the relationship to the previous works was demonstrated. As an interesting consequence


of our results, it was shown that Hu's system of moment invariants is dependent and incomplete.

Acknowledgements This work has been supported by the grants Nos. 102/96/1694 and 106/97/0827 of the Grant Agency of the Czech Republic.

Appendix A. Proof of Theorem 2 Completeness of B: Let I be an arbitrary element of I. Thus L I" cIG , NGOG G where c 3M6M*. The product can be decomposed NGOG into two factors according to the relation between p and G q: G L L I" cIGG G cIGG G NO NO G GL> where p *q if i)n and p (q if i'n . G G G G Let us construct another invariant J from the elements of B only as follows:

1409

independence of the moments, it must hold p"p and q"q . That means, according to the above assumption, there exist invariants '(p , q ),2,'(p , q ) and L L '(s , t ),2,'(s , t ) from B!+'(p , q ), such that K K '(p , q )HIG L '(p , q )IGL (A.1) '(p , q )" G G G l GL> G G l . K '(s , t ) GK '(s , t )H G G G G GK> G G Substituting into Eq. (9) and grouping the factors c and c together, we get NO ON IGNG\OGc LGL >IGOG\NGL cIG L c LG cIG NO G NGOG GL> lOGNG. '(p , q )" O NK l c G GQG\RGc KGK >lGRG\QGK clG K c GG G ON NO G QGRG GK> R Q (A.2) Comparing the exponents of c and c on the both NO ON sides we get the constraints K L K " k (p !q )! l (s !t )"1 G G G G G G G G and

(A.3)

L K K " k (q !p )! l (t !s )"1. (A.4) G G G G G G GL> GK> Since the rest of the right-hand side of Eq. (A.2) must be equal to 1 and since the moments themselves are mutually independent, the following constraints must be ful"lled for any index i:

L L J" '(p , q )IG '(q , p )HIG. G G G G G GL> Grouping the factors c and c together we get ON NO

n "m , n"m, p "s , q "t , k "l . G G G G G G Introducing these constraints into Eqs. (A.3) and (A.4), we get K "K "0 that is a contradiction 䊐.

L L L L J"c G IGNG\OGc GL>IGOG\NG cIGG G cIGG G ON NO NO NO G GL>

References

L G G G

L GL> G G

I O \NGI. "c I N \OG c ON NO Since I is assumed to be an invariant, it must hold L L k (p !q )# k (p !q )"0 G G G G G G G GL> and, consequently, L L k (p !q )" k (q !p )"K. G G G G G G G GL> Now I can be expressed as a function of the elements of B: I"'(p , q )\)J. Thus, I has been proven to be dependent on B. Independence of B: Let us assume B is dependent, i.e. there exists '(p, q)3B, such that it depends on B ! +'(p, q),. As follows immediately from the mutual

[1] M.K. Hu, Visual pattern recognition by moment invariants, IRE Trans. Inform. Theory 8 (1962) 179}187. [2] G.B. Gurevich, Foundations of the Theory of Algebraic Invariants, Noordho!, Groningen, The Netherlands, 1964. [3] I. Schur, Vorlesungen uber Invariantentheorie, Springer, Berlin, 1968. [4] S.A. Dudani, K.J. Breeding, R.B. McGhee, Aircraft identi"cation by moment invariants, IEEE Trans. Comput. 26 (1977) 39}45. [5] S.O. Belkasim, M. Shridhar, M. Ahmadi, Pattern recognition with moment invariants: a comparative study and new results, Pattern Recognition 24 (1991) 1117}1138. [6] R.Y. Wong, E.L. Hall, Scene matching with invariant moments, Comput. Graphics Image Process. 8 (1978) 16}24. [7] A. Goshtasby, Template matching in rotated images, IEEE Trans. Pattern Anal. Mach. Intell. 7 (1985) 338}344. [8] J. Flusser, T. Suk, A moment-based approach to registration of images with a$ne geometric distortion, IEEE Trans. Geosci. Remote Sensing 32 (1994) 382}387. [9] R. Mukundan, K.R. Ramakrishnan, An iterative solution for object pose parameters using image moments, Pattern Recognition Lett. 17 (1996) 1279}1284.

1410


[10] R. Mukundan, N.K. Malik, Attitude estimation using moment invariants, Pattern Recognition Lett. 14 (1993) 199}205. [11] A. Sluzek, Identi"cation and inspection of 2-D objects using new moment-based shape descriptors, Pattern Recognition Lett. 16 (1995) 687}697. [12] F. El-Khaly, M.A. Sid-Ahmed, Machine recognition of optically captured machine printed arabic text, Pattern Recognition 23 (1990) 1207}1214. [13] K. Tsirikolias, B.G. Mertzios, Statistical pattern recognition using e$cient two-dimensional moments with applications to character recognition, Pattern Recognition 26 (1993) 877}882. [14] A. Khotanzad, Y.H. Hong, Invariant image recognition by Zernike moments, IEEE Trans. Pattern Anal. Mach. Intell. 12 (1990) 489}497. [15] J. Flusser, T. Suk, A$ne moment invariants: A new tool for character recognition, Pattern Recognition Lett. 15 (1994) 433}436. [16] S. Maitra, Moment invariants, Proc. IEEE 67 (1979) 697}699. [17] T.M. Hupkens, J. de Clippeleir, Noise and intensity invariant moments, Pattern Recognition 16 (1995) 371}376. [18] L. Wang, G. Healey, Using Zernike moments for the illumination and geometry invariant classi"cation of multispectral texture, IEEE Trans. Image Process. 7 (1998) 196}203. [19] L. van Gool, T. Moons, D. Ungureanu, A$ne/photometric invariants for planar intensity patterns, Proceedings of the Fourth ECCV'96, vol. 1064, Lecture Notes in Computer Science, Springer, Berlin, 1996, pp. 642}651. [20] J. Flusser, T. Suk, S. Saic, Recognition of blurred images by the method of moments, IEEE Trans. Image Process. 5 (1996) 533}538. [21] J. Flusser, T. Suk, Degraded image analysis: an invariant approach, IEEE Trans. Pattern Anal. Mach. Intell. 20 (1998) 590}603. [22] R.J. Prokop, A.P. Reeves, A survey of moment-based techniques for unoccluded object representation and recognition CVGIP: Graphical Models and Image Processing, Vol. 54, 1992, pp. 438}460

[23] C.H. Teh, R.T. Chin, On image analysis by the method of moments, IEEE Trans. Pattern Anal. Mach. Intell. 10 (1988) 496}513. [24] Y.S. Abu-Mostafa, D. Psaltis, Recognitive aspects of moment invariants, IEEE Trans. Pattern Anal. Mach. Intell. 6 (1984) 698}706. [25] S.X. Liao, M. Pawlak, On image analysis by moments, IEEE Trans. Pattern Anal. Mach. Intell. 18 (1996) 254}266. [26] M. Pawlak, On the reconstruction aspects of moment descriptors, IEEE Trans. Inform. Theory 38 (1992) 1698}1708. [27] R.R. Bailey, M. Srinath, Orthogonal moment features for use with parametric and non-parametric classi"ers, IEEE Trans. Pattern Anal. Mach. Intell. 18 (1996) 389}398. [28] Y.S. Abu-Mostafa, D. Psaltis, Image normalization by complex moments, IEEE Trans. Pattern Anal. Mach. Intell. 7 (1985) 46}55. [29] M. Gruber, K.Y. Hsu, Moment-based image normalization with high noise-tolerance, Pattern Recognition 19 (1997) 136}139. [30] L. Yang, F. Albregtsen, Fast and exact computation of cartesian geometric moments using discrete Green's theorem, Pattern Recognition 29 (1996) 1061}1073. [31] J. Flusser, T. Suk, Pattern recognition by a$ne moment invariants, Pattern Recognition 26 (1993) 167}174. [32] T.H. Reiss, The revised fundamental theorem of moment invariants, IEEE Trans. Pattern Anal. Mach. Intell. 13 (1991) 830}834. [33] Y. Li, Reforming the theory of invariant moments for pattern recognition, Pattern Recognition 25 (1992) 723}730. [34] W.H. Wong, W.C. Siu, K.M. Lam, Generation of moment invariants and their uses for character recognition, Pattern Recognition Lett. 16 (1995) 115}123. [35] M.R. Teague, Image analysis via the general theory of moments, J. Opt. Soc. Am. 70 (1980) 920}930. [36] A. Wallin, O. Kubler, Complete sets of complex Zernike moment invariants and the role of the pseudoinvariants, IEEE Trans. Pattern Anal. Mach. Intell. 17 (1995) 1106}1110.

About the Author*JAN FLUSSER was born in Prague, Czech Republic, on April 30, 1962. He received the M.Sc. degree in mathematical engineering from the Czech Technical University, Prague, Czech Republic in 1985 and the Ph.D. degree in computer science from the Czechoslovak Academy of Sciences in 1990. Since 1985 he has been with the Institute of Information Theory and Automation, Academy of Sciences of the Czech Republic, Prague. Since 1995 he is the head of Department of Image Processing. Since 1991 he has been also a$liated with the Faculty of Mathematics and Physics, Charles University, Prague, where he gives the courses on Digital Image Processing. His current research interests include digital image processing, pattern recognition and remote sensing. He has authored or coauthored more than 30 scienti"c publications in these areas. Dr Flusser is a member of the Pattern Recognition Society, the IEEE Signal Processing Society, the IEEE Computer Society and the IEEE Geoscience and Remote Sensing Society.


Coarse-to-"ne planar object identi"cation using invariant curve features and B-spline modeling Yu-Hua Gu *, Tardi Tjahjadi Department of Signals and Systems, Chalmers University of Technology, Gothenburg, S-41296, Sweden School of Engineering, University of Warwick, Coventry, CV4 7AL, UK Received 7 January 1999; accepted 1 June 1999

Abstract This paper presents a hybrid algorithm for coarse-to-"ne-matching of a$ne-invariant object features and B-spline object curves, and simultaneous estimation of transformation parameters. For coarse-matching, two dissimilar measures are exploited by using the signi"cant corners of object boundaries to remove candidate objects with large dissimilarity to a target object. For "ne-matching, a robust point interpolation approach and a simple gradient-based algorithm are applied to B-spline object curves under MMSE criterion. The combination of coarse and "ne-matching steps reduces the computational cost without degrading the matching accuracy. The proposed algorithm is evaluated using a$ne transformed objects. 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Curve matching; Feature extraction; Dissimilar measures; B-splines; Object recognition

1. Introduction The identi"cation, matching and analysis of objects of interest are of prime importance in application domains including target tracking in tra$c [1,2], robot autonomous navigation [3], reconstruction of body structures in medicine [4], the use of eyes as an interface for controlling computer graphics [5], gesture analysis and recognition [6], image browsing and retrieval [7], and object-oriented video coding [8]. Object matching means establishing correspondences between object boundaries, shapes, texture and various other features in a set of images. Since an object may undergo various a$ne transformations including rotations, scalings and translations, object-matching also means computing any transformation parameters based on 2D or 3D object representations [9]. These methods can be further classi"ed as boundary, region and model-based methods [10].

* Corresponding author. Tel.: #46-31-772-1796; fax: #4631-772-1782. E-mail address: [email protected] (Y.-H. Gu).

If an object is simple, the matching and analysis of a$ne objects could be successfully performed using boundarybased approaches. A direct approach is to match object boundary curves extracted from images, where the objects may undergo various a$ne transformations (e.g. rotations, scalings and translations) and small deformations. One method to compare two object curves is to match the corresponding points on two curves assuming a$ne rigid objects [11]. However, the method is both sensitive to noise because of imperfect data, and to imprecise correspondences. Instead of point matching, a variety of less noise-sensitive approaches have been proposed based on comparing corresponding line segments and polygonal arcs [12,13]. Many techniques have also been proposed for representing a curve by a set of features or by models. These include Fourier descriptors [14], B-splines [15,16], autoregressive model [17], moments [18], curvatures [19], HMM models [20] and wavelets [21]. Among these methods, B-splines are often used in computer graphics because of their properties of continuity, local shape controllability, and a$ne invariance [22]. In [15] a scheme for matching and classi"cation of B-spline curves is presented. However, it requires two separate


1412

Y.-H. Gu, T. Tjahjadi / Pattern Recognition 33 (2000) 1411}1422

stages for classifying objects and estimating the corresponding a$ne parameters. Furthermore, the method uses the point where a curve crosses the positive horizontal x-axis, which is arbitrary for a$ne curves, as the starting point. This will result in a lack of good correspondences between curve segments and hence increase the error in curve matching. Motivated by the above problems, we present a hybrid two-step matching scheme which explores both dissimilarities of object features and of B-spline approximated curves. In the feature-based coarse-matching step, a small number of signi"cant corners is extracted from each discrete object curve to form a$ne-invariant object features. Since objects may undergo various a$ne transformations, features which are a$ne invariant are more attractive for object-matching. Many curve features, e.g., arc length and the area within an object boundary, are a$ne invariant [23,24]. We introduce two dissimilar measures on a$ne-invariant features based on signi"cant corners. These enable a fast identi"cation between a$ne and non-a$ne objects based on a small set of features. To maintain high reliability in object identi"cation, only candidate objects with large a$ne dissimilarities to a target object are rejected in the coarse-matching step. The non-rejected objects are then passed to the "ne-matching process, which not only enables matching based on the entire curve-shape information, but also enables the estimation of parameters associated with a$ne objects. We also introduce a gradient-based algorithm which simultaneously matches B-spline modeled object curves and estimates transformation parameters. This algorithm incorporates a robust method of assigning corresponding curve points via interpolation. The remaining of the paper is organized as follows. Section 2 describes the coarse-matching algorithm including the method for estimating signi"cant corners and the two dissimilar measures. Section 3 describes the "nematching algorithm, including the selection of corresponding curve points and the gradient-based MMSE matching algorithm. Section 4 presents the experimental results and evaluations. Finally Section 5 concludes the paper.

2. Coarse-matching based on a7ne-invariant features 2.1. Extracting signixcant corner points A corner point on an object curve subtends a sharp angle between its neighboring points. Signi"cant corners of an object curve are estimated and exploited for both coarse and "ne-matching. Let the boundary curve of an object be described by discrete points r "[x y ], I I I k"0, 1,2, n!1, and denote matrix transpose. Since the x and y coordinates of each curve point can be handled separately, the vector form will not be empha-

sized throughout the paper. De"ne an angle u assoI ciated with each point r as the angle between the two I vectors (r , r ) and (r , r ), i.e., I> I I I\ a#b!c u "cos\ , (1) I 2ab

where a""r !r ", b""r !r " and c""r !r ". I\ I I> I I> I\ To prevent arti"cial variation in angular values due to discrete points, a curve is smoothed before the angles are calculated. A smoothed curve rQK"[xQKyQK], k" I I I 0, 1,2, n!1, is obtained as 1 I>U 1 I>U xQK" x, yQK" y , (2) I J I J 2w #1 2w #1 JI\U JI\U where w is a small integer number. Signi"cant corners are those points which have the local minimum angular values below a pre-determined threshold. To prevent them being too closely located to each other, only one signi"cant corner is allowed in each curve interval w . That is, the angle u of a signi"cant corner point r must I I satisfy +u :" u "local minimum angle; u , min u; I I I J JZ I\U I>U

((u (Threshold) or (the ¸ most signi"cant I corners)),,

(3)

where the ¸ most signi"cant corners correspond to the ¸ smallest u . I 2.2. Dissimilar measures Let the ith object be described by the discrete boundary curve +rG ,, k"0, 1,2, n!1, and its centroid be I rG "[xG yG ]. Denote r G "rG !rG as the curve points havA A A I I A ing the coordinate origin shifted to the object centroid, and cnG and cn G , j"0, 1,2, ¸!1, as the signi"cant H H corners before and after shifting the coordinate origin. Further, assume the discrete points +rG , are densely I populated on the original curve such that the arc length between any two adjacent points is small. 2.2.1. Dissimilar Measure 1 Dissimilar Measure 1 computes the dissimilarity of two objects based on the accumulated di!erence of normalized arc lengths between adjacent signi"cant corners of the two object curves. Without loss of generality, assume that adjacent corners cnG and cnG are equal to H\ H curve points rG and rG , respectively. The arc length I I between two adjacent signi"cant corners is approximated by the accumulated chord length of adjacent curve points within these two corners, i.e., I len(cnG , cnG )" "rG !rG ", H\ H I I\ II>

(4)


where j"0, 1,2, ¸!1, k implies mod (k) and j implies L mod ( j), (e.g. mod (7)"2), due to the closed curve. * Eq. (4) is a good approximation under the assumption that +rG , are densely populated. Since arc lengths are I a$ne invariant [24], arc lengths between adjacent signi"cant corners are also a$ne invariant. To obtain scale invariance, arc lengths in Eq. (4) are normalized by the total length of the object curve as follows: len(cnG , cnG ) H\ H . ¸EN(cnG , cnG )" (5) H\ H *\len(cnG , cnG ) H\ H H Dissimilar Measure 1 is de"ned as the accumulated difference of corner-to-corner arc lengths between the two object curves l and m, *\ dissim (l, m)" min "¸EN(cnJ , cnJ ) H>Q\ H>Q WQW*\ H !¸EN(cnK , cnK)", (6) H\ H where s is used to eliminate the in#uence of rotation between the two objects. 2.2.2. Dissimilar Measure 2 Dissimilar Measure 2 computes the dissimilarity of two objects based on the accumulated di!erence of corner-to-centroid distance between the two object curves. Obviously, it is invariant to object translation and rotation. To obtain scale invariance, the scale factor b is estimated from the ratio of the curve lengths of two objects by using Eq. (4), i.e., *\len(cnJ , cnJ ) H\ H . bK " H (7) *\len(cnK , cnK) H\ H H The proof of Eq. (7) is straight forward: Since r J I is assumed to be a scaled version of r K, r J "br K, it folI I I lows that *\len(cn J , cn J )" L\ " r J !r J "" H\ H I\ H I I L\b " r K!r K ""b *\len(cn K ,cn K), where I I\ H\ H H I "r !r ""((x !x )#(y !y ). Note the I I\ I I\ I I\ translational-invariant property of any arc length, len(cn , cn ) in Eq. (7) is equal to len(cn , cn ). The H\ H H\ H Dissimilar Measure 2 between curves l and m with respect to l is de"ned as *\ "cn J !bK cn K" Q>H H . dissim (l, m)" min (8) "cn J " WQW*\ H Q>H Similar to Eq. (6), s is applied to eliminate the in#uence of rotation between the two objects. 2.3. Coarse-matching algorithm In coarse-object-matching, the two dissimilar measures are used for matching S candidate objects to a target object. If both dissim *¹h1 and dissim *¹h2 are satis"ed for a candidate object, where ¹h1 and ¹h2

1413

Table 1 Algorithm for coarse-matching For all candidate curves and a given mth target curve, do: For the lth candidate curve, do: Compute smoothed curve points rQK using Eq. (2); I Compute angle u using Eq. (1), where a, b and c are I calculated using rQK; I Extract ¸ signi"cant corners using Eq. (3); Compute the arc length between adjacent signi"cant corners len(cn , cn ) using Eq. (4); H\ H Compute the normalized arc length ¸EN(cn , cn ) using H\ H Eq. (5); Calculate dissim using Eq. (6); Estimate bK using Eq. (7), and calculate dissim using Eq. (8); If (dissim *¹h1 and dissim *¹h2), then eliminate this curve from the candidate list; otherwise forward this curve to the "ne-matching algorithm.

are pre-determined thresholds, this candidate object is considered to be a$ne dissimilar to the target object, and is thus removed from the candidate list. Otherwise, the candidate object is passed to the "ne-matching process to be analyzed further. The coarse-matching algorithm is summarized in Table 1. Since these dissimilar measures are based on a few sparsely located corners, the coarsematching process is only designed for discriminating candidate objects which are obviously a$ne di!erent from the target object. In general, di!erent types of object contain di!erent number of signi"cant corners. However, objects of similar type, e.g., hand-gesture curves or aircrafts, may contain a similar number of signi"cant corners. If the di!erence in number between two unknown objects is large, e.g., more than 50%, it is assumed that the two objects are a$ne dissimilar. Therefore, the algorithm simply regards the candidate object as dissimilar without applying the dissimilar measures. This object is then removed from the candidate list. Otherwise, extra corners are added to the object which has fewer corners by selecting the "nal number of signi"cant corners ¸"max +¸ , in Eq. (3), where ¸ is the initial number of G G G signi"cant corners for the ith object in a group.

3. Fine-matching of object curves The "ne-matching method is applied to those nonrejected candidate curves from the coarse-matching. The criterion is to minimize the MSE between the continuous B-spline target curve and B-spline candidate curves with respect to the unknown transformation parameters. However, the e$cient implementation requires comparing two sets of B-spline curve points at corresponding curve positions. This ensures that the resulting MMSEs

1414


from matching truly re#ect the a$ne dissimilarity of objects, while the contributions of error from inaccurate point positions on curves are negligible. We present a robust approach for selecting points from each B-spline curve, which results in good correspondences if two objects are a$ne related. This includes exploiting signi"cant corners and interpolating an extra set of corresponding points on each curve. We then present a gradient-based algorithm which simultaneously estimates the transformation parameters and sub-optimally determines the a$ne-similar object from all candidate objects using the MMSE criterion. To limit the scope of this paper, only objects with closed boundary curves are considered. The types of a$ne transformation are limited to translations, rotations and scalings. Further, cubic B-spline is chosen as a tradeo! between computation and smoothness of curves [22]. 3.1. B-spline curve modeling and approximation For "ne object-matching, a discrete-object curve +r ,, k"0, 1,2, n!1, is modeled by a continuous BI spline curve r(t). Cubic B-spline curve model is brie#y reviewed here in order to facilitate the subsequent descriptions and clarify the notations used. Readers are referred to de Boor [22] and Rogers and Adames [25] for further details. Let a discrete closed curve consist of vector points r "[x y ], k"0, 1,2, n!1, in a 2D plane, and be I I I modeled by a continuous periodic cubic B-spline curve r(t)"[x(t) y(t)], where t3[t , t ] is a parameter.

A B-spline knot point r( "[x( y( ], j"0, 1,2, m!1, is H H H de"ned as a position vector at the connecting point of two B-spline curve segments, and is equal to the discrete curve sample. For B-spline curve modeling, m"n holds. The jth curve segment of a B-spline curve can be described by the cubic B-spline coe$cients (or, the control points) +C , on the four basis functions +N (t),: H G r (t)" C N (t), H H>G\ K G G

(9)

where t (min))t)t (max), C "C , and C "C . H H K \ K\ The basis functions N (t) may be derived from the G Cox}deBoor recursive formulae [22]. A B-spline curve with m normalized segments is represented by r(t)" K\r (t!t ), where t3[t , t ], and t"t!t 3 [0,1] H K\ H H H is the range of normalized parameter in the jth segment. It follows that r(t) can then be spanned on (m#3) normalized cubic B-spline basis functions Q (t) with coe$H cients +C ,, H K> r(t)" C Q (t). H> K H H

(10)

Setting t"t in Eq. (10) yields the knot points H r( , j"0, 1,2, m!1. A B-spline curve is fully described H by m knot points +r( ,, or, by m coe$cients +C , [22]. H H If m(n, the resulting B-spline curve is an approximation of the original curve. B-spline curve approximation involves determining the &best' m and "nding the m &best' knot points, or coe$cients, such that the resulting curve is as close as possible to the original curve under a given criterion. This is non-trivial in general [22,26]. In this paper, knot points of a Bspline object curve are chosen using a constraint active B-spline curve model under MMSE criterion which automatically includes all signi"cant corners (de"ned in Section 2.1) as a subset of knot points. Let a B-spline object curve r(t) be described by m knot points r( and ¸ of H these knot points are equal to the signi"cant corners on the curve, i.e., cn 3+r( ,, s"0, 1,2¸!1, and ¸(m. Q H The constraint active B-spline model "nds the &best' remaining (m!¸) knot points such that the mean-square error, MSE(r( ,2,r( )"(1/n) L\#r !r(t )#, is mini* K\ I I I mized, where r is a discrete sample from the original I curve, r(t ) is either a B-spline knot point or an interI polated point, and t is a parameter. To limit the scope of I this paper, the detailed algorithm for "nding these knot points is not included in the paper. 3.2. Method for selecting curve points In order to calculate the MMSE distance between B-spline object curves, the same number of curve points must be chosen at corresponding positions on the curves to be compared. Let a B-spline object curve be represented by m knot points, i"1, 2. In general, m Om , or, G m "m but the curve segments of two a$ne-object curves lack correspondences. The task is to interpolate a small number of m points at corresponding positions so that the distance between the two curves can be computed. The selected "nal curve points of each object may contain three parts: (a) the knot points from the B-spline object curve itself; (b) points interpolated at corresponding positions of the knot points from another object to be compared; (c) extra points by uniform interpolation. The basic set of points is selected from (a) and (b), and contains the minimum number of points as follows: All knot points including signi"cant corners are included in the basic set. In addition, a set of points is interpolated using the lengths of B-spline curve segments and the sharpest signi"cant corner as the starting point. Assume curve i, rG(t), i"1, 2, contains m knots. To interpolate G points rG(t ), t are determined from the normalized H H lengths of curve segments (i.e., the ratios of the lengths of curve segments to the total length of the curve) by applying Eq. (5) to the knot points. Starting from the sharpest signi"cant corner of curve 1, (m !1) new points r(t ) are interpolated at t equal to H H


the normalized curve segment lengths of curve 2. Similarly, using the sharpest corner of r(t) and the normalized curve segment lengths of r(t), (m !1) new points r(t ) are interpolated on curve 2. Once t are H H determined, points rG(t ) are interpolated using Eq. (10). H Hence, the basic set for each curve contains m" (m #m !1) corresponding points. As an example, Fig. 1 illustrates the point interpolation process for twohand curves, one is rotated by 303. Initially, the hand curves contain 39 and 52 knot points, respectively. If the number of points in the basic set is very small, more corresponding points may be interpolated to each of the object curves. As indicated in (c), this is done by uniformly selecting on both curves the same normalized parameters t starting from their sharpest corner. H Whether these extra points are required is a tradeo! between the computation and the accuracy of estimation. Our experiments indicate that using points in the basic set is su$cient in most cases, e.g., in the hand curves, a basic set contains approximately m"70 points and results in MMSE close to those using 600 points. 3.3. B-spline object-matching and parameter estimation A gradient-based algorithm is introduced to simultaneously compute the MMSE distance between two object curves, and to estimate the associated translation, rotation and scaling parameters. Let r2(t) be the B-spline curve of a target object boundary curve, and rQ(t), s"1, 2,2, S, be the candidate object-boundary curves. The mean-square error (MSE) distance between r2(t) and rQ(t) is calculated by 1 K\ e(r2(t), rQ(t))" #r2(t )!bR(h)rQ(t )!b#, (11) H H m H where r(t ) denote B-spline points including both knot H points and interpolated points.

1415

The objective of object-matching is to "nd the &best' a$ne-similar object curve between the target r2(t) and one of the candidate curves rQ(t), s"1, 2 ,2, S, such that the two curves have a minimum distance with respect to the unknown parameters, i.e, e (r2(t),

rQ(t))"min (min e(r2(t), rQ(t))). If this minimum disQ @F@ tance is greater than an acceptable level, all candidate curves are rejected. A sub-optimal matching process is applied. First, the displacement vector is estimated from the distance between the estimated centroids of the two object curves. Then, using the gradient-based method, the error e(r2(t), rQ(t)) is minimized with respect to b and h, i.e., by setting *e/*h "0 and *e/*b "0. The rotaI I I I tion and scaling parameters at (k#1)th iteration are updated from the kth iteration using the steepest-descent method: h "h !k I, I> I F

b "b !k I, I> I @

(12)

where I"*e/*h , I"*e/*b , k and k are small F I I @ I I positive constants which control the speed of convergence and the steady-state performance. De"ning d(r(t ))"r2(t )!bR(h)rQ(t )!b, Eq. (11) is H H H equivalent to e(r2(t), rQ(t))"(1/m) K\d(r(t )). Noting H H e"[e(x) e(y)], d(r(t ))"[d(x(t )) d(y(t ))], r(t)"[x(t) y(t)], H H H b"[b b ] and V W

R"

cos h

sin h

!sin h cos h

,

the 2 elements in d(r(t )) are equal to H d(x(t ))"x2(t )!b(cos h xQ(t )#sin h yQ(t ))!b , H H H H V d(y(t ))"y2(t )!b(!sin hxQ(t )#cos h yQ(t ))!b H H H H W (13)

Fig. 1. Interpolating corresponding curve points on two hand curves. (&o': the estimated sharpest corner). From left to right: (a) A hand curve with 39 knot points &*'; (b) The rotated hand curve with 52 knot points &*'; (c) Interpolated points &#' on (a); (d) Interpolated points &#' on (b).

1416


and "d(r(t ))""d(x(t ))#d(y(t )) holds. Hence, *e/*h " H H H I I (1/m) K\(*d(x(t ))/*h #*d(y(t ))/*h ) and *e/*b " H I H I I H I I I (1/m) K\(*(d(x(t ))/*b #*(d(y(t ))/*b ), which are H I H I I H I equivalent to

extending Eq. (11) to include both types of curves

2b K\

I" I d (x(t ))(sin h xQ(t )!cos h yQ(t )) F I H I H I H m H

where l is the number of curves, including the object boundary and all its inner curves.

#d (y(t ))(cos h xQ(t )#sin h yQ(t )), I H I H I H

(14) 4. Experimental results

2 K\

I"! d (x(t ))(cos h xQ(t )#sin h yQ(t )) @ I H I H I H m H !d (y(t ))(sin h xQ(t )!cos h yQ(t )). I H I H I H

(15)

The new b and h are obtained using Eq. (12). I> I> Substituting b and h into Eq. (11) and noting I> I> Eq. (13) yields e (r2(t ), rQ(t )) I> H H 1 K\ " +(x2(t )!b (cos h xQ(t ) H I> I> H m H #sin h

I>

1 KJ\ e(r2(t), rQ(t))" #r2J(t )!bR(h)rQJ(t )!b#, H H m J J J H

yQ(t ))!b )#(y2(t ) H V H

#b (sin h xQ(t )!cos h yQ(t ))!b ),. (16) I> I> H I> H W The iteration is repeated until Eq. (16) converges, or a pre-de"ned maximum number of iteration is reached. For each candidate curve s, the iterations can be summarized as updating h using Eqs. (12) and (14), upI> dating b using Eqs. (12) and (15), and updating I> e (r2(t ), rQ(t )) using Eq. (16). This process is repeated I> H H for all candidate curves. The candidate curve s which has the minimum e(r2(t), rQ(t)) value, s"1, 2,2, S, is selected, i.e., s "argmin +e(r2(t), rQ(t)),. If this value is Q within an acceptable level then a correct match is assumed, and the corresponding parameters are assigned as the transformation parameters. Otherwise all candidate curves are rejected. In situations where a rigid object is characterized not only by its outside boundary but also by some inner closed curves, the algorithm may easily be generalized by

Four sets of object boundaries, including hand-gesture curves, tools, aircrafts and animals, were used for tests. These object boundaries were extracted using the edgecurve estimation algorithms [27,28] followed by a boundary tracing and closing algorithm [29]. Figs. 2}5 show, respectively, the four sets of closed object-boundary curves. A$ne hand curves were obtained by applying various rotations, scalings and translations to the original discrete curves. For all other object curves, a$ne curves were obtained by applying various rotations, scalings and translations to the original two-tone images followed by boundary extraction. 4.1. Evaluation of the coarse-matching algorithm The four sets of curves were used to evaluate the coarse-matching algorithm. The initial number of signi"cant corners for each curve are listed in Table 2. These initial corners are marked on the curves in Figs. 2}5. For each set of curves, the number of "nal signi"cant corners ¸ was set equal to the maximum number of initial signi"cant corners in that set (3rd column of Table 2). For each object curve, dissimilar measures were applied to its a$ne transformed curves (denoted as &a$ne curves') as well as other object curves (i.e., non-a$ne-similar curves or non-a$ne curves). Since the estimated signi"cant corners of two a$nerelated curves may not correspond at the exact curve locations, it is desirable that the dissimilar measures be insensitive to small positional shifts of corners due to estimation errors. To determine the sensitivity of the

Fig. 2. Hand-gesture curves with marked corners. From left to right: hd1}hd6. &*': the signi"cant corner; &o': the sharpest corner.


1417

Fig. 3. Boundaries of planes with marked corners. From left to right, and top to bottom: plane 1 to plane 9. &*': Signi"cant corner; &o': the sharpest corner.

Fig. 4. Boundaries of animals with marked corners. From left to right, and top to bottom: a1}a8. &*': Signi"cant corner; &o': the sharpest corner.

dissimilar measures, dissimilar values were computed by shifting signi"cant corners of a$ne-related curves within a range of discrete sample points along the original curve. The set of six hand curves was used for such tests. The

corresponding dissimilar values were then compared to determine whether large di!erences appear from small positional shifts of corners. Table 3 shows the test results, where the range of shifts are indicated by `shifted

1418


Fig. 5. Boundaries of tools with marked corners. From left to right: tool1}tool4. &*': Signi"cant corner; &o': the sharpest corner.

Table 2 Number of initial and "nal signi"cant corners Objects

No. of initial corners

No. of "nal corners (¸)

hd1}hd6 Plane 1}Plane 9

11, 8, 9, 9, 10, 7 14, 16, 20, 19, 17, 20, 16, 17, 15 17, 16, 18, 11, 20, 14, 13, 16 2, 12, 6, 9

11 20

a1} a8 Tool1}Tool4

20 12

samples from cornersa. The results show that the dissimilarities of a$ne curves are much smaller than that of non-a$ne-similar curves, hence these two types of curve are well separated in the feature spaces. These results also hold when the signi"cant corners of two a$ne curves do not correspond exactly due to the shifts. Consequently, the two proposed dissimilar measures are e!ective criteria for discriminating candidate objects which are obviously di!erent from the target object. Tables 4}6 show the measured dissimilar values, dissim and dissim , and the associated mean values of dissim and dissim for all non-a$ne objects in the sets of planes, animals and tools, respectively. For a$ne object curves, the values of dissim and dissim were in the order of 10\ (not included in the tables), well below one half of the corresponding mean dissimilar values of non-a$ne objects. These results indicate that the proposed dissimilar measures are valid and e!ective measures, since there is a large inter-class distance between the features of a$ne and non-a$ne objects.

a small number of knot points (3rd column of Tables 7}10) including the number of signi"cant corners (¸ in Table 2). In all these tests k "k "0.1 were selected, using the principles that a smaller k gives a better G steady-state performance but a slower convergence, while a large k can cause the algorithm to be unstable and to G oscillate around the minimum. Tables 7}10 show the results from the "ne object-matching algorithm. In these tables, the accuracies of the estimated parameters were evaluated by the di!erence in rotations "*h"""hK !h", in scalings "*b"""bK !b", and in translations "*b""((bK !b )#(bK !b ). The MMSE values V V W W from matching all non-a$ne object curves in the same set are also included in these tables as references. The results consistently showed that the algorithm has estimated parameters with high accuracy and with small MMSEs for a$ne objects. The convergence of the algorithm was fast, with an average number of 50 iterations for hand curves, 49 for aircraft curves, 46 for animal curves, and 42 for tool curves. Further, for all non-a$ne objects, much larger MMSE values were obtained. Consequently, the "ne-matching algorithm easily distinguishes a$ne and non-a$ne object curves due to the large separable MMSE distances between these two cases. Comparing the results of matching aircraft curves in Table 8 to those shown in Tables II}IV of Cohen et al. [15] (the "rst two rows corresponding to rotations, scalings and translations), our results show a signi"cantly increased capability for object discrimination, i.e., with smaller MMSE values for all a$ne-similar planes (approximately 10 times smaller), and larger MMSE values for the remaining non-a$ne plane curves (approximately 10 times larger). Consequently, our algorithm has a greater potential in discriminating object curves.

4.2. Evaluation of the xne-matching algorithm 4.3. Coarse-to-xne-matching of objects The four sets of object curves and their a$ne transformed versions were used to evaluate the curve matching and parameter estimation using only the "nematching algorithm. In the experiments, each object curve was approximated by the cubic B-spline using

The coarse-to-"ne-matching algorithm was applied, respectively, to the four sets of object curves and their a$ne transformed versions. Since dissimilar values between a$ne object curves were well below the mean


1419

Table 3 Coarse-matching of a$ne and non-a$ne-similar hand curves. hd1}hd6: the original hand curves; t-hd1}t-hd6: a$ne transformed hand curves Object

Objects compared

Shifted samples from corners

hd1

t-hd2,3,4,5,6 t-hd1 t-hd1 t-hd1,3,4,5,6 t-hd2 t-hd2 t-hd1,2,4,5,6 t-hd3 t-hd3 t-hd1,2,3,5,6 t-hd4 t-hd4 t-hd1,2,3,4,6 t-hd5 t-hd5 t-hd1,2,3,4,5 t-hd6 t-hd6

0 [!10,10] [!20,20] 0 [!10,10] [!14,14] 0 [!10,10] [!20,20] 0 [!10,10] [!12,12] 0 [!10,10] [!20,20] 0 [!10,10] [!20,20]

hd2

hd3

hd4

hd5

hd6

dissim 0.2446}0.7141 )0.0090 )0.0157 0.4026}0.5657 )0.0171 )0.0197 0.2892}0.6477 )0.0128 )0.0197 0.4071}0.7141 )0.0124 )0.0149 0.2412}0.5748 )0.0125 )0.0154 0.4026}0.4947 )0.0112 )0.0198

Mean (dissim ) 0.4600 * * 0.4492 * * 0.4243 * * 0.5677 * * 0.4018 * * 0.4532 * *

dissim 0.2924}0.5217 )0.0886 )0.1945 0.3138}0.9084 )0.0999 )0.1532 0.3059}0.5770 )0.0933 )0.1821 0.3881}1.1559 )0.0864 )0.1022 0.3113}0.5003 )0.1557 )0.2100 0.3487}1.0332 )0.0780 )0.1441

Mean (dissim ) 0.4004 * * 0.5562 * * 0.4230 * * 0.7757 * * 0.3991 * * 0.7075 * *

Table 4 Coarse-matching of non-a$ne-similar airplane curves Object

dissim (min, max)

Mean (dissim )

dissim (min, max)

Mean (dissim )

Plane1 Plane2 Plane3 Plane4 Plane5 Plane6 Plane7 Plane8 Plane9

(0.307, (0.347, (0.388, (0.284, (0.307, (0.317, (0.298, (0.337, (0.284,

0.400 0.464 0.459 0.390 0.442 0.395 0.384 0.421 0.414

(0.289, (0.319, (0.309, (0.132, (0.232, (0.352, (0.321, (0.495, (0.155,

0.437 0.454 0.404 0.386 0.414 0.418 0.535 0.788 0.378

0.519) 0.524) 0.544) 0.451) 0.544) 0.461) 0.450) 0.508) 0.501)

0.567) 0.636) 0.489) 0.556) 0.606) 0.505) 0.729) 1.068) 0.464)

Table 5 Coarse-matching of non-a$ne-similar animal curves. a1}a8: cat, elephant, horse, lizard, mouse, rabbit, squirrel, bear Object

dissim (min, max)

Mean (dissim )

dissim (min, max)

Mean (dissim )

a1 a2 a3 a4 a5 a6 a7 a8

(0.456, (0.452, (0.425, (0.482, (0.459, (0.425, (0.467, (0.452,

0.495 0.509 0.492 0.523 0.528 0.488 0.506 0.495

(0.233, (0.259, (0.252, (0.280, (0.237, (0.233, (0.234, (0.271,

0.367 0.458 0.385 0.333 0.358 0.344 0.366 0.448

0.537) 0.588) 0.564) 0.588) 0.564) 0.513) 0.562) 0.540)

0.751) 0.971) 0.733) 0.379) 0.752) 0.679) 0.663) 0.888)

1420


dissimilar values of non-a$ne object curves, ¹h1 and ¹h2 were chosen to be proportional to the mean dissimilar value. Curve `tool1a in Table 6 is a special case since the number of initial signi"cant corners is obviously di!erent from the remaining tools. `tool1a was therefore considered by the coarse-matching algorithm as nona$ne-similar and removed from the candidate list before the dissimilar measures were applied. The parameters used in these tests are summarized in Table 11.

Table 6 Coarse-matching of non-a$ne-similar tool curves Object Tool1 Tool2 Tool3 Tool4

dissim Mean (dissim ) * *0.4556 *0.5091 *0.6992

* 0.6516 0.6202 0.7618

dissim Mean (dissim ) * *0.4917 *0.3374 *0.6793

* 0.5745 0.7869 1.4395

Table 7 Fine-matching of hand curves Object

hd1 hd2 hd3 hd4 hd5 hd6

A$ne-transformed

Non-a$ne-similar

No. samples in discrete curve

No. B-spline knots

"*h3"

"*b" (10\)

"*b" (10\)

MMSE

1223 675 894 656 950 783

34 34 34 33 33 34

0.1316 0.0096 0.0198 0.0184 0.0078 0.0099

0.93 3.65 0.64 2.03 7.88 7.00

5.640 0.141 0.358 0.250 0.364 0.248

3.020 0.025 0.088 0.090 0.545 0.207

MMSEs

*1910.6 * 678.3 * 624.9 * 510.5 *1226.4 * 580.1

Table 8 Fine-matching of airplane curve Object

Plane1 Plane2 Plane3 Plane4 Plane5 Plane6 Plane7 Plane8 Plane9

A$ne-transformed

Non-a$ne-similar


No. B-spline knots

"*h3"

"*b" (10\)

"*b" (10\)

MMSE

663 672 735 757 725 706 539 552 656

35 35 35 36 35 41 35 34 35

0.0448 0.0641 0.0082 0.0244 0.0370 0.0271 0.0205 0.0276 0.1128

0.096 1.198 0.166 0.967 0.447 1.151 0.560 0.339 0.625

0.170 0.016 0.154 0.193 0.266 0.105 0.100 0.062 0.090

0.1818 0.3257 0.1202 1.1389 0.2372 0.5699 0.3083 0.2003 0.9178

MMSEs

*196.5 *215.3 *171.0 *184.4 *341.5 *300.9 *149.8 *184.2 *163.5

Table 9 Fine-matching of animal curves Object

a1 a2 a3 a4 a5 a6 a7 a8

A$ne-transformed

Non-a$ne-similar


No. B-spline knots

"*h3"

"*b" (10\)

"*b" (10\)

MMSE

924 799 881 868 916 887 769 539

35 38 35 35 35 34 34 29

0.0130 0.0023 0.0710 0.0200 0.0163 0.0598 0.0410 0.0269

0.580 0.234 0.026 0.170 0.517 1.145 0.711 0.028

0.170 0.016 0.154 0.193 0.266 0.105 0.100 0.062

0.4122 0.2835 1.0300 0.2633 0.2081 0.5984 0.7484 0.1673

MMSEs

*1615.9 * 807.2 *1446.7 *1440.1 *1718.6 *1056.1 * 990.9 * 480.7


1421

Table 10 Fine-matching of tool curves Object

Tool1 Tool2 Tool3 Tool4

A$ne-transformed

Non-a$ne-similar


No. B-spline knots

MMSE

"*h3" (10\)

"*b" (10\)

"*b" (10\)

431 201 777 576

21 32 31 32

0.2717 0.0901 0.0545 0.0001

1.804 8.633 26.56 2.563

16.71 8.61 4.21 0.38

1.6991 1.607 0.738 1.293

Table 11 The parameters used in coarse-to-"ne curve matching tests Parameter

In Eq.

Value(s) used in tests

w w ¸ Threshold ¹h1 ¹h2 k and k

(2) (3) (3) (3) In Section 2.3 In Section 2.3 (12)

[1,5] [10,15] 3rd column of Table 2 [1453, 1703] 0.5 mean(dissim ) 0.5 mean(dissim ) 0.1

The "nal results obtained from the coarse-to-"nematching were, as expected, the same as those obtained by just applying the "ne-matching algorithm. However, most candidates were removed from the coarse-matching step. The approximate program execution time for the coarse-to-"ne comparison of two curves on a DECAXP-3000 workstation was 3 s.

MMSEs

* 149.6 * 159.9 *2577.4 * 195.3

corners are extracted for each object using the smoothed discrete object boundary curve. By applying the proposed dissimilar measures over the object features derived from the signi"cant corners, our test results showed small dissimilar values among a$ne similar objects and large dissimilar values among non-a$ne similar objects. Consequently, the coarse-matching step is e!ective in removing most a$ne-dissimilar objects with low computation. By exploiting signi"cant corners for interpolating curve points and by using the proposed "ne-matching algorithm, we simultaneously achieved small MMSE values for matching a$ne objects and obtained the estimated transformation parameters with good accuracy. Further, we obtained large MMSE di!erences between matching a$ne object curves and matching non-a$ne object curves. Comparing to the results shown in [15], our "ne-matching algorithm has shown a greater capacity for discriminating objects. Finally, by combining the coarse and the "ne-matching steps the algorithm is robust in terms of matching accuracy and computational e$ciency.

4.4. Limitations References Since the "ne-matching step uses B-spline approximated curves, the algorithm requires that objects be rigid, or only with small deformations. The algorithm is also sensitive to the quality of the object boundary curves. Inaccurately estimated boundary curves are equivalent to curves from deformed objects. Consequently, a poorly estimated curve of an a$ne object might be recognized as an a$ne-dissimilar object. Finally, the a$ne transformations considered are limited to only rotations, scalings and translations.

5. Conclusions A coarse-to-"ne curve-matching algorithm is proposed and tested for matching planar objects. The signi"cant

[1] G. Sullivan, Visual interpretation of known objects in constrained scenes, Philos. Trans. Roy. Soc. Lond. B. 337 (1992) 361}370. [2] X. Wu, B. Bhanu, Gabor wavelet representation for 3-D object recognition, IEEE Trans. Image Process. 6 (1) (1997) 47}64. [3] R.C. Nelson, J. Aloimonos, Obstacle avoidance using #ow "eld divergence, IEEE Trans. Pattern Anal. Mach. Intell. 11 (10) (1989) 1102}1106. [4] G.E. Mailloux, F. Langlois, P.L. Simard, M. Bertrand, Restoration of the velocity "eld of the heart from twodimensional echocardiograms, IEEE Trans. Med. Imaging 8 (2) (1989) 143}153. [5] A. Gee, R. Cipolla, Fast visual tracking by temporal consensus, Image Vision Comput. 14 (2) (1994) 105}114. [6] V.I. Pavlovic, R. Sharma, T.S. Huang, Visual interpretation of hand gestures for human}computer interpretation:

1422

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

[15]

[16]

[17]

Y.-H. Gu, T. Tjahjadi / Pattern Recognition 33 (2000) 1411}1422 a review, IEEE Trans. Pattern Anal. Mach. Intell. 19 (1997) 667}695. B.S. Manjunath, W.A. Ma, Texture features for browsing and retrieval of image data, IEEE Trans. Pattern Anal. Mach. Intell. 18 (1997) 837}842. H.G. Musmann, M. Hotter, J. Ostermann, Object-oriented analysis-synthesis coding of moving images, Signal Process. Image Commun. 1 (2) (1989) 117}138. R. Chellappa, B. Girod, D.C. Munson Jr., M. Tekalp, M. Vetterli, The past, present and future of image and multidimensional signal processing, IEEE SP Mag. 15 (2) (1998) 21}58. L. Torres, M. Kunt, Video Coding: The 2nd Generation Approach, Kluwer Academic Publisher, Dordrecht, 1996. J. Weng, N. Ahuja, T.S. Huang, Motion and structure from point correspondences with error estimation: Planar surfaces, IEEE Trans. Signal Process. 39 (12) (1991) 2691}2717. B. Kamgar-Parsi, B. Kamgar-Parsi, Matching sets of 3D line segments with application to polygonal arc matching, IEEE Trans. Pattern Anal. Mach. Intell. 19 (1997) 1090}1099. B. Kamgar-Pasi, A. Magalit, A. Rosenfeld, Matching general polygonal arcs, CVGIP: Image Understanding 53 (1991) 227}234. E. Persoon, K.S. Fu, Shape discrimination using Fourier descriptors, IEEE Trans. Systems Man Cybernet. 7 (1977) 170}179. F.S. Cohen, Z. Huang, Z. Yang, Invariant matching and identi"cation of curves using B-spline curve representation, IEEE Trans. Image Process. 4 (1995) 1}10. F.S. Cohen, J. Wang, Modeling image curves using invariant 3D object curve models*a path to 3D recognition and shape estimation from image contours, IEEE Trans. Pattern Anal. Mach. Intell. 16 (1994) 1}23. M.J. Paulik, M. Das, N.K. Loh, Nonstationary autoregressive modeling of object contours, IEEE Trans. Signal Process. 40 (1992) 660}675.

[18] J. Flusser, T. Suk, Pattern recognition by a$ne moment invariants, Pattern Recognition 26 (1993) 167}174. [19] P.J. Besl, R.C. Jain, 3D object recognition, Comput. Surv. 17 (1) (1985) 75}145. [20] A.L.N. Fred, J.S. Marques, P.M. Jorge, Hidden Markov models vs syntactic modeling in object recognition, Proceedings of International Conference on Image Processing (ICIP '97), 1997, pp. 893}896. [21] Q.M. Tueng, W.W. Boles, Wavelet-based a$ne invariant representation: a tool for recognizing planar objects in 3D space, IEEE Trans. Pattern Anal. Mach. Intell. 19 (1997) 846}857. [22] C. de Boor, in: A practical guide to splines, Applied Mathematical Sciences, vol. 27, Springer, Berlin, 1978. [23] K. Arbter, W.E. Snyder, H. Burkhardr, G. Hirzinger, Application of a$ne-invariant Fourier descriptors to recognition of 3D objects, IEEE Trans. Pattern Anal. Mach. Intell. 12 (1990) 452}459. [24] D. Cyganski, R.F. Vaz, A linear signal decomposition approach to a$ne invariant contour identi"cation, Proc. SPIE: Intell. Robots Comput. Vision X 1607 (1991) 98}109. [25] D.F. Rogers, J.A. Adames, Mathematical Elements for Computer Graphics, McGraw-Hill Publishing Company, New York, 1990. [26] F.S. Cohen, J. Wang, Modeling image curve using invariant 3-D object curve models*a path to 3-D recognition and shape estimation from image contours, IEEE Trans. Pattern Anal. Mach. Intell. 16 (1994) 1}12. [27] Y.H. Gu, Adaptive multiresolution Hermite-Binomial "lters for image edge and texture analysis, Proc. SPIE 2308 (1994) 748}759. [28] K.H. Liang, T. Tjahjadi, Y.H. Yang, Bounded di!usion for multiscale edge detection using regularized cubic B-spline "tting, IEEE Trans. Systems Man Cybernet. 29 (2) (1999) 291}297. [29] B.D. Chen, P. Siy, Forward/backward contour tracing with feedback, IEEE Trans. Pattern Anal. Mach. Intell. 9 (1987) 438}446.

About the Author*(IRENE) YU-HUA GU received M.Sc. degree from East China Normal University, in 1984, Ph.D. degree in electrical engineering from Eindhoven University of Technology, The Netherlands, in 1992. She was a research fellow at Philips Research Institute IPO (NL) and Sta!ordshire University (UK), and a lecturer at The University of Birmingham (UK) during 1992}1996. Since September 1996 she has been an assistant professor in the Department of Signals and Systems at Chalmers University of Technology, Sweden. Her current research interests include multispectral image processing, object recognition, time}frequency and time-scale domain signal analysis. Yu-Hua Gu is a member of the IEEE Signal Processing Society. About the Author*TARDI TJAHJADI received the B.Sc. (Hons.) degree in mechanical engineering from University College London, UK, in 1980, the M.Sc. degree in management sciences (operational management) and the Ph.D. in total technology from the University of Manchester Institute of Science and Technology, UK, in 1981 and 1984, respectively. He joined the Department of Engineering at the University of Warwick and the UK Daresbury Synchrotron Radiation Source Laboratory as a Joint Teaching Fellow in 1984. Since 1986 he has been a lecturer in computer systems engineering at the same university. His research interests include multiresolution image processing, model-based colour image processing and fuzzy expert systems.


Computing the shape of a planar points set Mahmoud Melkemi*, Mourad Djebali L.I.G.I.M. Universite& Claude Bernard Lyon1. 43 boulevard du 11 Novembre 1918, Bat. 710, 69622 Villeurbanne, France Received 7 January 1999; accepted 23 April 1999

Abstract In this article, we introduce a mathematical formalism de"ning the shape of a "nite point set which we call A-shape. The parameter A is a "nite set of points which positions variation allows A-shape to generate a family of graphs extracted from Delaunay triangulation. Each graph corresponds to an element of a shapes set presenting more and more details and going from the convex hull of the points set to the points set itself. It is obvious that the shape having the suitable level of details is obtained by a judicious choice of A. We also propose a method to determine A for which A-shape gives the adequate shape for points sets containing dense and sparse regions. 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Pattern recognition; Computational geometry; Shape hull; Voronoi diagram

1. Introduction We understand by the shape of a "nite points set, the description of this set by connected components, where each component is a polygon which vertices are selected among the initial points of this set. This polygonal representation must re#ect the forms perceived by a human observer of the dot patterns set. In the literature, several works treat the problem of shape computation. According to the followed approach, we distinguish two complementary categories: the "rst category is a set of algorithms using Delaunay triangulation and based on heuristics. In certain complex situations, these algorithms give satisfactory results. However, on the one hand, they are related to a signi"cant number of thresholds to be adjusted. On the other hand, the interpretation of the obtained results is often related to visual criteria. In this class, Jarvis [1] proposes an algorithm based on the idea of regarding the shape as a generalization of the convex hull. Ahuja and Tuceryan [2] describe a method of classi"cation based on the geometrical properties of a Voronoi region. Other algo-

* Corresponding author. Tel.: #4-72432630; fax: #472431312. E-mail address: [email protected] (M. Melkemi).

rithms [3,4] use the minimum spanning tree or the relative neighborhood graph to de"ne "rst a neighborhood relationships between the points of the set. These proximity informations are used in a classi"cation process. The second category gathers the various concepts, which reveal morphological properties of a point set. Among these concepts the minimum spanning tree [5], the Gabriel graph [6], the relative neighborhood graph [7,8] and the b-skeletons [9], which are subgraphs of the Delaunay triangulation and exhibit the organization of the points according to neighborhood relationships. In this class, we are interested by a concept related to the shape computation according to a mathematical formalism. Indeed, Edelsbrunner et al. [10] describe the shape by a family of graphs, named a-shape, and obtained from Delaunay triangulation, where the parameter a is a real number, which controls the level of details revealed by each graph. Conceptually, this formalism regards the notion of a shape as a generalization of the convex hull. In this context, the convex hull is a description of the shape with an insu$cient level of details. By varying the a parameter from #R to 0, a-shape reveals gradually more details. On the basis of this construction, one can conclude that for a "nite points set, there is always a suitable a for which a-shape re#ects the shape with the wished level of details. This conclusion "nds however counter-examples. Indeed, there are points


1424

M. Melkemi, M. Djebali / Pattern Recognition 33 (2000) 1423}1436

sets for which, the a-shape family does not contain elements representing suitable shapes for these sets. Concretely, Fig. 1 shows a points set for which there is no a-shape which detects the hole H while modeling correctly the sparse region. This region is built so that each disc, which does not contain any point of the set and which touches two points, has a diameter higher than r, r being the diameter of the largest disc contained in H. With the aim of curing the limits of a-shape, we propose to describe the shape by a family of graphs extracted from Delaunay triangulation. Each graph is called Ashape where A is a "nite points set controlling the level of details re#ected by this graph [11]. A-shape "nds its base through transcribing the concept of connected components of the continuous case to the discrete one. The analogue of A-shape in the continuous case, named A-shape, corresponds to observations of a domain D on various scales. The level of details re#ected at each observation is controlled by parameter A. Concretely, A-shape is the border of a domain obtained by a dilation of D. An example of A-shape is illustrated by Fig. 1(b) where A-shape corresponds to a suitable level of details of the dense region and the sparse one. The implementation of A-shape requires an adequate choice of the parameter A. In the simple case of a uniformly distributed points set, we de"ne the parameter A relative to a threshold t*0. The points of A are the vertices of the Voronoi diagram of the set and are centers of the Delaunay circles having radii higher than t. The A points, thus de"ned, allow A-shape to detect the holes and the cavities with more and more details, as t decreases. In a more general case, where the points sets contain dense and sparse zones, we adapt the construction of the simple case by associating with each point of the set,

a disc centered at this point. Each disc is an in#uence zone of each point, the zone is large, if the point is in a sparse region and weak if not. By taking into account the points weights, the points of A are computed in a similar way to that of the simple case. This article is organized as follows: in Section 2, we present the Voronoi diagram and a-shape which are related to A-shape concept. In the following section, we describe A-shape and its principal properties. The last section relates to the implementation of A-shape, it is a matter of computing the suitable A for the case of sets containing sparse and dense regions. 2. Voronoi diagram and a-shape 2.1. Dexnition of the Voronoi diagram Let S"+p , p ,2, p , be a set of n distinct points of the L plan, d(p, q) is the Euclidean distance between two points p and q. We de"ne a Voronoi polygon of a point p by: G R(S, p )"+p31; d(p, p )(d(p, p ) ∀j such that iOj,. G G H The set ). RM (X5>)LRM (X)5RM (>). XL>NRM (X)LRM (>). RM (U)"U. XLRM (X). RM (R(X))LX. RM (RM (X))LRM (X). RM (R(X))LR(X). R(X!6>)LR(X)!6R(>). R(X)LRM (X).

Properties L1 and U1 express the duality between lower and upper approximations. These properties allow to derive relations U1}U10 from relations L1}L10. Properties L2, L6, U2 and U6 express limit conditions for the empty set and the whole space. Compatibility with union and intersection is expressed by L3, L4, U3 and U4. Properties L5 and U5 express the increasingness with respect to set inclusion. The basic notions of lower and upper approximations can be found in properties L7, U7 and LU. Properties L8}L10 and U8}U10 concern the composition of approximations. Note that if L7 and L9 are simultaneously satis"ed, we have, due to L5 R(X)"R(R(X)), i.e. lower approximation is idempotent. In the same way, if U7 and U9 are simultaneously satis"ed, then upper approximation is idempotent. If L7 and L10 are simultaneously satis"ed, then we have R(X)"R(RM (X)) and a similar expression for the upper approximation.

3. Comparison of basic properties 3.1. A list of properties of interest

3.2. Which properties do the rough sets and mathematical morphology have in common?

In this section, we list the properties that are of interest in the theory of rough sets. We follow here the presentation provided in Ref. [8]. The satisfaction of these properties depending on the chosen de"nition is detailed in the next subsection.

In Table 1 we compare the properties that are satis"ed by the di!erent de"nitions of rough sets with those satis"ed by the four basic morphological operators.

R(X)"[RM (X!)]!, where X! denotes the complementation of X in U. R(U)";. R(X5>)"R(X)5R(>). R(X6>)MR(X)6R(>). XL>NR(X)LR(>). R()". R(X)LX. XLR(RM (X)).

From the results in Table 1, it appears clearly that lower approximations share many properties with erosion and with opening, while upper approximations share many properties with dilation and closing. These algebraic properties make rough set algebra similar to mathematical morphology algebra. Having made these observations, we can now establish formal links between set approximations and morphological operators.

L1. L2. L3. L4. L5. L6. L7. L8.

3.3. Second conclusion

1490

I. Bloch / Pattern Recognition 33 (2000) 1487}1496

Table 1 Comparison between the properties of rough sets depending on the properties of R with those of mathematical morphology operators. A cross (;) indicates that the property is satis"ed. The "rst column contains the list of properties, according to the notations given in Section 3.1. The next four columns are for rough sets, de"ned from any relation, a tolerance relation, a relation that is re#exive and transitive, and an equivalence relation, respectively. The two last columns are for morphological operators, erosion and opening in the upper part (corresponding to the properties of lower approximation) and dilation and closing in the lower part (corresponding to the properties of upper approximation) Property

Any R

Tolerance rel.

R re#ex. and trans.

Equivalence rel.

Erosion/Dilation

Opening/Closing

L1 L2 L3 L4 L5 L6 L7 L8 L9 L10

; ; ; ; ;

; ; ; ; ; ; ; ;

; ; ; ; ; ; ;

; ; ; ; ; ; ; ; ; ;

; ; ; ; ; ; ; ;

; ; only L ; ; ; ;

U1 U2 U3 U4 U5 U6 U7 U8 U9 U10

; ; ; ; ;

; ; ; ; ; ; ; ; ; ;

; ; ; ; ; ; ; ;

; ; only L ; ; ; ;

K LU

;

; ;

; ;

; ;

; ; ; ; ; ; ; ;

; ;

; ; ; ; ; ; ; ; ; ; ;

;

;

4. Formal links between rough sets and mathematical morphology

0 y!x3B,

(23)

0 x!y3B[ ("B),

(24)

Lower and upper approximations can be obtained from erosion and dilation. For a given structuring element, the corresponding relation is then as follows:

0 x3B , W 0 yRx

(25)

xRy 0 y3B . V From R, we derive r(x) as

(18)

∀x3U, r(x)"+y3U"y3B ,"B . (19) V V We always assume that the origin of U belongs to the structuring element B. It follows that: ∀x3U, x3B V and therefore,

(20)

∀x3U, xRx,

(21)

i.e. R is re#exive. Moreover, if B is symmetrical (i.e. B"B[ ), we have ∀(x, y)3U, xRy 0 y3B , V

(22)

(26)

which proves that R is symmetrical. If follows that R is a tolerance relation. Let us show that for this relation, erosion and lower approximation coincide: ∀XLU, R(X)"+x3U"r(x)LX,, "+x3U"B LX,, V "E (X).

(27) (28) (29)

In a similar way, dilation and upper approximation coincide, since we have ∀XLU, RM (X)"+x3U"r(x)5XO, "+x3U"B 5XO, V "D (X).

(30) (31) (32)


These results are com"rmed by the properties shown in Table 1, that are the same for lower and upper approximations derived from a tolerance relation, and for erosion and dilation. These equivalences are in accordance with the operator-oriented view of rough sets [10,8]. Let ¸ and H be two dual operators, such that ∀XLU, ¸(X)"H(X!)!

(33)

and satisfy the following properties: (1) H()", (2) H commutes with union: H(X6>)"H(X)6H(>).

∀XLU,

∀>LU,

Then there exists a relation R such that ¸(X)"R(X) and H(X)"RM (X). This relation is de"ned by xRy 0 x3H(+y,).

(34)

The results proved in this section provide concrete examples of operators ¸ and H, which are morphological erosions and dilations. Actually, a family of operators is obtained, indexed by the structuring element. The derived relation R is exactly the one introduced in Eq. (18), since we have for a symmetrical structuring element B: xRy 0 x3H(+y,) 0 x3D (+y,) 0 x3B 0 y3B . (35) W V Let us now consider opening and closing. Using the operator-oriented point of view, they can be used, respectively, as lower and upper approximations, since they have most of the required properties as shown in Table 1. However, since closing does not commute in general with union (only a inclusion holds for property U3), the direct derivation of R as in Eq. (34) cannot be applied. Even if it is not as obvious as for dilation and erosion to "nd an expression for opening and closing based on a relation, these operators have interesting properties that deserve to consider them as good operators for constructing rough sets. In particular, they are idempotent, which is particularly useful if we take the topology-oriented point of view. This is the scope of the next section.

1491

Let us consider again two dual operators ¸ and H, but satisfying some more axioms: (1) H()", (2) H commutes with union: ∀XLU, ∀>LU, H(X6>)"H(X)6H(>). (3) H is extensive: ∀XLU, XLH(X). (4) H is idempotent: ∀XLU, H(H(X)"H(X). (5) ∀XLU, XL¸(H(X)). If properties (1)}(4) are satis"ed, then the relation R that is derived from H using Eq. (34) is re#exive and transitive, and this de"nes a topological approximation space. Indeed, properties (1)}(4) are the properties of a common closure operator. Except for property (2) where we generally have only an inclusion, properties (1)}(4) are also satis"ed by closing (and the dual operator opening). Therefore, these morphological operators de"ne a topological approximation space. If properties (1)}(5) are satis"ed, then R is an equivalence relation. Property (5) is in general not satis"ed by opening and closing. If the set of considered objects is restricted to the objects that are opened by B (i.e. they do not contain details smaller than B), then property (5) holds. However, in such a case, the lower approximation does not modify the set. Let us now consider erosion and dilation. They do not satisfy property (4), but satisfy all others. The loss of idempotence for the closure operator corresponds to a pre-topology [11]. Therefore, using erosion and dilation introduces the notion of pretopological approximation space. This may be of interest for pattern recognition purposes, since non-idempotent closure allows to aggregate patterns using iterated closure operations. The basic topology on sets in mathematical morphology is the hit-or-miss topology, that is based on the intersection of a closed set with some open sets (the `hita part) and on the non-intersection of a closed set with some compact sets (the `missa part) [5]. It appears that the relations that de"ne this topology are the same as the ones de"ning lower and upper approximations. This leads as given in Ref. [7], to a construction of hit-or-miss topology on rough sets.

5. Topological aspects Important notions in topology are interior and closure operators. More local information is the notion of neighborhood. These two aspects will be dealt within the two parts of this section. 5.1. Topology and pre-topology The idea is that lower and upper approximations can be interpreted as interior and closure. Morphological operators also receive similar interpretations.

5.2. Neighborhood systems Let us now consider a topology de"ned through a neighborhood system. Let n(x) be a neighborhood of x and N(x) be a neighborhood system for x. Lower and upper approximations are then de"ned as [12]: N(X)"+x3U"n(x)3N(X)"n(x)LX,,

(36)

NM (X)"+x3U"∀n(x)3N(X)"n(x)5XO,.

(37)

1492


The de"nitions presented in Section 2 correspond to the case where only one neighborhood is considered, i.e. N(x)"+n(x),. The analogy with mathematical morphology is straightforward, if we consider that the structuring element translated at a point x of U is nothing but a neighborhood of x. If we set N(x)"+B ,, we obtain V N(X)"E (X), (38) NM (X)"D (X).

(39)

Moreover, if we consider a family of structuring elements B, 2, BI, and if we set N(x)"+B,2, BI ,, the BG being V V V considered as di!erent neighborhoods of x whose union builds the neighborhood system, we obtain N(X)" 8 E G (X), G

I

(40)

NM (X)" 8 D G (X). (41) G

I Let us now consider opening and closing. Similar relations are obtained, by setting this time N(x)"+B "y3U W and x3B ,. Then we obtain W N(X)"O (X), (42) NM (X)"C (X).

(43)

The proof of these results comes from the writing of an opening as O (X)"+x3U"y3U"x3B and B LX,. (44) W W As for erosion and dilation, we can consider a family of structuring elements for opening and closing. This view is particularly interesting for shape recognition, since in morphological recognition, an object has often to be tested or matched with a set of patterns, like directional structuring elements. This set of patterns is interpreted as a neighborhood system.

6. Extensions In this section, we give some hints on possible extensions of the results we obtained in this paper. These extensions concern the choice of the dual operators and the objects on which they are applied. 6.1. Thinning and thickening Among the dual operators used in mathematical morphology, thinning and thickening are of particular interest, since they allow to perform operations depending on various local con"gurations. The main di!erence with erosion, dilation, opening and closing, is that the structuring element is not only tested against object points

(B LX, B 5XO, etc.), but it is also tested against V V background points (i.e. points of X!). Let us "rst recall the de"nitions of these operations (the reader may refer to Refs. [5,6] for more details). The structuring element is divided into two disjoint parts (¹ , ¹ ), where ¹ is tested against points of X, while ¹ is tested against points of X!. The hit-or-miss transformation is de"ned as HM¹ (X)"E (X)5E (X!). (45) 2 2 2 2 From this operation, thinning and thickening are de"ned as (X)"X!HM¹ (X), (46) 2 2 2 2 ¹hick (X)"X6HM¹ (X). (47) 2 2 2 2 Since ¹ 5¹ ", the origin of the space belongs either to ¹ or to ¹ . In the "rst case, the hit-or-miss trans formation provides a subset of X and it is meaningful to perform thinning. In the second case, the hit-or-miss transformation provides a subset of X! and it is meaningful to perform thickening. The duality that holds between thinning and thickening takes the following form: ¹hin

(X)"[¹hick (X!)]!. (48) 2 2 2 2 Therefore, the possible pairs that can be de"ned from these operators as lower and upper approximations can be of the following types:

¹hin

(1) (¹hin (X), ¹hick (X)) if the origin of U 2 2 2 2 belongs to ¹ , (2) (¹hin (X), ¹hick (X)) if the origin of U 2 2 2 2 belongs to ¹ , (3) (¹hin (X), X) if the origin of U belongs to¹ , 2 2 (4) (X, ¹hick (X)) if the origin of U belongs to ¹ . 2 2 In the following, we restrict our study to the "rst case, where the origin of U belongs to ¹ . The second case is similar. Taking the operator-oriented point of view, the rough sets that can be built from thinning and thickening according to the "rst pair are obtained for the following operators: , H"¹hick . (49) 2 2 2 2 Since thickening generally does not commute with union (except for particular structuring elements where it is equivalent to dilation), it is not possible to derive directly a relation according to which the rough sets are de"ned. Among the properties listed in Table 1, L1, L2, L6, L7, U1, U2, U6, U7 and LU are always satis"ed. This shows that several properties are lost, and therefore we call `generalized rough setsa the pairs obtained from thinning and thickening. The ones obtained using erosion and dilation by a structuring element B are particular cases, corresponding to ¹ de"ned as the family of structuring ¸"¹hin


elements containing the origin and at least another point of B, and ¹ "B!¹ . Let us consider now the third and fourth possible pairs of approximations. They bring an original aspect which is a kind of assymetry between lower and upper approximations. Taking the operator-oriented point of view, we have for the third pair: ¸(X)"¹hin (X), H(X)"X, 2 2

(50)

¸(X)"X, H(X)"¹hick (X). 2 2

(51)

Here the duality is not directly between ¸ and H, or between ¸ and H, but between ¸ and H and between ¸ and H, since we have ¸(X!)"¹hin

2 2

(X!)"[¹hick (X)]! 2 2

"[H(X)]!,

(52)

¸(X!)"X!"[H(X)]!.

(53)

The topological interpretation is particularly interesting when using thinning and thickening. Indeed, the pair (¹ , ¹ ) de"nes a neighborhood around a point, describ ing which points of the neighborhood should belong to the set X and which ones should belong to its complement. For instance, taking just the origin for ¹ and B!¹ for ¹ (B being any structuring element, or neighborhood), the hit-or-miss transformation using (¹ , ¹ ) selects the points that are isolated in the back ground. The thinning by (¹ , ¹ ) removes such points, leading to a lower approximation of X that has no isolated points, while the thickening by (¹ , ¹ ) "lls up isolated points of the background, leading to an upper approximation that has no holes constituted by only one point. This shows that very "ne operations can be obtained using these operators. Another interesting point is that these operations can be iterated by using families of structuring elements [5] (for instance, rotations of a generic structuring element). In this way, we can use for instance the skeleton as the lower approximation, the convex hull as the upper approximation, etc., which are useful tools in shape representation and recognition. Moreover, several thinnings and thickenings are homotopic operators, i.e. that deform shapes while preserving their homotopy. This leads to homotopic rough sets, that deserve probably a deeper study. 6.2. Algebraic rough sets using algebraic operations Another possible extension may be derived from algebraic operators. Algebraic erosions and dilations are de"ned on complete lattices as operators that commute with intersection and union, respectively [5,6]. Therefore L3 and U3 are directly satis"ed. Properties L2, L5, L6, U2, U5 and U6 are also satis"ed by these operators. Note

1493

that morphological erosions and dilations are particular cases of algebraic operators, if translation invariance holds. Algebraic openings and closings are de"ned as increasing, anti-extensive (respectively extensive) and idempotent operators [5,6]. Therefore properties L5, L7, L9 and U5, U7, U9 are automatically satis"ed. The use of algebraic erosion/dilation or opening/closing for de"ning lower and upper approximations lead to what we call `algebraic rough setsa. 6.3. Rough functions Since mathematical morphology also applies to functions [5,6], we can use the de"nitions of dilation, erosion, opening and closing on functions to de"ne lower and upper approximations of functions. This seems to be a natural extension of rough sets. Let f be a function de"ned on U, and let B be a set of U (structuring element), that we consider here symmetrical, containing the origin of U. Using erosion and dilation, we de"ne lower and upper approximations of f as ∀x3U, B( f )(x)"E ( f )(x)" inf f (y), WZ V

(54)

∀x3U, BM ( f )(x)"D ( f )(x)"sup f (y). WZ V

(55)

Using opening and closing, we de"ne lower and upper approximations of f as ∀x3U, B( f )(x)"O ( f )(x)"D [E ( f )](x),

(56)

∀x3U, BM ( f )(x)"C ( f )(x)"E [D ( f )](x).

(57)

The properties of these rough functions are direct transpositions of the ones of rough sets: L1. L2. L3. L4. L5. L6. L7. L8. L9. L10. U1. U2. U3. U4. U5. U6. U7. U8.

B( f )"![BM (!f )]. B( f )"f , where f is any constant function. A A A B(min( f, g))"min[B( f ),B(g)]. B(max( f, g))*max[B( f ), B(g)]. f)gNB( f ))B(g). B(f )"f , where f is identically zero. B( f ))f. f)B(BM ( f )). B( f ))B(B( f )). BM ( f ))B(BM ( f )). BM ( f )"![B(!f )]. BM ( f )"f . BM (max( f, g))"max[BM ( f ),BM (g)]. BM (min( f, g)))min[BM ( f ),BM (g)]. f)gNBM ( f ))BM (g). BM ( f )"f . A A f)BM ( f ). BM (B( f )))f.

1494

U9. U10. K. LU.


BM (BM ( f )))BM ( f ). BM (B( f )))B( f ). B(max(!f, g)))max[!B( f ), B(g)]. B( f ))BM ( f ).

Using erosion and dilation, properties L1}L8, U1}U8, K and LU hold, as for the case of sets. Using opening and closing, we usually have an inequality only for L3 and U3, properties L8, L10, U8 and U10 being generally not satis"ed, but L9 and U9 are always satis"ed. This construction can be further extended using a function g as structuring element: ∀x3U, g( f )(x)"E ( f )(x)"inf+f (y)!g(y!x), y3U,, E (58) ∀x3U, g ( f )(x)"D ( f )(x)"sup+ f (y)#g(y!x), y3U,. E (59) Similar properties are obtained. 6.4. Fuzzy rough sets In Ref. [9], we de"ned erosion and dilation of a fuzzy set k by a fuzzy structuring element l as follows: ∀x3U, E (k)(x)"inf ¹[c(l(y!x)), k(y)], J WZU

(60)

(61) ∀x3U, D (k)(x)"sup t[l(y!x), k(y)], J WZU where t is a t-norm (fuzzy intersection), ¹ a t-conorm (fuzzy union) and c a fuzzy complementation. The reader may refer to Ref. [13] for more details about fuzzy connectives. Fuzzy opening and closing are de"ned for crisp sets as combinations of erosion and dilation. Fuzzy morphological operations have the same properties as that of crisp ones, as shown in Ref. [9]. Most of the properties hold for any t-norm and t-conorm. Only idempotence and extensivity (respectively, anti-extensivity) of closing (respectively opening) are satis"ed for particular t-norms and t-conorms only, as for instance Lukasiewicz operators, de"ned as: t(a, b)"max(0, a#b!1) and ¹(a, b)"min(1, a#b). Therefore, fuzzy rough sets de"ned from these morphological operators have exactly the same properties as crisp rough sets, at least for particular t-norms and t-conorms. It turns out that these de"nitions using fuzzy erosion and dilation are generalizations of the ones proposed in Ref. [14], for t"min and ¹"max in a completely di!erent context, using a fuzzy relation k . The equiva0 lence is obtained as in the crisp case by setting k (x, y)"l(y!x). 0

(62)

The interpretation is similar to that in the crisp case: the degree of relation between x and y is equal to the degree to which y!x belongs to the structuring element, i.e. to the degree to which y belongs to the structuring element translated at x. This extension brings together three di!erent aspects of the information: rough sets represent coarseness, fuzzy sets represent vagueness and mathematical morphology brings a geometrical, topological and morphological aspect. The conjunction of vagueness and coarseness had already been pointed out in Ref. [14]. In this paper, we bring an additional morphological point of view, by de"ning fuzzy rough sets using fuzzy mathematical morphology.

7. Discussion The main results established in this paper are that morphological operators happen to be good tools for de"ning lower and upper approximations of sets, in the theory of rough sets, with the appropriate properties. Moreover, these operators lead to a generalization of rough sets to functions and to fuzzy sets. Several mathematical aspects are common to both theory, as set theoretical, algebraic and topological aspects. From an information point of view, several aspects are merged: coarseness, morphology, and vagueness for the extension to fuzzy sets. In addition to the formal similarities between both domains, contributions can be brought from each domain to the other. Mathematical morphology brings tools for analyzing shapes, the approximations it provides contain a regularization and "ltering aspect (in particular using opening and closing, or their combinations). Some formulations are particular cases of general lower and upper approximations, but others may bring some generalizations (for instance in the framework of pre-topology instead of topology, or using thinning and thickening). The use of fuzzy morphology for de"ning fuzzy rough sets provide formulations that are more general than the ones originally proposed in Ref. [14] using fuzzy relations. Mathematical morphology operators have a lot of properties that can from now on be used also in the theory of rough sets. One example of useful property is iterativity and combination (for instance dilating a set n times by a ball of radius 1 is equivalent to dilating the set once with a ball of radius n). Such properties allow to perform successive approximations, using the same operator or di!erent ones, in a controlled way, leading to di!erent levels of representation, or of precision. In the same way, compatibility with geometric transformations is useful for spatial applications. Also the choice of the structuring element B has a direct impact on the approximations that can be more or less strong depending on the size of B.


Since mathematical morphology is issued from a completely di!erent domain from the theory of rough sets, it also brings a large variety of applications, particularly in image processing. Until now, very few attempts have been done to use rough sets in image processing (see Refs. [15,16]). The results shown in this paper might provide a bridge to "ll this gap. Conversely, since the theory of rough sets has been mainly developed in the domain of arti"cial intelligence, it brings a new look to mathematical morphology, in particular for approximate reasoning and logics. What seems to deserve further study is for instance to build a possibilistic modal logic based on morphological operators. A modal logic based on rough sets is described e.g. in Ref. [8], where the connectives used are in a classical way negation, conjunction, disjunction, implication and equivalence, but also two modal operators, necessity 䊐 and possibility 䉫, that are de"ned from lower and upper approximations. The properties of Table 1 have equivalents in logical terms using these connectives, leading to reasoning rules like 䊐pP䉫p, 䊐pPp and several others. If we use morphological operators for de"ning lower and upper approximations, we have for instance: 䊐X"E (X), 䉫X"D (X). A set X may correspond to a proposition like `this object is Xa, or `this object is in Xa. Then 䊐X and 䉫X represent, respectively, the area where it is necessary that such a proposition holds, and the area where it is just possible. These two areas represent approximations of the location of X for instance, due to imprecision, incomplete knowledge, etc. The logic derived from rough sets provides then some tools for reasoning under imprecision in a morphological context.

8. Summary Rough set theory has been introduced in 1982, as an extension of set theory, mainly in the domain of intelligent systems. The objective was to deal with incomplete information, leading to the idea of indistinguishability of objects in a set. It is therefore related to the concept of approximation. In this framework, a set is approximated by two sets, called upper and lower approximations, that respectively contains and is included in the initial set. Mathematical morphology is originally also based on set theory. It has been introduced in 1964 by Matheron, in order to study porous media. But this theory evolved rapidly to a general theory of shape and its transformations, and was applied particularly in image processing and pattern recognition. Mathematical morphology provides operators that are extensive or anti-extensive, such as dilation and erosion (if the origin of the space belongs to the chosen structuring element), or closing and opening. One of the basic properties of upper and lower set approximations is duality. A similar property holds for mathematical morphology between dilation and erosion,

1495

and between opening and closing. In fact, most of the morphological operators go by pairs of dual operators. Based on these elementary observations, it is tempting to look at closer links between both domains. Here we try to link lower and uper approximations directly to morphological operators. To our knowledge, this is the "rst time that such links are established. We "rst start from the basic de"nitions of rough sets, in particular those based on a similarity relations, and of mathematical morphology, in particular its four basic operators. Then we compare both the theories in the light of a list of properties that are commonly used in rough set theory. Then we establish formal links between upper and lower approximations on the one hand, and dilation and erosion (respectively opening and closing) on the other hand. We then look more closely on some topological aspects (topology and pre-topology de"ned from a closure operator, and neighbourood systems). Then, we propose some extensions of these links, using other operators like thinning and thickening, or algebraic operators. We also extend this work to funtions and on fuzzy sets, and show how mathematical morphology on functions and on fuzzy sets can be used for de"ning rough functions and rough fuzzy sets. This brings together three di!erent aspects of the information: vagueness (through fuzzy sets), coarseness (through rough sets) and shape (through mathematical morphology). Finally we provide some insights on the respective contributions of each domain to the other, those that can be anticipated from this work.

References [1] Z. Pawlak, Rough sets, Int. J. Inform. Comput. Sci. 11 (5) (1982) 341}356. [2] L. Zadeh, Fuzzy sets and information granularity, in: M. Gupta, R. Ragade, R. Yager (Eds.), Advances in Fuzzy Set Theory and Applications, North-Holland, Amsterdam, 1979, pp. 3}18. [3] G. Matheron, EleH ments pour une theH orie des milieux poreux, Masson, Paris, 1967. [4] G. Matheron, Random Sets and Integral Geometry, Wiley, New York, 1975. [5] J. Serra, Image Analysis and Mathematical Morphology, Academic Press, London, 1982. [6] J. Serra, in: J. Serra (Ed.), Image Analysis and Mathematical Morphology, Part II: Theoretical Advances, Academic Press, London, 1988. [7] L. Polkowski, Rough set approach to mathematical morphology: approximate compression of data, in: Information Processing and Management of Uncertainty IPMU'98, Paris, 1998, pp. 1183}1189. [8] Y.Y. Yao, Two views of the theory of rough sets in "nite universes, Int. J. Approximate Reasoning 15 (4) (1996) 291}310. [9] I. Bloch, H. Mam( tre, Fuzzy mathematical morphologies: a comparative study, Pattern Recognition 28 (9) (1995) 1341}1387.

1496


[10] T.Y. Lin, Q. Liu, Rough approximate operators: axiomatic rough sets theory, in: W.P. Ziarko (Ed.), Rough Sets, Fuzzy Sets and Knowledge Discovery, Springer, London, 1994, pp. 256}260. [11] H. Emptoz, Mode`le preH topologique pour la reconnaissance des formes. Applications en neurophysiologie, The`se de Doctorat d'Etat, Univ. Claude Bernard, Lyon I, Lyon, France, 1983. [12] T.Y. Lin, Neighborhood systems: a qualitative theory for fuzzy and rough sets. in: Second Annual Joint Conference on Information Science, Wrightsville Beach, NC, 1995, pp. 255}258.

[13] D. Dubois, H. Prade, A review of fuzzy set aggregation connectives, Inform. Sci. 36 (1985) 85}121. [14] D. Dubois, H. Prade, Rough fuzzy sets and fuzzy rough sets, Int. J. General Systems 17 (1990) 191}209. [15] Z. Wojcik, Rough approximation of shapes in pattern recognition, Comput. Vision Graphics Image Process. 40 (1987) 228}249. [16] Z.M. Wojcik, Application of rough sets for edge enhancing image "lters, in: IEEE International Conference on Image Processing ICIP'94, Vol. II, Austin, Texas, 1994, pp. 525}529.

About the Author*ISABELLE BLOCH is a professor at ENST Paris (Signal and Image Department), and is in charge of the Image Processing and Interpretation Group. She is graduated from Ecole des Mines de Paris in 1986, received a Ph.D. from ENST Paris in 1990, and the `Habilitation a` Diriger des Recherchesa from University Paris 5 in 1995. Her research interests include 3D image and object processing, structural pattern recognition, 3D and fuzzy mathematical morphology, decision theory, data fusion in image processing, fuzzy set theory, evidence theory, medical imaging, aerial and satellite imaging.


Fast nearest-neighbor search algorithms based on approximation-elimination search V. Ramasubramanian , Kuldip K. Paliwal* Computer Systems and Communications Group, Tata Institute of Fundamental Research, Homi Bhabha Road, Bombay 400 005, India School of Microelectronic Engineering, Grizth University, Brisbane, QLD 4111, Australia Received 28 July 1997; received in revised form 5 January 1999; accepted 5 January 1999

Abstract In this paper, we provide an overview of fast nearest-neighbor search algorithms based on an &approximation}elimination' framework under a class of elimination rules, namely, partial distance elimination, hypercube elimination and absolute-error-inequality elimination derived from approximations of Euclidean distance. Previous algorithms based on these elimination rules are reviewed in the context of approximation}elimination search. The main emphasis in this paper is a comparative study of these elimination constraints with reference to their approximation}elimination e$ciency set within di!erent approximation schemes. 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Fast nearest-neighbor search; Approximation}elimination search; Partial-distance search; L , L constraints; Fast vector quantization encoding

1. Introduction Nearest-neighbor search consists in "nding the closest point to a query point among N points in K-dimensional space. The search is widely used in several areas such as pattern classi"cation, nonparametric estimation, information retrieval from multi-key data bases and in image and speech data compression. Reducing the computational complexity of nearest-neighbor search is of considerable interest in these areas. In this paper, we discuss fast nearest-neighbor search in the context of vector quantization encoding. Vector quantization encoding is a powerful data compression technique used in speech coding, image coding and speech recognition [1}7]. Vector quantization encoding is the minimumdistortion quantization of a vector x"(x , x ,2, x ), ) (referred to as the test vector) using a given set of N

* Corresponding author. Tel.: #61-7-3875-6536; fax: #617-3875-5198. E-mail address: [email protected] (K.K. Paliwal).

K-dimensional codevectors called the codebook , of size N, under a given distance C"+c , G G2 , measure d(x, y). This involves "nding the nearest-neighbor of x in C, given by q(x)"c : d(x, c ))d(x, c ), H H G i"1,2, N which requires N distance computations d(x, c ) using the exhaustive full search. G 1.1. Elimination-based fast nearest-neighbor search The basic structure of the sequential full-search which obtains the nearest-neighbor codevector c is as follows: H d "R (a very large number) ALL For i"1,2, N d "d(x, c ) G G if d (d then j"i; d "d G ALL ALL G next i Here, at any given stage in the search, c is the currentH nearest-neighbor and the current-nearest-neighbor ball b(x, d ) is the surface de"ned by x : d(x, c )"d . For ALL H ALL the Euclidean distance between c "(c , k"1 ,2, K) H HI and x"(x , k"1 ,2, K) given by d(x, c )" I H [ ) (x !c )], the current-nearest-neighbor ball HI I I

0031-3203/00/$20.00 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 1 - 3 2 0 3 ( 9 9 ) 0 0 1 3 4 - X

1498

V. Ramasubramanian, K.K. Paliwal / Pattern Recognition 33 (2000) 1497}1510

is a hypersphere of radius d with center at x. The ALL sequential full-search "nds the nearest-neighbor by progressively updating the current-nearest-neighbor c when H a codevector is found closer to the test vector than the current-nearest-neighbor, with each update shrinking the current-nearest-neighbor ball radii to the actual nearestneighbor distance. The "nal nearest-neighbor is one of the codevectors inside the current-nearest-neighbor ball at any stage of the search and consequently, the currentnearest-neighbor ball assumes importance as a geometric object in de"ning the primary search space of interest. In the case of sequential full-search, the location of a codevector is determined with respect to the currentnearest-neighbor ball by computing the distance of each successive codevector to the test vector for comparison with the current-nearest-neighbor distance. As a result, the distances of all N codevectors to the test vector are computed with the search complexity being N distance computations per test vector. A direct approach in reducing the complexity of the above sequential &exhaustive full-search' is to reject a codevector by a &quick-elimination'*without computing its actual distance to the test vector by using computationally less expensive rules to determine which codevectors cannot be nearer to the test vector than the current-nearest-neighbor. This is equivalent to approximating the current-nearest-neighbor ball by geometric constraints which allow easy elimination of codevectors which do not lie inside the current-nearest-neighbor ball. Given the current-nearest-neighbor ball b(x, d ) ALL let E(x, d ) represent the spatial approximation of ALL the ball by some geometric constraint such that b(x, d )-E(x, d ). Then, codevectors which do not ALL ALL belong to E(x, d ) cannot be inside b(x, d ) and elimALL ALL ination consists of rejecting these codevectors. The e$ciency of elimination depends on two intrinsic factors of the elimination constraint: (i) the relative cost of the elimination computation with respect to the distance computation and (ii) the number of codevectors retained after elimination in comparison to the actual number of codevectors inside the current-nearest-neighbor ball. This is essentially governed by the computational cost in determining whether a codevector does not belong to E(x, d ) and the extent to which E(x, d ) approximates ALL ALL b(x, d ) spatially in terms of volume. Fig. 1 illustrates ALL this with a simple geometric constraint used in one of the earliest elimination-based fast search [8] and which forms the basis of hypercube-based elimination search reported subsequently [9}12]. Here, the geometric constraint is simply the projection of the current-nearestneighbor ball b(x, d ) on to a coordinate axis k, given by ALL E(x, d )"+y: "y !x "(d ,, i.e., E(x, d ) is the reALL I I ALL ALL gion between the hyperplanes H"+y: y "x !d , I I ALL and H"+y: y "x #d ,. A direct elimination is I I ALL achieved by rejecting the codevectors +c : c 'x # G GI I d or c (x !d ,, incurring atmost two scalar ALL GI I ALL

Fig. 1. Example of geometric constraint for elimination.

comparisons to reject a codevector as against a vector distance computation. (This elimination is carried out more e$ciently as in Ref. [8], using a binary search on a list of ordered coordinates of the codevectors to eliminate codevectors whose coordinates fall outside the projections of the current-nearest-neighbor ball b(x, d ).) ALL Thus, in the sequential full-search, given a currentnearest-neighbor ball, the code-book is reduced by applying the elimination rule C"C!+c : c , E(x, d ),. G G ALL Since the current-nearest-neighbor ball shrinks for every update of the current-nearest-neighbor, i.e., when a codevector closer to the current-nearest-neighbor has bean found, the above elimination need to be done only for every update of the current-nearest-neighbor using the E(x, d ) corresponding to the updated d . The ALL ALL codebook size thus reduces progressively with the aid of the elimination step and results in a reduction in the number of distances computed if each elimination step rejects some codevectors. 1.2. Approximation-elimination search In addition to the intrinsic nature of the geometric constraint in approximating any given current-nearestneighbor ball volume, another important aspect in#uencing the elimination e$ciency of the constraint is the size of the current-nearest-neighbor ball. Clearly, for a given constraint, the elimination e$ciency is better for a smaller current-nearest-neighbor ball. Hence the relative closeness of the current-nearest-neighbor codevector to the test vector and the sequence in which the test vectors are selected for distance computation in#uences the rate at which the codebook size is reduced in the elimination based search. In general, in the sequential search with the elimination step, selecting a codevector close to the test vector in the initial stages of the search will help in the elimination


of a large number of codevectors in the early steps of the search thereby avoiding the distance computations of these codevectors. For instance, considering the codevectors in the order of decreasing distance to the test vector represents a worst-case condition, since no geometric constraint of the current-nearest-neighbor ball can eliminate any codevector at each update. In contrast, selecting the codevectors in the order of increasing distance represents the best possible situation, since the search then starts with the actual nearest-neighbor and subsequent search merely amounts to verifying that no other codevector lies within the given eliminationconstraint approximating the current-nearest-neighbor ball. In this case, the elimination step rejects the maximum number of codevectors in the "rst step itself and subsequent search is performed on a reduced set of codevectors. The resultant overall reduction in search complexity is then maximum for a given eliminationconstraint. Thus, the distances d(x, c ), i"1,2, N used in the G search provides the exact information for the best ordering in which the codevectors should be considered for search under the elimination based sequential search. However, getting this exact ordering information amounts to a full-search. Therefore, in order to gain any advantage in selecting codevectors (for distance computation) such that the distance to codevectors close to the test vector are computed in the early stages of search, it is necessary to characterize the closeness of a codevector to the test vector using some approximation of the distance measure; such an explicit &codevector selection' step is referred to as an &approximationstep' [13]. In order to gain any advantage from the approximation step in reducing the complexity of the search, the approximation criterion has to be computationally less expensive than the actual distance measure being used in the search, and should characterize the spatial organization of the codevectors with respect to the test vector so as to re#ect their relative closeness to the test vector. The approximation}elimination procedure as described above can be given as a fast-search generalization over the sequential full-search as follows: d "R (a very large number) ALL Do until codebook C is empty c "arg minc C Approx(x, c ) [Approximation] G Z J d"d(x, c ) [Distance computation] G if d(d then ALL j"i; d "d ALL C"C!+c :c ,E(x, d ), [Elimination] K K ALL endif enddo In the above, the approximation and elimination costs incurred in the computation of arg mincJ C Approx(x, c ) Z J

1499

and +c : c , E(x, d ), in comparison to the number of K K ALL distance computations saved by these operations crucially determine the overall reduction in the search complexity. The important issue in achieving fast search under the approximation}elimination framework is in "nding e$cient approximation and elimination criteria which are computationally less expensive than the distance computations but at the same time provide e$cient approximation of the distance measure and the currentnearest-neighbor ball volume. 1.3. Previous work The main elimination rules which have been employed for fast nearest-neighbor search are the partial distance elimination, hypercube elimination, elimination based on the absolute-error-inequality approximation of the Euclidean distance and the triangle-inequality-based elimination. Some of the earlier papers which are based on these elimination rules and which have addressed the issue of determining a &good' search sequence for a given test vector to improve the e$ciency of elimination schemes are [8}26]. 1.3.1. Partial-distance search A simple but e$cient elimination-based search which o!ers improvement over the sequential exhaustive fullsearch is the &partial distance search' method [10,27]. This is applicable to cases where the partial-vector distance attains the full-vector distance in a monotonically nondecreasing manner with the addition of the vector component distances. This allows a codevector to be rejected on the basis of its partial or accumulated distance, i.e., without completing the total distance computation * the elimination being carried out during the distance computation itself. In an earlier paper [28], we have addressed the role of the approximation step in the partial distance-based elimination. Here, we had proposed an ordering of codevectors according to the sizes of their corresponding clusters, i.e., sequencing the codevectors in the order of decreasing probability of being the nearest-neighbor to a given test vector, to improve the savings in computation achieved by the partial distance elimination. The ordering is done in a preprocessing stage and serves as an implicit approximation step in the actual search by providing a favorable sequence of codevectors for the partial distance based elimination. In addition, it was noted that the codebooks obtained at the end of the training process using the clustering algorithms such as the Linde}Buzo}Gray algorithm [29] have arbitrary orderings and are not guaranteed to be arranged in the favorable order. In addition, the implications of dimensionality on the additional complexity reduction due to the proposed ordering was brought out using the asymptotic equipartition property of block coding and

1500


it was shown that the proposed favorable ordering would be most useful for codebooks designed on low entropy distributions such as when the dimensionality is low. 1.3.2. Triangle-inequality-based fast search Another important elimination rule which has been used extensively for fast nearest-neighbor search is the triangle inequality-based elimination, applicable when the distance measure is a metric [13}52]. This elimination rule corresponds to a hyperannulus constraint of the search space, where codevectors lying outside the hyperannulus region formed between two concentric hyperspheres centered at an &anchor point' are eliminated. The e$ciency of the triangleinequality (or hyperannulus) elimination increases with the number of distinct anchor points used, as the intersection volume of the hyperannulus corresponding to the multiple anchor points becomes more localized, consequently retaining less codevectors after elimination. The di!erent applications in which triangle-inequality-based elimination has found use and motivated development of fast algorithms are information retrieval from data bases [31], document retrieval [32], nonparametric classi"cation [33], fast isolated word recognition [13,38,40,50,51], and fast vector quantization of speech and image data [20,41,43]. The search based on the triangle inequality elimination in most algorithms is based on the interpoint distances computed during a preprocessing phase and set in a branch and bound framework where the search is organized using a hierarchical decomposition of clustered data into a tree structure [13,31}33,36}39,42,49]. The main algorithmic aspects which have received attention under this framework have been the determination of the clusters and the anchor points within each cluster [31,32], e$cient hierarchical decomposition and tree structuring of the given points [33,36,39], improvements in providing additional constraints for elimination [37], the issue of optimum number and location of the "xed anchor points [34], [41], [44}46,48,52], and optimum use of a given set of precomputed interpoint distances by using procedures which approximate the missing interpoint distances [42]. 1.4. Organization of the paper In this paper, we are concerned mainly with the class of elimination rules based on the approximations of Euclidean distance, namely, hypercube elimination and elimination based on the absolute-error-inequality approximation of the Euclidean distance. The main emphasis of this paper is to provide an overview of fast search using these elimination rules under an explicit &approximation}elimination' framework. In addition,

the paper is particularly oriented towards providing additional geometric insight to the main approximation-elimination schemes in an uni"ed framework and empirical studies characterizing the approximation-elimination e$ciency of the di!erent approximation}elimination search in detail. In Section 2, algorithms based on the ¸ , ¸ approxi mation of the ¸ norm are considered with reference to earlier work related to the hypercube constraint of the search space [9,11], the ¸ -based &minmax' algorithm [10] and its subsequent improvement using the absolute-error-inequality-based elimination [12]. These algorithms are viewed under the approximation}elimination framework and geometric details, not brought out originally in these algorithms, are provided for additional insight. The absolute-error-inequality is noted to be the general equivalence of the ¸ and ¸ norms. The ¸ , ¸ -based elimination criterion are geometrically seen as based on the constrained minimization and maximization of the ¸ , ¸ norms given ¸ norm. Approxima tion}elimination based on the ¸ constraint is shown to be more e$cient than the ¸ constraint based on geometric interpretation of the norm equivalence relationship and the volume ratios of the ¸ surface and the ¸ (hypercube) surface for a given ¸ ball. The rela tive e$ciencies of these approximation}elimination schemes are characterized and studied with respect to dimensionality.

2. Search based on L , L 1 approximation One of the earliest elimination-based fast search [8] is based on the projections of the codevectors on a coordinate axis and forms the basis of the ¸ (hypercube) search. This algorithm represents a classic example of search under the approximation}elimination framework. The basic principle behind this algorithm, as illustrated in Fig. 1, is the use of the projection of the current-nearest-neighbor ball b(x, d ) on to a coordiALL nate axis k, as a simple geometric constraint for elimination. This is given by E(x, d )"+y : "y !x "(d ,, ALL I I ALL i.e., E(x, d ) is the region between the hyperplanes ALL H"+y : y "x !d , and H"+y : y "x #d ,. I I ALL I I ALL The algorithm given in Ref. [8] can be described under an approximation}elimination framework as follows: In a preprocessing step, the codevector coordinates are ordered on some coordinate axis k and the corresponding indices stored. During the actual search, for a given test vector x, the codevectors are examined in the order of their projected distance from x on axis k, i.e., the approximation criterion is simply arg mincG C "x !c ". EliminaZ I GI tion is done by using two binary searches to truncate the ordered list of codevectors so as to retain only those codevectors whose coordinates fall within the projections of the current-nearest-neighbor ball b(x, d ). ALL


This algorithm can be notionally represented in the approximation}elimination form as follows: d "R (a very large number) ALL Do until codebook C is empty c "arg mincl C "x !c "; C"C!c G Z I GJ G d"d(x, c ) G if d(d then ALL j"i; d "d ALL C"C!+c : "c !x "*d , K KI I ALL endif enddo Friedman et al. [8] analyze the performance of this algorithm for uniform distribution and Euclidean distance and obtain an upper bound for the average number of codevectors examined for "nding k nearest-neighbors as n\[kK¹(K/2)])(2N)\). This performance e$ciency decreases rapidly with increase in dimension K; for instance, the average number of distances computed for K"8 and N"1000 being about 600 as reported in the simulation results of Ref. [8]. This is primarily due to the fact that use of only one axis for approximation and elimination provides a very poor constraint for higher dimensions. 2.1. Hypercube-based elimination A direct generalization of the above single-axis-based search is to use all the axis k"1 ,2, K to provide an improved geometric constraint around the current-nearest-neighbor ball. This results in the hypercube-based elimination search as used in subsequent fast search algorithms [9}12]. The geometric constraint in the hypercube-based elimination is E(x, d )"+y : "y !x "( ALL I I d , ∀k"1 ,2, K,, i.e., E(x, d ) is the smallest hyperALL ALL cube formed by the 2K hyperplanes given by HY "+y : y "x !d , and H "+y : y "x #d , I I I ALL I I I ALL for k"1, 2, K, containing the current-nearest-neighbor ball. Elimination by the hypercube constraint can be realized within a full-search structure as a part of the squared-error distance computation, by directly checking if each codevector is within the hypercube. By this, a codevector c can be rejected if c 'x !d or G GI I ALL c 'x #d for any k"1 ,2, K, thus requiring 1 to GI I ALL 2K scalar comparisons for the rejection of a codevector. Alternately, a codevector c can be rejected, as in Ref. G [11], by checking if each of the component distances "c !x " is greater than the current-nearest-neighbor GI I distance d . By this, a codevector can be rejected with ALL a cost of 1 to K scalar subtractions and comparisons prior to a full distance computation. Any codevector passing all the K tests is inside the hypercube inscribing the current-nearest-neighbor ball and is therefore tested by the full distance to determine if it is inside the currentnearest-neighbor ball.

1501

However, a computationally more elegant and simpler scheme for hypercube-based elimination was proposed much earlier by Yunck [9]. This is based on the observation that the hypercube is de"ned by HC(x, d )" ALL I (x !d , x #d ) and the codevectors conI ALL I ALL I tained within the hypercube can be obtained as ) S I I where S "+c : x !d )c )x #d ,. For a speI G I ALL GI I ALL ci"ed hypercube, the sets S , k"1,2, K are determined I using two binary searches on ordered codevector indices on each of the axis and the subset of codevectors within the hypercube given by their intersection is determined by a simple and e$cient multiset-intersection method in Ref. [9]. The basic algorithm in Ref. [9] based on this procedure is mainly directed towards "nding the nearest-neighbor under the ¸ metric (or Minkowski &max metric') and can also be easily applied for search under other general ¸ metric (Minkowski p-metrics). N The ¸ metric is given by ¸ (x, y)"( ) "x !y "N)N, N N I(x, Iy), is Igiven and the ¸ metric, obtained as lim ¸ N N "x !y ". The isometric surby ¸ (x, y)"max I I2 ) I face determined by the ¸ metric is an hypercube of side 2¸ (x, y) centered at x and with the vector y lying on one of its sides. The hypercube based search constraint for a current-nearest-neighbor ball of radius d centered at ALL x, is thus the isometric ¸ surface +y : ¸ (x, y)"d , ALL inscribing the current-nearest- neighbor ball. This constraint eliminates all codevectors c whose ¸ (x, c ) disG G tance is greater than d . This is essentially a part ALL of the general equivalence between ¸ and ¸ norms given by ¸ (x, y)/(K)¸ (x, y))¸ (x, y). This is illustrated in Fig. 2. Fig. 2(a) shows the bounds on ¸ (x, y) given ¸ (x, y)"d. The lower bound ¸ (x, y)"d/(K and the upper ¸ (x, y)"d correspond to y being at A and B, respectively, for a given x. This can also be seen geometrically as in Fig. 2(b) which shows the bounds on ¸ (x, y) given ¸ (x, y)"d. The lower bound ¸ (x, y)"d and the upper bound ¸ (x, y)"d(K cor respond to y being at A and B, respectively, for a given x. The above inequalities describing the norm equivalence correspond to constrained minimization, maximization of the ¸ norm given x and +y : ¸ (x, y)"d, or the constrained minimization, maximization of the ¸ norm given x and +y : ¸ (x, y)"d,. Two metrics d and d on a space X are equivalent if there exist constants 0(c (c (R such that c d (x, y)) d (x, y))c d (x, y)∀x, y3X. Alternatively, the inequality can also be given as: there exist constants 0(e (e (R such that e d (x, y))d (x, y))e d (x, y)∀x, y3X. The ¸ norms N satisfy this equivalence in general. The equivalence between ¸ and ¸ norms forming the hypercube constraint and ¸ and ¸ norms forming the absolute-error-inequality constraint (Section 2.3) directly follow from this.

1502


Fig. 2. Equivalence of relationship between ¸ and ¸ norms: (a) lower (¸ ) and upper (¸ ) bounds of ¸ (x, y) given x and ¸ (x, y)"d and, (b) lower (¸ ) and upper (¸ ) bounds of ¸ (x, y) given x and ¸ (x, y)"d.

Finding the nearest neighbor under the ¸ metric thus basically involves "nding the smallest hypercube with no codevectors inside it, but with one codevector on one of its sides. In Ref. [9], this is achieved by means of an expanding hypercube procedure, where the search starts with a small hypercube and expands the hypercube until it contains exactly one codevector on its side and no codevectors inside. For search with the general ¸ metric, the search consists of essentially carrying N out a full-search within the small subset of codevectors falling within an initially small, expanding hypercube until the current-nearest-neighbor ball is within the hypercube. This is illustrated for search with the ¸ norm in Fig. 3. Here, the codevector c is the nearest neighbor of the test vector under ¸ norm. However, the ¸ nearest neighbor is codevector c which is closer to x than c . (This can be understood based on the equivalence relation shown in Fig. 2.) The ¸ nearest neighbor is determined in a second step by using the hypercube corresponding to ¸ (x, c ). The main step in this algorithm by Yunck [9] is the determination of the subset of codevectors within a speci"ed hypercube using the multi-set intersection procedure described before. An important factor determining the e$ciency of the search is the size of the initial hypercube; if no codevectors are found inside the cube the search incurs additional overheads in appropriately expanding the cube. The main complexity of the search is in this &cube"nding' step which is estimated to be proportional to KN in terms of shift-operations as implemented in Ref. [9]. The hypercube method in Ref. [10] is an identical search procedure, with the additional feature being the analysis pertaining to "nding the optimal initial hypercube size for a uniform distribution. The optimality is with respect to "nding an e!ective trade o! between the

Fig. 3. Approximation by ¸ (x, c ); c "argmincG C ¸ (x, c ); G Z G c "argminci C ¸ (x, c ). Z G

two complexities: (i) the overhead complexity in expanding the hypercube until it contains a codevector and (ii) the average number of codevectors contained in the initial hypercube which determines the main complexity of number of distances calculated in performing a full-search in the subset of codevectors within the hypercube. Here, it is shown that the average number of codevectors for the optimal initial hypercube thus determined is approximately equal to 2), the average complexity of the search thus increasing exponentially with dimension. In terms of determining the subset of codevectors within a speci"ed hypercube, Ref. [10] uses a multiset-intersection method similar to that proposed earlier in Ref. [9].


2.2. Minmax method In the &minmax' method [10], the ¸ metric is used explicitly as an approximation criterion in selecting the initial codevector. Subsequent elimination is carried out using the ¸ values computed between the test vector and the codevectors for approximation. This algorithm can be given in the approximation-elimination form as follows: d "R (a very large number) ALL Do until codebook C is empty c "arg mincJ C ¸ (x, c ); C"C!c G Z J G d"d(x, c ) G if d(d then ALL j"i; d "d ALL C"C!+c : ¸ (x, cm )*d , ALL K endif enddo The above search is carried out by computing ¸ (x, c ) G for i"1,2, N in a "rst step. After the "rst elimination using the ¸ values, the remaining codevectors are ordered according to their ¸ distance to the test vector such that subsequently, approximation consists in examining the codevectors sequentially in the order of increasing ¸ distances to the codevectors. This can be viewed as a direct generalization over the single-axis projectionbased selection procedure of Ref. [8]. The main shortcoming in the minmax method is the high overhead complexity of the ¸ -based approxima tion. The initial ¸ -based approximation is essentially a full-search using the ¸ distance and has a cost of [0, NK, (N!1)#N(K!1)] macs (multiplications, additions, comparisons) per vector. This initial approximation step of the search itself has an additional overhead cost of N(K!1) comparisons over the ¸ -distance based full-search and the search complexity reduction here is mainly with respect to reducing the NK multiplications per test vector to a low value. 2.3. Absolute error inequality Soleymani and Morgera [12] proposed an algorithm as an improvement over the minmax method of Ref. [10] with two additional features: (i) it reduces the overhead cost of determining the initial codevector by ¸ approxi mation by using a procedure which is essentially a partial-distance realization of the ¸ -based full-search, and (ii) it uses an additional elimination rule termed the &absolute error inequality' to eliminate codevectors during the distance computation alongwith the hypercube elimination. The absolute error inequality is based on the observation that ) ) if (x !c )(d then "x !c "(d(K. I GI I GI I I

(1)

1503

A codevector c which does not satisfy ) "x ! G I I than c "(d(K, i.e., whose absolute error is greater GI d(K will therefore have its squared error distance greater than d and can be rejected on the basis of its absolute error without having to compute the more expensive squared error distance. This algorithm can be given in terms of the following steps: E Select initial codevector c as ¸ nearest neighbor in G a full-search with partial distance realization: E Distance computation: d "d(x, c ) ALL G E Perform full-search of codebook C using (i) hypercube and (ii) absolute error inequality elimination rules during distance computation with each codevector c in K C. Its distance to the test vector d(x, c ) is computed K and compared with the current-nearest-neighbor distance d only if the codevector passes these two tests. ALL The two elimination rules are carried out as follows for each codevector c 3C: K E Reject c if "x !c "'d , k"1,2, K. K I KI ALL E Reject c by partial distance method in the computaK tion of the absolute error ) "x !c " with the KI I I reference absolute error distance being d (K. ALL 2.4. ¸ -approximation-based search In this section, we highlight the &absolute error inequality'-based elimination in the above algorithm and interpret it geometrically as a space constraint provided by the smallest ¸ surface inscribing the current-near est-neighbor ball, i.e., the given ¸ norm. We consider this notion with additional details and compare the ¸ and ¸ -based approximation}elimination using a search procedure in the minmax form. The proposed search procedure using the ¸ values for approxima tion}elimination is a more e$cient realization of ¸ based elimination than that of Ref. [12] and is shown to o!er lower complexity than the minmax procedure [10] which uses the ¸ values for approximation}elimina tion. The absolute error ) "x !c " in the inequality I GI used in Ref. [12] is the ¸Inorm between x and c and G inequality (1) is basically the equivalence between ¸ and ¸ norms given by ¸ (x, y))¸ (x, y))¸ (x, y)(K. (2) This is illustrated in Fig. 4. Fig. 4(a) shows the bounds on ¸ (x, y) given ¸ (x, y)"d. The lower bound ¸ (x, y)"d and the upper bound ¸ (x, y)"d(K cor respond to y being at A and B, respectively, for a given x. This can also be seen geometrically as in Fig. 4(b) which shows the bounds on ¸ (x, y) given ¸ (x, y)"d. The lower bound ¸ (x, y)"d/(K and the upper ¸ (x, y)"d correspond to y being at A and B respective ly for a given x.

1504


Fig. 4. Equivalence of relationship between ¸ and ¸ norms: (a) lower (¸ ) and upper (¸ ) bounds of ¸ (x, y) given x and ¸ (x, y)"d and, (b) lower (¸ ) and upper (¸ ) bounds of ¸ (x, y) given x and ¸ (x, y)"d.

The above inequalities describing the norm equivalence can be obtained as a constrained minimization, maximization of the ¸ norm given x and +y : ¸ (x, y)"d, or the constrained minimization, maximization of the ¸ norm given x and +y : ¸ (x, y)"d,. (Soleymani and Morgera [12] obtain the absolute error inequality (2) by induction. Though the inequality is "nally shown to be satis"ed for general K, we note here that, an intermediate step in Ref. [12] is incorrect: this involves proving an alternate inequality ((K!1)(d!z )#z (d(K. In showing ) ) this, Ref. [12] incorrectly obtains the maximum of the to be (2K!1)d/ function ((K!1)(d!z )#z ) ) (4K!3 at z "d/(4K!3; the correct maximum of the ) function is d(K at z "d/(K.) ) The use of absolute error inequality in elimination can be seen in Fig. 4(a) as it implies that points lying outside the surface ¸ (x, y)"d(K cannot lie within the ¸ (x, y)"d surface and hence cannot be nearer to x than y. The geometric constraint provided by the absolute error inequality around the current-nearest-neighbor ball is thus the isometric surface ¸ (x, y)"d(K. This is essentially a simplex polytope, which can be viewed as formed by 2) hyperplanes each given by +y: ) "y "" I Iaxes d(K, with intercepts at d(K on the coordinate in each quadrant of the K-dimensional space with the origin translated to x. Note that the above algorithm essentially obtains a subset of candidate codevectors C"+c : ¸ (x, c )) H H d (K and ¸ (x, c ))d , by employing the upper ALL H ALL bounds of both ¸ , and ¸ distances corresponding to the initial codevector c obtained by the ¸ based approxiG mation in Step 1 (c "arg mincH C ¸ (x, c ); d "d(x, c )). G Z H ALL G The codevectors in C are potential updates for the current-nearest-neighbor and have their ¸ (x, c ) distances H bound by [¸ (x, c ), d (K] as can be seen from Figs. G ALL 2(b) and 4(b), corresponding to the initial codevector

c obtained by ¸ approximation and the ¸ (x, c ) G G bound for ¸ "(x, c )"d . The algorithm selects can G ALL didate codevectors from C by ¸ ordering and performs elimination using both hypercube and absoule error inequality constraints. 2.4.1. Comparison of ¸ and ¸ elimination constraints The relative elimination e$ciencies of the ¸ and ¸ surfaces in providing space constraint for search can be seen by comparing their respective volumes for an unit hyperspheres, i.e., corresponding to an unit current-nearest-neighbor ball radius ¸ (x, y)"1. The volume is the luminance or brightness, I is the hue, and Q is the saturation or depth of color. Luminance refers to color intensity. Hue is the dominant color such as orange, red or yellow and saturation is the amount of white light mixed with a hue. With respect to a television set, > represents the black and white image and is what is displayed by a black and white television set. I and Q correspond to the two color adjustment knobs found on a color television set and contain the additional information used to produce color images. For this reason, I and Q together are sometimes referred to as chrominance values. An RGB representation can be obtained from a YIQ representation by inverting the above transformation as

1556

C. Clausen, H. Wechsler / Pattern Recognition 33 (2000) 1555}1560

follows:

R

1.00

0.96

0.62

>

G " 1.00 !0.27 !0.65

I .

B

Q

1.00 !1.10

spatial correlation, further compression is possible. In the next section, a new method for encoding color images using a learning method based upon principal component analysis which takes advantage of spatial correlation of color is presented.

1.70

2. Learning method

Television transmission uses less bandwidth for the hue and saturation components than it does for luminance. Similarly, when compressing color images, the two chrominance values, I and Q, can be decimated by between one-quarter and one-half without perceptible image degradation [2]. It is also possible to decimate > as results shown in Section 3 below. Through decimation, the number of bits required to represent the original color image can be reduced by as much as or more before gray-scale encoding begins. Encoding a color image then consists of converting an RGB image to YIQ coordinates, decimating >, I and Q and then applying a gray-scale compression algorithm such as JPEG [3, pp. 347}408] to each of the >, I, and Q coordinates. To reconstruct a color image, decode each of >, I, and Q and then interpolate the >, I and Q to their original size. Finally, use the inverse transformation above to obtain RGB coordinates for the image. Another color transformation is the YUV code used by MPEG 1. Again > is brightness or luminance and ; and < are chrominance values. The luminance > is de"ned as before, but the color information is stored di!erently. The YUV transformation is de"ned as follows:

0.59

0.11

R

; " !0.15 !0.29

0.44

G .

>

G " 0.994 !0.381 !0.583

; .

B

, ;, and , I, and Q components, respectively, and then use the inverse transform to convert back to the RGB representation. We perform the same operations for the YUV representation, using 6, 5, and 5 bits, respec-

Fig. 2. Reduced color image.

1558


Table 1 Color compression results YIQ

YUV

PCA/BP Learning

Picture

R

G

B

C

R

G

B

C

R

G

B

C

Lady Baboon Boy Cave Chalk Coral

29.4 28.2 29.3 28.2 28.9 29.3

41.3 41.1 41.6 40.2 40.0 39.4

31.0 32.8 31.3 33.8 30.7 31.6

31.7 31.5 31.8 31.7 31.3 31.8

31.5 30.9 30.9 31.4 29.3 30.6

41.3 39.5 40.8 40.9 41.0 40.0

26.7 27.3 27.0 27.8 27.1 27.8

30.1 30.3 30.2 30.9 29.7 30.6

32.2 34.6 31.5 31.7 34.9 29.5

32.3 34.7 31.7 36.1 35.7 28.5

35.6 35.5 34.7 32.8 34.8 30.2

33.1 34.9 32.4 33.2 35.1 29.3

Fig. 3. Test images.

tively, for >, ;, and < components. For each of YIQ and YUV we obtain a compression ratio of 1.5. For the learning approach, we learn the transform weights as well as inverse transform weights as described above using a value of 1.0E-6 for the learning rate and use 20,000

(3)(8)/(6#5#5)"1.5

repetitions. Learning takes about 9 s on a 133 MHz Pentium processor. We convert each image to the principal component representation as in Fig. 2 and use 8, 6, 6, and 6 bits, respectively, for each of the "rst four principal components. As explained above, this bit allocation results in a compression ration of 3.7. Next we use the weights for the inverse transform to convert back to the RGB representation. For each image and each color reduction method, we compare the original RGB image


1559

Table 2 Progress of learning

1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15.

Step

Picture

R

G

B

C

Rep

Train Train Train Train Train Test Test Train Train Test Test Train Train Test Test

Chalk Cave Chalk Cave Chalk Boy Lady Cave Chalk Boy Lady Cave Chalk Boy Lady

9.3/25.6 23.9/23.5 25.0/25.8 24.4/25.7 25.4/27.3 29.5 29.5 26.2/25.4 25.2/26.8 30.4 28.9 25.5/27.0 27.3/31.4 31.0 30.5

9.3/26.9 25.1/25.7 27.3/27.0 25.9/27.2 26.4/29.0 22.7 23.5 27.4/28.1 28.7/30.8 23.9 24.2 28.4/29.6 31.5/34.1 24.4 24.7

7.3/25.5 24.4/24.4 25.0/26.9 25.4/26.8 28.2/29.3 29.3 28.4 27.6/28.6 30.0/32.2 28.9 28.6 29.7/30.8 31.8/34.5 29.1 28.9

8.5/26.0 24.4/24.4 25.6/26.5 25.2/26.5 26.5/28.4 25.9 26.3 27.0/27.1 27.5/29.3 26.8 26.7 27.4/28.8 29.8/33.1 27.2 27.3

0/1000 1000/2000 2000/3000 3000/4000 4000/5000 5000 5000 5000/6000 6000/7000 7000 7000 7000/8000 8000/9000 9000 9000

with the resulting RGB image produced from the reduced image. We calculate four PSNR values for each image in order to make comparisons. These four values correspond to the noise induced in red (R), green (G), blue (B), and combined (C) channels. Results are shown in Table 1. In Table 1, R, B, G, and C columns hold PSNR values for red, green, blue and combined RGB, respectively. In "ve of six cases, we see that learning outperforms YIQ and YUV with respect to PSNR, even though in all six cases, learning achieves more than twice the compression as the other two methods. The performance of learning is due to two factors. First, the learning method adapts to individual images where YIQ and YUV apply globally without change to all images. Second, the learning method takes advantage of spatial correlation of color in images which the YIQ and YUV methods do not. The primary disadvantage of the learning approach is the computation time required for learning. Once learn PSNR is Peak Signal to Noise Ratio. PSNR"20 log (d/RMS) where d is the peak signal which is usually 255 and RMS is the root-mean-square error between the original image component and the estimated image component. For each of red, green and blue PSNR calculations,

1 , (p !p( ) G G N G where N is the number of pixels in the image, p is the correG sponding red, green, or blue value of the ith pixel in the original image and p( is the estimate of p . The combined channel RMS, G G denoted RMS "( (RMS#RMS#RMS) where RMS , A P E @ P RMS , and RMS are the RMS for red, green and blue compoE @ nents, respectively.

RMS"

ing has taken place, the conversion to the reduced image and back to the new RGB image is very fast. Hence, if the results of learning for one image could be applied to another image, signi"cant improvement in overall compression time would result. To show that learning from one image does apply to other images we try a second experiment in which we train on the Chalk and Cave images in Fig. 3 and apply training results to the Boy and Lady images. Table 2 below shows PSNR results for this experiment where normal rows correspond to training exemplars and bold rows correspond to test images. Learning using the training examples proceeds in steps of 1000 repetitions (approximately 0.5 s) instead of 20,000 repetitions as used for results in Table 2. The learning rate is again 1.0E-6. In cells where two PSNR values are depicted (e.g. 9.3/25.6), PSNR values are for training images and the "rst number is the PSNR just before training is performed and the second number is the PSNR just after training is performed. In cells where only one PSNR value is shown, the PSNR is for a test image and no training was conducted at that step. The Rep column shows the number of cumulative training repetitions before and after training at that step. Hence, two numbers are shown for the training exemplars and one number for the test images. Step 1 shows the results for the chalk image when the weights are initialized to random values between !1.0 and 1.0. Table 2 shows that as learning proceeds on the training images, that reproduced image quality of the training images improves, as would be expected. However, we also see that the image quality of the test images improves, even though the test images are not used for training. This experiment shows that it is feasible to

1560


apply training from one image to previously unseen images.

4. Conclusion This paper introduces a hybrid learning method consisting of PCA and backpropagation for reducing (&compressing') an RGB color image to a black and white image with the same number of pixels as the original color image. This reduction achieves a 3.7 compression ratio while retaining the capability of reproducing an approximation of the original color image that is optimal with respect to the minimum square error. The resulting black and white image can be further compressed using a gray-scale compression method such as JPEG. Experimental results show that as our hybrid learning method adapts to local (spatial) image characteristics it outperforms the YIQ and YUV standard compression methods. Our experiments also show that it is feasible to apply training results from one image to previously unseen images.

References [1] D.B. Judd, D.L. McAdam, G. Wyszecki, Spectral distribution of typical daylight as a function of correlated color temperature, J. Opt. Soc. Amer. A 54 (1964) 1031}1040. [2] Y. Fisher (Ed.), Fractal Image Compression, Theory and Application, Springer, New York, NY, 1995. [3] M. Nelson, The Data Compression Book, M & T Books, San Mateo, CA, 1982. [4] Healey, E. Glenn, A.S. Steven, B.W. Lawrence, Physics Based Vision: Principles and Practice: Color Vol. 2, A.K. Peters Ltd., 1992. [5] D.B. Judd, G. Wyszecki, Color in Business, Science, and Industry, Wiley, New York, 1975. [6] F. Grum, C.J. Bartleson (Eds.), Optical Radiation Measurements, Vol. 2, Academic Press, New York, 1980. [7] T.D. Sanger, Optimal unsupervised learning in a singlelayer linear feedforward neural network, Neural Networks, (2) (1989) 459}473. [8] J. Hertz, K. Anders, G.P. Richard, Introduction to the Theory of Neural Computation, Addison-Wesley, Reading, MA, 1991.

About the Author*HARRY WECHSLER received the Ph.D. in Computer Science from the University of California, Irvine, in 1975, and he is presently Professor of Computer Science at George Mason University. His research on intelligent systems has been in the areas of PERCEPTION: Computer Vision, Automatic Target Recognition, Signal and Image Processing, MACHINE INTELLIGENCE: Pattern Recognition, Neural Networks, Machine Learning (ML), Information Retrieval, Data Mining and Knowledge Discovery, EVOLUTIONARY COMPUTATION: Genetic Algorithms and Animats, MULTIMEDIA and VIDEO PROCESSING: Large Image Databases, Document Processing, and HUMAN-COMPUTER INTELLIGENT INTERACTION: Face and Hand Gesture Recognition, Biometrics and Forensics. He was Director for the NATO Advanced Study Institutes (ASI) on `Active Perception and Robot Visiona (Maratea, Italy, 1989), `From Statistics to Neural Networksa (Les Arcs, France, 1993), `Face Recognition: From Theory to Applicationsa (Stirling, UK, 1997), and he served as co-Chair for the International Conference on Pattern Recognition held in Vienna, Austria, in 1996. He authored over 200 scienti"c papers, his book `Computational Visiona was published by Academic Press in 1990, he is the editor for `Neural Networks for Perceptiona (Vols. 1 & 2), published by Academic Press in 1991, and co-editor for `Face Recognition: From Theory to Applicationsa, published by Springer-Verlag in 1998. He was elected as an IEEE Fellow in 1992 and as an Int. Assoc. of Pattern Recognition (IAPR) Fellow in 1998. About the Author*CLIFFORD CLAUSEN received the Ph.D. in Information Technology from George Mason University, Fairfax, VA, in 1999. Currently, Cli!ord works for Unisys corporation as a software engineer. His research interests include image compression, signal processing, computer vision, neural networks, pattern recognition, medical information systems, information retrieval, learning systems, and data mining.


Classifying cervix tissue patterns with texture analysis Qiang Ji *, John Engel, Eric Craine Department of Computer Science, University of Nevada, Reno, NV 89557, USA Western Research Company, Tucson, Arizona, USA Received 16 December 1998; accepted 17 May 1999

Abstract This paper presents a generalized statistical texture analysis technique for characterizing and recognizing typical, diagnostically most important, vascular patterns relating to cervical lesions from colposcopic images. The major contributions of this research include the development of a novel generalized statistical texture analysis approach for accurately characterizing cervical textures and the introduction of a set of textural features that capture the speci"c characteristics of cervical textures as perceived by human. Experimental study demonstrated the feasibility and promising of the proposed approach in discriminating between cervical texture patterns indicative of di!erent stages of cervical lesions. 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Texture analysis; Cervix lesion classi"cation; Colposcopic images; Pattern recognition

1. Introduction The incidence of cervical cancer mortality has been dramatically reduced since the introduction of the Papanicolaou (Pap) test. However, the Pap test is unable to accurately and consistently identify premalignant and malignant disease of the cervix. The incidence of falsenegative Pap tests has become shockingly high. In fact, certain studies [1] show that the false-negative Pap tests account for up to 40% of the actual positives in the sample. The high false-negative rate from Pap tests has motivated the use of colposcopy as a standard screening procedure for precancer examination [1]. Another serious problem with Pap-smears is that they produce an enormous number of positive "ndings which need to be followed up. Most of these "ndings turn out to be false positives based on insigni"cant cellular changes. A computer-supported colposcopy can help reduce the false positives of Pap-smears, therefore substantially reducing costs and patient su!ering associated with the

* Corresponding author. Tel.: #775-784-4613; fax: #775784-1877. E-mail address: [email protected] (Q. Ji).

unnecessary biopsies. Finally, colposcopy can signi"cantly improve the ability of physicians to perform accurate punch biopsies for histological analysis. Today, regular cytological screening of the cervix via the Pap test, supplemented by colposcopy and colposcopically directed cervical biopsies, has become an established part of modern gynecological practice. One of the major factors hindering the full utilization of colposcopy is the di$culty in training physicians in the recognition of pathology. Colposcopic images contain complex and confusing lesion patterns. Correctly analyzing and classifying di!erent types of tissues require substantial training. The average physicians in private practice do not get to examine enough patients to maintain such expertise and may need to rely on consultation with colposcopy experts. Furthermore, there is also the rapidly increasing trend for people other than highly trained colposcopists, such as primary care physicians, to be involved in the screening procedure. Therefore, it is necessary to simplify the use of colposcopy and to enhance its capability so that average physicians can correctly recognize various tissue patterns in a consistent and uniform fashion. For this reason, we developed an image analysis system to help physicians better interpret various patterns on colposcopic images and to simplify the operation of colposcopy.


1562

Q. Ji et al. / Pattern Recognition 33 (2000) 1561}1573

Fig. 1. Typical vascular patterns encountered in cervical lesions (adapted from Ref. [2]). (a) network capillaries in original squamous epithelium; (b) hairpin capillaries in original squamous epithelium; (c) and (d) punctation vessels in dysplasia and carcinoma in situ; and (e) and (f) mosaic vessels as seen in dysplasia and carcinoma in situ.

A careful examination of various cervical images reveals regular and repeatable vascular patterns, indicating di!erent stages of dysplasia. In fact, vascular pattern is the most important diagnostic criteria used by colposcopists for recognizing pathology [2]. For example, two basic types of vascular patterns observable in normal or benign lesions are hairpin and network capillaries. On the other hand, di!erent versions of punctation and mosaic vascular patterns may be observed in areas of dysplasia and carcinoma in situ. Fig. 1 shows typical vascular patterns encountered in cervical lesions. Various texture analysis methods have been developed to analyze and classify patterns like these. Classi"cation based on texture analysis is a well-established technique. Many techniques have been developed [3}7] over the years. They have been successfully applied in many "elds including remote sensing, medical diagnosis, and product inspection. The applications of texture analysis techniques in biomedicine focus on cell and tissue classi"cation. For example, Wu and Chen [8] proposed a multi-threshold texture vector for liver tissue classi"cation. Houston et al. [9] investigated di!erent texture measures of ultrasound images for diagnosing prostate cancer and for identifying cancerous lesions. Lachmann et al. [10] proposed to use texture analysis for brain tissue classi"cation from MRI images. Finally, Fortin et al. [11] advocated the use of texture analysis for segmentation of cardiac images.

These examples demonstrate the importance and utility of texture analysis for tissue classi"cation. However, the use of texture analysis techniques (or other image processing approaches) for recognizing and classifying cervix lesions has not been reported in the literature to the best of our knowledge. For this project, we introduce a texture analysis technique for recognizing and classifying cervical tissues. Texture analysis results in a set of feature metrics that describe the characteristics of di!erent vascular patterns. For example, a texture analysis of mosaicism will result in a set of feature measurements which characterize the class of tissues exhibiting mosaicism. Texture analysis is a three-layer process. The "rst layer identi"es the texture primitives, of which texture patterns are composed. The texture primitives are usually geometric entities like pixels or line segments. The second layer extracts certain properties of the identi"ed texture primitives. Depending on the types of primitives, the properties can be tonal properties like intensities for pixels or geometric attributes like length and direction for line segments. The last layer describes the spatial and/or statistical distribution of the primitives in terms of their attributes. The description can be symbolic or statistical. Parallel to the above categorizations, there are two basic approaches to the analysis of textures: structural and statistical. In the structural approach, textures are characterized by the texture primitives and the placement rules. The placement rules relate the centroids of various


primitives in terms of their relative positions like adjacency or nearness. Structural approach emphasizes the shape aspects of the primitives. All structural methods assume that textures are made up of primitives which appear in a near regular, repetitive or periodic spatial arrangement. In statistical methods, the granulation, linearation and randomness that constitute a texture pattern are characterized by taking statistics on the pixels. Traditional statistical methods assume that spatial intensity distribution is related to texture patterns. For example, "ne textures have a spectrum rich in high frequencies while coarse textures are rich in low spatial frequencies. Di!erent techniques [4,6,12] have been developed to describe the intensity distribution. For example, texture may be described as the auto-correlation function, which measures the linear dependence between pixels; or as the gray tone run length, which characterizes textures by the statistics of the gray tone of the run. One of the most popular statistical methods is the spatial gray tone cooccurrence matrix (GTCM) [12]. The entries of the cooccurrence matrix are the second-order joint conditional probabilities of two gray tones occurring at certain distance apart in a given direction. First- and second- order statistics derived from this matrix are used as measures for texture analysis. Statistical methods are good for analyzing disordered textures. A careful examination of the textural patterns relating to cervical lesions revealed the followings. First, texture patterns for cervical lesions are primarily due to the vascular patterns. The non-vascular structures in the cervical images contribute very little to the formation of texture patterns in cervical lesions. Furthermore, the vascular structures are mainly characterized by the geometric shape and spatial distribution of capillaries. The gray levels and thickness of capillaries are irrelevant to vascular patterns. Thus, cervix texture patterns cannot be characterized by the spatial intensity distribution of capillaries. Second, texture patterns for cervical lesions do not exhibit regular repetitive or periodic structures. Based on the above observations, we may conclude that the conventional structural and statistical texture analysis approaches are not directly applicable to characterizing cervical texture patterns. The structural approach is not applicable in that it looks for regular repetitive or periodic spatial arrangements, which are not present in cervical texture patterns. On the other hand, the statistical approach characterizes textures by their intensity distribution. It depends more on the intensity transitions within texture elements than on the structural organization of the texture. To best characterize the texture patterns relating to cervical lesions, we proposed a novel generalized statistical method. Recognizing the fact that cervical textures are primarily represented by the vascular structures, we assume that a signi"cant proportion of the texture in-

1563

formation in cervix lesion is contained in the vascular structures and that the vascular structures can be extracted and approximated by a set of connecting line segments of di!erent lengths and orientations. We therefore chose line segments as the textural primitives. Other researchers proposed the use of edgels as primitives [13]. We believed that capillaries can be better characterized by the high level line segments rather than the low level edgels. Given line segments selected as primitives, the length and orientation of line segments are the natural choices for the primitive properties. In summary, our approach "rst extracts the vascular structures from the original cervical lesion images, followed by vectorizing the extracted vascular structure using line segments. Statistical distributions of the line segments are then constructed. First- and second- order statistics derived from the joint and/or marginal distributions are used as textural measures for cervical lesions classi"cation. The beauty of such a texture characterization is that while it takes full advantage of the statistical approach, it also inherents the power of the structural approach by emphasizing the shape aspects of textures.

2. Algorithm development In this section, we detail our texture analysis technique for analyzing the textural patterns relating to cervical lesions. Following the 3-layer texture characterization process described in the previous section, this section further divides into three subsections. The "rst subsection focuses on texture primitives (line segments) extraction, the second subsection discusses properties of the extracted texture primitives, and the third subsection deals with the texture feature formulation and extraction. 2.1. Texture primitive extraction In the preceding section, we introduced our generalized statistical approach for characterizing the texture patterns relating to cervical lesions. The approach captures the cervical textural information contained in the vascular structures using a set of connecting line segments. This section describes our approach for extracting the line segments that approximate the vascular structures. The approach consists of three steps: image preprocessing, skeletonization, and vectorization as detailed below. 2.1.1. Image preprocessing Image preprocessing is important for classi"cation since it directly a!ects the classi"cation accuracy. This is particularly relevant to this project since vascular structures often coexist with other irrelevant artifacts on the surface of a cervix. Furthermore, the presence of #uids and/or other discharges on cervix surface causes

1564


non-uniform luminance and contrast to the underlying vascular structures. Fig. 2 shows a colposcopic image of a cervix displaying mosaic vascular pattern. As a result, the primary purpose of preprocessing is to digitally remove artifacts present on the surface of a cervix and to compensate the uneven luminance. To this end, a gray-level morphological operation was performed to remove the artifacts, followed by an image normalization operation to adjust for non-uniform luminance and

Fig. 2. A colposcopic image of a cervix displaying mosaic vascular pattern. Note the presence of artifacts and uneven luminance and contrast in the underlying mosaic vascular pattern.

contrast of the original image so that vascular structures di!er only in luminance and contrast would not be distinguished. Artifacts on the surface of a cervix image are identi"ed and separated from the underlying vascular structures based on the morphological di!erences between artifacts and capillaries. The capillaries usually are much more tortuous than artifacts. Artifacts, on the other hand, are usually short and straight segments or small dots. The di!erences in morphology were exploited in separating them based on the theory of mathematical morphology. The technique of morphological opening with a rotating structure creates an image containing only artifacts, which are then subtracted from the original images, yielding images containing predominantly vascular structures. The rotating structure elements are necessary due to random orientations of the artifacts. Detailed description of this algorithm may be found in Ref. [14]. Image normalization is achieved through background subtraction. The morphological rolling ball algorithm [15] creates a mask image that contains the background. The mask image is subtracted from the original to obtain a normalized image, with enhanced vascular structures. Fig. 3(b) shows an example of a morphologically enhanced image. The radius of the ball varies depending on the vascular patterns being studied and image scales. For the examples shown in Fig. 3, a radius of 3 pixels was chosen. 2.1.2. Vascular structure extraction Given the images after preprocessing, the next step is to extract the vascular structures. This is accomplished with a thresholding operation, resulting in a binary image containing primarily vascular structures. Adaptive thresholding is chosen due to variations in local contrasts as shown in Fig. 3(b). Such variations make the vascular

Fig. 3. (a) original image containing mosaic patterns; (b) morphologically enhanced image; (c) binary image of (a) after adaptive thresholding of (b).


structures di$cult to separate from its surroundings with a global threshold. We therefore implemented an automatic adaptive thresholding technique introduced in Ref. [16]. Fig. 3(c) shows the binary image after adaptive thresholding. 2.1.3. Skeletonization and vectorization Following preprocessing, the binary images of vascular structures are skeletonized since we observed that vascular structures can be fully captured by their skeletons. Skeletonization can be regarded as a shape normalization process so that the shape of capillaries are independent from digitization, lighting, scale, and other factors. Furthermore, skeletonization can preserve the critical shape information while removing redundant pixels, greatly facilitating the feature extraction and improving the pattern recognition e$ciency. For the skeletonization process, we employed the thinning algorithm provided by Zhang [17]. The algorithm is simple, fast, and easy to implement. With this algorithm, each capillary is thinned to a skeleton of unitary thickness. Fig. 4(a) shows the skeletonized image of the image in Fig. 3. Primitive extraction is subsequently accomplished through vectorization. Vectorization approximates the thinned vascular structures with connecting line segments. The Hough Transform (HT) is used for line segment detection. A drawback with the conventional HT is that it does not provide any information regarding the line segments. It merely clusters collinear points. Points on a line may represent di!erent line segments. A heuristic method was therefore developed through this research to extend the conventional HT for detecting line segments.

1565

Speci"cally, the line segment detection method developed through this research consists of two steps subsequent to the conventional HT: the "rst step involves grouping collinear points into line segments based on their proximity. To do so, the collinear points are "rst ordered by their column coordinates if their orientations are greater than 903 and by their raw coordinates otherwise. Two sequential pixels belong to the same line segment if the distance between their end points is less than a pre-speci"ed distance. For each line segment, this continues until the distance to the next ordered pixel exceeds the threshold. This ends the previous line segment and starts a new line segment. In many cases, the "rst step results in many short line segments lying along straight lines, some of which are broken because of erroneous edge directions. These shorter line segments must therefore be replaced by the corresponding longer line segments through line merging. This is accomplished in the second step. A metric is needed for line segment merging. It measures how likely two or more shorter line segments are part of the same longer line segment in the image. The metric we chose consists of the following three criterion: "rst, the line segments to be merged must be collinear. Collinearity is measured by the two HT parameters h and o. Two merging line segments should have the same (or close) h and o. The second criteria is proximity, i.e., they must be adjacent to each other. Proximity (nearness) is measured by the ratio of distance between the end points of the two line segments to the length of the line segments. After merging, line segments shorter than a prespeci"ed threshold are discarded as noises. Fig. 4(b) shows the vectorized image of the image shown in Fig. 3(c).

Fig. 4. (a) the skeletonized image of the image in Fig. 3 (c); (b) the vectorized image of the image in (a). In (b), (a) is approximated by connecting line segments.

1566


2.2. Primitive properties extraction and distribution construction With texture primitives extracted, we have available a list of primitives (line segments) that model the vascular structures of the original image data. We need to proceed to the next phase of texture characterization}texture primitive attributes computation and their distributions construction. 2.2.1. Primitive properties What we need to do here is to de"ne and extract properties that can best characterize the selected primitives. Since a line segment can be fully described by its length and orientation, line segment length and orientation are the natural choice of their properties. Given two end points of a line segment, its length and orientation can be easily derived. To be consistent, the orientation of a line segment, a real number, takes counterclock with respect to the positive X-axis and is limited to 0}179 inclusive. Similarly, the length, also a real number, can be computed using the two end points. 2.2.2. Distribution construction The properties of line segments can be treated as random variables. To study these properties, we need to estimate their probability density functions, based on sample observations. Since line segment length and orientation are real values, discretization becomes necessary to study the statistical distributions of line segments. Discretization groups line segments into bins, based on their original values. Speci"cally, the orientations of line segments are uniformly discretized into 180 bins ranging from 0 to 1793. The line segment length is discretized in a similar fashion into ¸ bins, where ¸ was empirically selected as 50. After discretization, the length of each line segment is referred to by the number of the bin it belongs to rather than by its actual length.

With the attributes of line segments discretized, we can proceed to construct the density functions (histograms) of the line segments in terms of length and orientation. A total of three distributions were constructed: one joint distribution and two marginal distributions. The joint distribution characterizes the line segment distribution by its length and orientation. Each point in the joint distribution represents the probability of a line segment with a particular length and orientation. The marginal distributions represent the line segment distribution with respect to length/orientation. Each point in a marginal distribution represents the probability of the line segment of a particular length (or orientation). Since line segments are of di!erent lengths, the orientation distribution of line segments is therefore weighted by their lengths. This results in a more realistic orientation distribution. Fig. 5 shows the two marginal distributions of line segment length and orientation for the vectorized mosaic pattern shown in Fig. 4(b). 2.2.3. Distribution normalization Since texture features are subsequently extracted from the above distributions, we need to ensure the invariance of the above distributions to a$ne transformations, i.e., rotation, translation, and scale invariant. These will, in turn, ensure the extracted texture features invariant to a$ne transformations, which is an important practical concern for classi"cation. To do so, the above distributions should be normalized. The normalization approaches we followed need to achieve only rotation invariance for orientation distribution since it is invariant to translation and scale. Similarly, for length distribution, only scale normalization is needed since it is invariant to translation and rotation. The length distribution is normalized via discretization as discussed before. Since "xed discretization level (50) is used to discretize line segment length, scale only a!ects the discretization intervals. Given the fact that the lengths of the discretized line segments are referred to by

Fig. 5. (a) Marginal probability density functions for orientation (a) and length (b) of the line segments in the vectorized mosaic pattern shown in Fig. 4(b).


their bin numbers rather than by their actual lengths, the discretization makes the distribution independent of scale. Fig. 6 shows the normalized length distributions for the image with mosaic pattern at two di!erent scales. It is clear that their shapes are very much similar after normalization. For orientation distribution, a rotation (which is equivalent to adding or subtracting an angle to each line) may not only cause a linear shift for the interior bins (which may not a!ect the shape of the distribution) but also cause a circular shift for the boundary bins (bins close to 0 or 179) as shown in Fig. 7(a), where the local peak at 03 results from a circular shift of the corresponding local peak at about 1403 in Fig. 5(a). This will alter the shape of the original distribution, rendering incorrect feature values. Therefore, it must be normalized. The normalization is carried out as follows. Identify the peak

1567

of each distribution and shift the peak to the 903 bin and perform the same amount of shift for other angles. The choice of the peak for normalization rather than other distribution marks like valley is because the peak is less sensitive to noises. Fig. 7 shows the orientation distributions image with mosaics at two di!erent orientations before and after normalization. 2.3. Texture features extraction Feature extraction is a dimension reduction process. It consists of deriving characteristic measurements from the input image or its transformations. The characteristic measurements characterizing the cervical lesion patterns are derived from the vectorized images and from the statistical distributions of the extracted line segments. Most textual features we suggest are extracted from

Fig. 6. The normalized length distribution of image with mosaic patterns in Fig. 5 (b) at two di!erent scales 1:1 for (a) and 5:3 for (b).

Fig. 7. The angle distributions of the mosaic image shown in Fig. 4 (b) with and without normalization: (a) The angle distribution of the mosaic image with a 45-degree rotation (without normalization); (b) the angle distribution of (a) after normalization (its shape is very similar to that of Fig. 5(a)).

1568


distributions of line segments. They characterize the shapes of line segments distributions. Additional features are derived directly from the vectorized images to describe the spatial complexity and density (vascular concentration) of the texture patterns. E!orts were made during feature extraction to select features that can relate to the speci"c textural characteristics of the cervical texture patterns. For example, some features relate to such textural characteristics as randomness, contrast, correlation, etc., while others characterize the spatial complexity of the texture patterns. We suggest a set of nine features which can be extracted from each of the two marginal distributions, four features from the joint distribution, and two features directly from the vectorized images, yielding a total of 24 features (2;9#4#2). For illustrative purpose, we will de"ne 4 of the 24 features in this section and explain their signi"cance in relating to the speci"c characteristics of the cervical textures as perceived by human. For detailed de"nitions of all 24 features, refer to Appendix A. Peak density ( f ): measures the strength of the local dominant peak in a marginal distribution, i.e., length or orientation distribution. f "max(p(i)) for i"1, 2,2, N, where p(i) is the prob ability of the ith bin and N is the discretization level. Entropy ( f ): measures the randomness or homogeneity of a distribution with respect to length or orientation. Ratio of the number of intersection points to the number of endpoints ( f ): measures the spatial complexity of the textures. Density ( f ): measures the coarseness (or "neness) of a texture in terms of the amount of edgels per unit area. The average number of edgels per unit area for all pixels is used as a measure of the density. During feature design process, every e!ort was made to devise features that represent the speci"c characteristics of the cervical textures as perceived by human. In this section, we will analyze some of the textural features we proposed and try to relate them to certain textural characteristics. Peak density ( f ) measures the strength of the domi nant length (or orientation) of line segment distribution with respect to a particular attribute. Take the orientation for example, hairpin texture pattern, with most line segments oriented in one direction as shown in Fig. 1, should have the highest f values among all cervical textural patterns while mosaics, with line segment directions scattering in all directions as shown in Fig. 1, should have the lowest. Similarly, for length distribution, hairpin texture pattern also has the highest peak density. Therefore, this feature can discriminate the mosaic and hairpin patterns. Entropy ( f ) measures the randomness or homogen eity of a distribution with respect to length or orientation. Entropy takes a higher value for more random distribution. Take the orientation for example, Mosaics

takes the highest value while hairpin takes the lowest value for the same reason as explained before. Therefore, this feature can discriminate the hairpin pattern from other patterns. f , the ratio of number of intersection points to the number of endpoints ( f ) measures the spatial complex ity of a texture. It takes a large value if capillaries interweave each other. For example, mosaics has the highest f value while punctation takes the lowest. Fur thermore, network pattern also has much higher value than those of hairpin and punctation. Therefore, this feature can discriminate mosaic and network patterns from others. Density ( f ) measures the coarseness (or "neness) of texture in terms of amount of edgels per unit area. Coarse textures have a small number of edgels per unit area while "ne texture have a high number of edgels per unit area. It measures the capillary concentration. For example, network pattern has the highest density while punctation has the lowest density. Therefore, this feature can discriminate the network from punctation patterns. While these features re#ect our subjective perception, much more experimentation and analysis should be done on this subject to analyze the correlation of other proposed features with speci"c characteristics of cervix texture patterns. 2.4. Feature analysis and selection Before proceeding to carry out classi"cation, an analysis of the extracted 24 features was conducted to study the e!ectiveness of each feature. Such an analysis is useful in determining which feature may be eliminated due to redundancy and which may be eliminated due to weak discriminatory power. Motivations for such an analysis include cost savings associated with data acquisition and processing. Feature analysis started with the removal of the correlated features, followed by ranking the remaining features. Redundant features are features whose presence does not contribute very much to the improvement of the classi"cation accuracy. One type of redundant features is features that are linearly correlated with each other. The linearly correlated features can be identi"ed by computing the linear correlation coe$cients between any two features. The identi"ed correlated features are subsequently removed via feature pruning. Criterion used for feature pruning include consistency, invariance, and discriminatory power. Consistency is measured by the within group variance, invariance measures features invariance to transformation, and discriminatory power determines a feature's discriminatory capability as discussed below. For example, angle peak density negatively correlates well with angle contrast, angle contrast feature can be removed since it has lower discriminatory power as shown in Appendix B.


Feature ranking rates the discriminatory power of each feature based on its capability in discriminating all classes. The ratio of between-class variance to the within group variance was used as a criteria. Appendix B shows the result of feature ranking. Analysis of the inter-class and intra-class variance revealed some of the features to be ine!ective, yielding a feature vector of reduced dimensionality. For example, angle kurtosis and skewness are ine!ective measures since they are the measures for unimode distribution while angle distributions contain multiple modes. On the other hand, length kurtosis and skewness are e!ective measures since length distributions are close to unimode. Subsequent to correlation computation and feature ranking, a subset of features can be selected to form a new feature set that will be used for classi"cation. This requires retaining those features having the highest discriminating power and deleting those that are redundant or provide minimum information for class separation. Redundant features can be deleted through removing correlated features. Features providing minimum information for classi"cation can be removed through ranking. We will demonstrate this in the experimental section.

2.5. Classixer design With features extracted, algorithms are needed to take the extracted features as input and output the required class labels. Of many di!erent classi"ers, we chose the minimum distance classi"er. To design the minimum distance classi"er, we compute the mean feature values for each pattern class. The distance to the mean of each class from a given input vector is computed and the vector is assigned to the class for which the distance from the vector to the class mean is the shortest since that class has feature measurements most similar to those of the unknown sample. Euclidean distance is used as a measure of the `nearnessa of a cluster, although other measures of similarity are also possible. To prevent features with large numerical values from dominating the distance calculations, both the feature values and the distances are normalized. This normalization procedure takes into account the spread of the values within a cluster. For example, in the one-dimensional case, for a cluster with a mean k and G a standard deviation p , if ( f!k )"( f!k ) but G p 'p , the sample would be classi"ed as belonging to class 1 since it has a larger variability, where f a query vector. For each of the six classes, the distance from the unknown sample to the class mean is computed. The class to which the distance is minimum is the class to which the unknown sample is assigned. The results of this classi"cation are recorded in a confusion matrix.

1569

3. Experimental evaluation In this section, we present the preliminary results of our studies on the usefulness of the proposed texture features for categorizing a series of typical vascular patterns as shown in Fig. 1. This evaluation starts with a discussion of image data acquisition, followed by textural feature extraction and analysis, classi"er design, and "nally a presentation and analysis of the classi"cation results. 3.1. Image acquisition and feature extraction The experiment started with cervical image acquisition and preparation. The images to be prepared should contain the typical vascular pattern classes characterizing di!erent stages of dysphasia. Typical vascular patterns to be recognized in this project include six vascular pattern classes. They are network capillaries, hairpin capillaries in normal and benign cervical lesions; and two di!erent versions of punctation and mosaic patterns in preinvasive and invasive lesions as shown in Fig. 1. Sources of images include our existing library of colposcopic images, published photographs, and commercial slides. All images came with diagnosis and biopsy results. To characterize each vascular pattern class accurately, 50 images were collected for each vascular pattern class, resulting in a total of 300 images. These images represent six classes. For each acquired image, we identi"ed and marked a rectangular region corresponding to a known vascular pattern. The identi"ed regions of interest were preprocessed. The preprocessed images were subsequently skeletonized and vectorized. The line segments distributions were then constructed, followed by the extraction of the 24 texture features. To perform feature selection, 5 images were randomly selected for each pattern class from the 300 images. Feature selection and analysis on the selected images yields 13 optimal features as shown in Table 1. Detailed results for feature analysis may be found in Appendix B.

Table 1 A subset of 13 optimal features Joint entropy Angle entropy Angle peak ratio Length peak density Angle peak density Length entropy CIntersections/CEnds Info entropy Energy Length median Angle contrast Length median/range

1570


3.2. Classixcation and results To study the classi"cation performance of the proposed technique, we divided the remaining images (270) into two sets: one for testing and one for training. Due to the large feature vector dimension (24) and relatively smaller number of images (45 for each class pattern), the conventional classi"cation technique, which requires dividing the original data set into separate training and testing data sets, cannot generate accurate results since we do not have su$cient data for training. Instead, cross-validation method was used for classi"cation design and evaluation. This method is widely used by investigators in the pattern recognition "eld when the data set does not contain su$cient samples to have separate training and testing data. The decision boundaries (training) are obtained by leaving out some samples in the original data set, the samples which are left out are then classi"ed (tested). The procedure is repeated for all samples in the data set, with new samples left out each time, to obtain the overall accuracy of the classi"cation scheme. The "nal classi"cation error is the average for each left-out. Speci"cally, for each training and testing session, 240 (40 for each class) sample images were used as the training data and the 30 (5 for each class) left-out were used as the testing data. This process iterated 9 times so that every sample image had the chance to be the left out and to be in the training set. The total number of correct

classi"cation over the 9 iterations was used to evaluate the classi"cation performance. For each testing, the minimum distance decision rule was used to classify the left-out image into one of six vascular pattern classes. Two experiments were conducted. First, classi"cation was performed using 24 features. Second, the 13 optimal features de"ned in Table 1 were computed and used for classi"cation. The performance of the classi"er was recorded in the confusion matrix as shown in Tables 2 and 3. The left most column of the confusion matrix is labeled with the actual classes, the top row shows the classi"ed classes. Table 2 shows the results using all 24 features while the confusion matrix in Table 3 shows the resultant output, using the optimal 13 features. In summary, we obtained the best discrimination performance (87.03%) by using all 24 features as shown in Table 2. However, by using only the 13 optimal features resulting from feature ranking, our technique experiences a loss in classi"cation accuracy (80.36%). The loss in accuracy seems to be minimal compared to the computational saving (almost 40%). Further experiments are needed to validate this.

4. Conclusion This paper describes a texture image analysis technique for characterizing and recognizing typical, diagnostically most important, vascular patterns relating to

Table 2 Confusion matrix for classi"cation using all 24 features

Mosaic 1 Mosaic 2 Punctation 1 Punctation 2 Network Hairpin Total

Mosaic 1

Mosaic 2

Punctation 1

Punctation 2

Network

Hairpin

% Correct

38 9 0 0 0 0 *

7 36 0 0 0 0 *

0 0 35 3 0 0 *

0 0 10 36 0 0 *

0 0 0 0 45 0 *

0 0 0 6 0 45 *

84.44 80.00 77.77 80.00 100.00 100.00 87.03

Table 3 Confusion matrix for classi"cation using 13 optimal features

Mosaic 1 Mosaic 2 Punctation 1 Punctation 2 Network Hairpin Total

Mosaic 1

Mosaic 2

Punctation 1

Punctation 2

Network

Hairpin

% Correct

31 9 0 0 0 0 *

6 36 0 0 0 0 *

0 0 32 0 0 0 *

0 0 7 38 10 0 *

8 0 0 0 35 0 *

0 0 6 7 0 45 *

68.88 80.00 71.11 84.44 77.77 100 80.36


cervical lesions. Preliminary experimental study demonstrated the feasibility of the proposed technique in discriminating between cervical texture patterns indicative of di!erent stages of cervical lesions. Study is underway currently to further characterize the performance of the proposed approach with a larger data set. The major contributions of this research include the development of a novel generalized statistical texture analysis technique for accurately characterizing cervical textures and the introduction of a set of textural features that capture speci"c characteristics of the cervical textures as perceived by human. These contributions could potentially lead to a system that can accurately recognize typical vascular patterns indicative of di!erent stages of cervix lesions. Appendix A. Textural features The following equations de"ne textural features. Notations: p(r, c) is the (r, c)th entry in a normalized joint line segment distribution with respect to both length and orientation. p(i) is the ith entry in a marginal line segment distribution with respect to either length or orientation. N is the number of distinct discretization levels with respect to either length or orientation. k and p are the mean and standard deviation of a marginal line segment distribution. A.1. Textural features for marginal density distributions: 1. Peak density ( f ) measures the strength of the local dominant orientation (or length) of a line segment distribution. f "max (p(i)) for i"1, 2 ,2, N. 2. Peak ratio ( f ) measures the relative strength of the second dominant orientation (or lengths of a line segment distribution. p f " , f p "max(p(i)) and p (f for i"1, 2,2, N. 3. Median ( f ) measures the homogeneity of a line seg ment distribution with respect to length or orientation. D p(i)"0.5. G 4. Mean ( f ) measures the global (average) trend of a line segment distribution with respect to length or orientation. , f " p(i) * i. G

1571

5. Median to range ratio ( f ) measures the homogeneity of a line segment distribution with respect to length or orientation. f f " . N 6. Contrast ( f ) measures the local variation or contrast in a distribution around its mean with respect to length or orientation.

i!k , p(i). f " abs p G 7. Entropy ( f ) measures the randomness or homogeneity of a distribution with respect to length or orientation. , f "! p(i) log(p(i)). G 8. Skewness ( f ) characterizes the degree of asymmetry of a univariate distribution around its mean.

1 , i!k f " . N p G 9. Kurtosis ( f ) measures the relative peakness or fatness of a univariate distribution.

1 , i!k \ . f " N p G A.2. Textural features for the joint distribution: 10. Joint entropy ( f ) measures the randomness or homogeneity of a joint distribution. , + f "! p(r, c) log(p(r, c)) P A 11. Informational entropy ( f ) measures the correla tion between two random variables (orientation and length) that are not brought about by the linear correlation coe$cient. f "(1!e\ &VW\D), where H "! p(i)p(j) log[p(i) * p(j)]. VW G H 12. Correlation ( f ) measures the linear dependence between the orientation and length of line segments. p f " J? , p p J ? where p , p , and p represent the covariance of J? J ? length and orientation, marginal variance of length and orientation, respectively. 13. Energy ( f ) measures the uniformity of the entries in the joint distribution. It is the lowest when all entries are equal. f " p(i, j). G H

1572


14. Ratio of the number of intersection points to the number of endpoints ( f ) measures the spatial com plexity of the textures. number of intersection points f " . number of end points 15. Density ( f ) measures the coarseness (or "neness) of a texture in terms of amount of edgels per unit area.

Appendix B. Feature ranking Features

Between G variance/ Within G variance

Joint entropy Angle entropy Angle peak ratio Length peak density Angle peak density Length entropy CIntersections/CEnds Info}entropy Energy Length median Angle contrast Length median/range Length skewness Length peak ratio Length mean Density Length kurtosis Correlation Length contrast Angle median/range Angle kurtosis Angle mean Angle median Angle skewness

125.87 29.22 25.92 16.98 15.06 12.42 10.87 10.78 10.57 9.52 9.37 5.48 5.42 4.67 4.64 4.37 2.84 2.15 1.62 1.07 0.99 0.85 0.84 0.57

References [1] B.L. Craine, E.R. Craine, J.R. Engel, N.T. Wemple, A clinical system for digital imaging colposcopy, Proceedings of SPIE Medical Imaging II, Newport Beach, CA, 1988.

[2] P. Kolstad, A. Sta#, Atlas of Colposcopy, University Park Press, 1977. [3] L. Davis, S. Johns, J.K. Aggarwal, Texture analysis using generalized co-occurrence matrices, IEEE PAMI, V. PAMI-1 3 (1979) 251}259. [4] R.M. Haralick, Statistical and structural approaches to texture, Proceedings of IEEE, 1979. [5] D.C. He, L. Wang, Textural "lters based on the texture spectrum, Pattern Recognition 12 (24) (1991) 1187}1195. [6] R.L. Kashyap, A. Khotanzad, Rotation invariant texture classi"cation using features derived from random "eld model, Proceedings of IEEE Proceedings of conference on Computer Vision and Pattern Recognition, Washington DC, 1983. [7] J. Mao, A.K. Jain, Texture classi"cation and segmentation using multiresolution simultaneous autoregressive models, Pattern Recognition 25 (2) (1992) 173}187. [8] C.M. Wu, Y.-C. Chen, Multi-threshold dimension vector for texture analysis and its application to liver tissue classi"cation, Pattern Recognition 26 (1) (1993) 137}144. [9] A. G. Houston, S. B. Premkumar, Statistical interpretation of texture for medical applications, Proceedings of Biomedical Image Processing and Three Dimensional Microscopy, San Jose, 1991 [10] F. Lachmann, C. Barillot, Brain tissue classi"cation from mri data by means of texture analysis, Proceedings of Medical Imaging VI: Image Processing, Newport Beach, CA, 1992. [11] C. Fortin, W. Ohley, Automatic segmentation of cardiac images: texture mapping, Proceedings of 1991 IEEE 17th Annual Northeast Bioengineering Conference, Hartford, CT, 19. [12] R.M. Haralick, K. Shanmugam, I. Dinstein, Textural features for image classi"cation, IEEE Trans. Systems, Man, Cybernet. SMC-3 (6) (1973) 610}612. [13] J. Chomet, Screening for cervical cancer: a new scope for general practitioners? results of the "rst year of colposcopy in general practice, Br. Med. J. 294 (1987) 1326}1328. [14] B.D. Thackray, A.C. Nelson, Semi-automatic segmentation of vascular network images using a rotating structuring element (rose) with mathematical morphology and dual feature thresholding, IEEE Trans. Med. Imaging 12 (3) (1993) 385}392. [15] S. Sternberg, Biomedical Image Processing, IEEE Comput. Mag. (1973) 22}33. [16] E.L. Hall, R.P. Kruger, A survey of preprocessing and feature extraction techniques for radiographic images, IEEE Trans. Comput. C-20 (9) (1971) 1032}1044. [17] T.Y. Zhang, C.Y. Suen, A fast parallel algorithm for thinning digital patterns, Commun. ACM 27 (3) (1984) 236}239.

About the Author*QIANG JI received a MS degree in electrical engineering from the University of Arizona in 1993 and his Ph.D. degree in electrical engineering from the University of Washington in 1998. His areas of research include computer vision, image processing, pattern recognition, and robotics. Dr. Ji is currently an assistant Professor at the department of computer science at University of Nevada at Reno. Between May, 1993 and May 1995, he was a research engineer with Western Research Company, Tucson, Arizona, where he served as a principle investigator on several NIH funded research projects to develop computer vision and pattern recognition algorithms for biomedical applications. In summer 1995, he was a visiting technical sta! with the Robotics Institute, Carnegie Mellon University, where he developed computer vision algorithms for industrial inspection. From 1995 to 1998, he worked as a research associate at the Intelligent Systems Laboratory (ISL) at the University of Washington, involving in a Boeing-funded research project consisting of developing computer vision techniques for 3D geometric tolerancing of manufactured parts from their images.


1573

He has published numerous papers in referred journals and conferences in these areas. His research has been funded by local and federal government agencies and by private companies including Boeing and HONDA. He currently serves as PI for a project funded by HONDA involving developing a computer vision system for monitoring driver's vigilance level. About the Author*Dr. JOHN ENGEL obtained his Ph.D. in physics from the University of California, Los Angles. He is now a senior engineer with Etec Systems, Inc. He has, since 1980, been involved with software engineering, scienti"c programming, algorithm development and image processing on micro-, mini-, and mainframe computer systems. He is a member of the Pattern Recognition Society, SPIE, IEEE, and the American Physical Society and the American Geophysical Union. About the Author*Dr. ERIC CRAINE received his Ph.D. in astrophysics from the Ohio State University. He has served as PI or co-PI on over 45 Federally and privately funded research programs. He has considerable experience in the management and consist of all phases of a diversity of research projects, and has an extensive bibliography of scienti"c books, papers, and monographs. He has directed a large University electronic imaging project, NASA supported projects, and has been intimately involved in the design and manufacture of the world's "rst all digital colposcopic systems.


Unsupervised segmentation using a self-organizing map and a noise model estimation in sonar imagery K.C. Yao *, M. Mignotte , C. Collet , P. Galerne , G. Burel Laboratoire GTS (Groupe de Traitement du Signal), Ecole Navale BP 600, 29 240 Brest Naval, France Universite& de Bretagne Occidentale, LEST-UMR CNRS 6616 6 av. Le Gorgeu, BP 809, 29 285 Brest, France Received 9 January 1998; received in revised form 11 June 1999; accepted 11 June 1999

Abstract This work deals with unsupervised sonar image segmentation. We present a new estimation and segmentation procedure on images provided by a high-resolution sonar. The sonar image is segmented into two kinds of regions: shadow (corresponding to a lack of acoustic reverberation behind each object lying on the seabed) and reverberation (due to the re#ection of acoustic wave on the seabed and on the objects). The unsupervised contextual method we propose is de"ned as a two-step process. Firstly, the iterative conditional estimation is used for the estimation step in order to estimate the noise model parameters and to accurately obtain the proportion of each class in the maximum likelihood sense. Then, the learning of a Kohonen self-organizing map (SOM) is performed directly on the input image to approximate the discriminating functions, i.e. the contextual distribution function of the grey levels. Secondly, the previously estimated proportion, the contextual information and the Kohonen SOM, after learning, are then used in the segmentation step in order to classify each pixel on the input image. This technique has been successfully applied to real sonar images, and is compatible with an automatic processing of massive amounts of data. 2000 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Kohonen self-organizing map; Segmentation; Parameter estimation; sonar imagery; Markov random "eld

1. Introduction In high-resolution sonar imagery, three kinds of regions can be visualized: echo, shadow and sea-bottom reverberation. The echo information is caused by the re#ection of the acoustic wave from the object while the shadow zone corresponds to a lack of acoustic reverberation behind this object. The remaining information is called the sea-bottom reverberation area. On the pictures provided by a classi"cation sonar, the echo features are generally less discriminant than the shadow shape for the classi"cation sonar of object lying on the sea#oor. For this reason, detection and classi"cation of an object

* Corresponding author. Tel.: 00-33-298-234-018; fax: 00-33298-233-857. E-mail address: [email protected] (K.C. Yao).

located on the sea#oor (as wrecks, rocks, man-made objects, and so on2) are generally based on the extraction and the identi"cation of its associated cast shadow [1]. Thus, before any classi"cation step, one must segment the sonar image between shadow areas and reverberation areas. In fact, the sea-bottom reverberation and the echo are considered as a single class. Unfortunately, sonar images contain speckle noise [2] which a!ects any simple segmentation scheme such as a maximum likelihood (ML) segmentation. In this simple case, each pixel is classi"ed only from its associated grey level intensity. In order to face speckle noise and to obtain an accurate segmentation map, a solution consists in taking into account the contextual information, i.e. class of the neighborhood pixels. This can be done using Markov random "eld (MRF) models [3] and this is why Markovian assumption has been proposed in sonar imagery [4]. In this global bayesian method, pixels are classi"ed


1576

K.C. Yao et al. / Pattern Recognition 33 (2000) 1575}1584

using the whole information contained in the observed image simultaneously. Nevertheless, simple spatial MRF model have a limited ability to describe properties on large scale, and may not su$cient to ensure the regularization process of the set of labels when the sonar image contains high speckle noise. Such a model can be improved by using a large spatial neighborhood for each pixel [5], or a causal scale and spatial neighborhood [6] but this rapidly increases the complexity of segmentation algorithms and the parameter estimation procedure required to make this segmentation unsupervised. Besides, the segmentation and the estimation procedure with such a priori model requires a lot of computing time. Moreover, the uses of such a global method does not allow to take into account the noise correlation on the sonar image [7]. An alternate approach adopted here, uses a local method, i.e. takes into account the grey levels of the neighborhood pixels. In this scheme, each pixel is classi"ed from information contained in its neighborhood. This method allowing to take into account the noise correlation is divided in two main steps: the model parameter estimation [8] and the segmentation algorithm which is fed with the previously estimated parameters. In this paper, we adopt for the parameter estimation step an iterative method called iterative conditional estimation (ICE) [9] in order to estimate, in the ML sense, the noise model parameters and specially the proportion of each class (shadow and reverberation). Followed by the training of a competitive neural network as a Kohonen self-organizing map (SOM) [10] in order to approximate the discriminating function (i.e. the contextual distribution function of the grey level). For the segmentation step, we develop a contextual segmentation algorithm exploiting e$ciently the previously estimated parameters, the input sonar image, and the topology of the resulting Kohonen SOM. This paper is organized as follows. In Section 2, we detail the parameter estimation step based on the ICE procedure in Section 2.1, and the training step of the SOM in Section 2.2. Section 3 presents the segmentation step. Experimental results both on real scenes and synthetic sonar images are presented in subsection 3.3, where we compare the results obtained with the proposed scheme, a ML segmentation and a classical monoscale Markovian segmentation. Then a conclusion is drawn in Section 4.

2. Estimation step 2.1. Iterative conditional estimation 2.1.1. Introduction We consider a couple of random "elds Z"(X, >) with >"+> , s3S, the "eld of observations located on a latQ

tice S of N sites s, and X"+X , s3S, the label "eld. Each Q > takes its value in " "+0,2,255, and each X in Q M@Q Q +e "shadow, e "reverberation,. The distribution of (X,>) is de"ned "rstly by P (x), the distribution of X as6 sumed to be stationary and Gibbsian (i.e. Markovian) in this estimation step, and secondly by the site-wise likelihoods P (y /x ). In this work, these likelihoods 7Q6Q Q Q depend on the class label x . The observation > is called Q the incomplete data whereas Z stands for the complete data. In this step, we estimate the parameter vector ' which W de"nes P (y/x) by using the iterative method of estima76 tion called iterative conditional estimation (ICE) [9]. This method requires to "nd an estimator, namely ' (X, >) for completely observed data. When X is unobW servable, the iterative ICE procedure de"nes ' I> as W conditional expectations of ') given >"y, computed W according to the current value ' I . This is the best W approximation of ' in terms of the mean square error W [9]. By denoting E , the conditional expectation using I ' I , this iterative procedure is de"ned as follows: W E Initialize the noise model parameters to ' . W E ' I> is computed from ' I and >"y by W W ' I> "E [') />"y]. W I W

(1)

The computation of this expectation is impossible in practice, but we can approach Eq. (1) owing to the law of large numbers by 1 ' I> " [') (x , y)#2#') (x , y)], W L W n W

(2)

where x , with i"1 ,2, n are realizations of X accordG ing to the posterior distribution P . Finally, we 67WVW I W can use the ICE procedure for our application because we get: E An estimator ') (X, >) of the complete data: we use W a maximum likelihood (ML) estimator for the noise model parameter estimation (see Section 2.1.2.) E An initial value ' not too far from the optimal W parameters (see Section 2.1.3). E A way of simulating realizations of X according to the posterior distribution P (y/x) by using the Gibbs 76 sampler [11]. For the prior model, we adopt an 8connexity spatial neighborhood (see Fig. 1) in which b , b , b , b represent the a priori potential associated to the horizontal, vertical, right and left diagonal binary cliques, respectively. In our application, we want to favour homogeneous regions. Then, we de"ne potential functions associated to the two-site cliques of the form b "k[1!d(x , x )], QR Q R

(3)


Fig. 1. Second-order neighborhood and two-site associated cliques.

where b "b , b , b or b according to the type of QR neighboring pair 1s, t2, k is a predetermined positive constant and d(.) is the Kronecker function. 2.1.2. Estimation of the noise model parameters for the complete data The Gaussian law N(k, p), is an appropriate degradation model to describe the luminance y within shadow regions (essentially due to electronic noise) [12]. The most natural choice of the estimator ') (x"e , y) is the W empirical mean and the empirical variance. If N pixels are located in the shadow areas, we have 1 y, (4) k( " Q +* N QZ1VQC 1 p " (y !k( ). (5) +* N !1 Q +* QZ1VQC In order to take into account the speckle noise phenomenon [2] in the reverberation areas, we model the conditional density function of the reverberation class by a shifted Rayleigh law R(min, a) [12]

y !min (y !min) P(y /x "e )" Q exp ! Q . Q Q a a

(6)

The maximum value of the log-likelihood function is used to determine a ML estimator of the complete data. If y( is the minimum grey level in the reverberation

areas and N the number of pixels located within this region, we obtain for ') (x"e , y) the following results W [12]: 1 (y !min ), (7) p " Q +* +* 2N QZ1VQC min Ky( !1. (8) +*

In the two cases, the proportion n of the kth class is I given by the empirical frequency N I n( " with k3+0, 1,. I N #N

(9)

2.1.3. Initialisation The initial parameter values have a signi"cant impact on the rapidity of the covergence of the ICE procedure

1577

and the quality of the "nal estimates. In our application, we use the following method: the initial parameters of the noise model ' are determined by applying a small W non-overlapping sliding window over the image and calculating the sample mean, variance and minimum grey-level estimates. Each estimation calculated over the sliding window gives a `samplea x , a three component G vector. These samples +x ,2, x , are then clustered + into two classes +e , e , using the K-means clustering procedure [13]. This algorithm uses a similarity measure based on the Euclidean distance between samples. A criterion is based on the minimization of the related cost function de"ned by ) (10) J" "x !c ", J G G VJZ!G where the second sum is over all samples in the ith cluster and c is the center of this cluster. It is easy to show that G for a given set of samples and class assignments, J is minimized by choosing c to be the sample mean of the ith G cluster. Moreover, when c is a sample mean, J is miniG mized by assigning x to the class of the cluster with the H nearest mean. A number of other criteria are given in Ref. [13]. The complete algorithm is outlined below: (1) Choose K initial clusters c ,2, c . These could be arbitrarily chosen, but areusually)de"ned by c "x (1)i)K). (11) G G (2) At the kth step, assign the sample x (1)l)M) to J cluster i if #x !c I #(#x !c I # (∀jOi). (12) J J G H In fact, we reassign every sample to the cluster with the nearest mean. In the case of equality, we assign x arbitrary to i or j. J (3) Let c I denote the ith cluster after Step 2. Determine G new clusters by 1 c I> " x, (13) J N G G VJZA I G where N represents the number of samples in c I . G G Thus, the new cluster position is the mean of the samples in the previous one. (4) Repeat until convergence is achieved, say c I> "c I , ∀i. G G Although it is possible to "nd pathological cases where convergence never occurs [13], the algorithm does converge in all tested examples. The rapidity of convergence depends on the number K, the choice of initial cluster centers and the order in which the samples are considered. In our application K"2. Fig. 2(a) represents a sonar image and the result of the K-means clustering algorithm is reported in Fig. 2(b).

1578


Fig. 2. K-means clustering procedure: (a) sonar picture involving an object and a rock shadow (b) segmentation according to the ML criterion with parameter estimation given by the Kmeans algorithm.

On the one hand, a small size window increases the accuracy of the segmentation and then the precision of the distribution mixture estimation. On the other hand, it decreases the number of pixels with which x 's are comJ puted and may increase the misclassi"cation error. In our application, good results are obtained with a 6;6 pixels window. The ML estimation is then used over the Kmeans segmentation in order to "nd ' . W 2.1.4. Parameter estimation procedure for the incomplete data We can use the following algorithm to estimate the noise model parameters. Let us recall that this method takes into account the diversity of the laws in the distribution mixture estimation. E Parameter initialization: The K-means algorithm is used. Let us denote by ' the obtained result. W E ICE procedure. ' I> is computed from ' I in the W W following way: E 䡩 Using the Gibbs sampler, n realizations x ,2, x are simulated according to the poste L rior distribution with parameter vector ' I , and W with P Q Q(y /x "e ) a Gaussian law for shadow 7 6 Q Q area, P Q Q(y /x "e ) a shifted Rayleigh law for 7 6 Q Q reverberation area. E 䡩 For each x with i"1 ,2, n, the parameter vecG tor ' is estimated with the ML estimator on each W class: E 䡩 ' I> is obtained from ' (x , y) with (1)i)n) W W G by using Eq. (2). If the sequence ' I becomes steady, the ICE procedure W is ended and one proceeds the segmentation using the estimated parameters. We can use all these estimated parameters in order to get a complete unsupervised Markovian segmentation (see Section 3.2) or only use the proportion of each class in the Kohonen SOM-based unsupervised segmentation described in Section 2.2.

Fig. 3. Image histogram of the picture reported in Fig. 2 with the estimated Gaussian and Rayleigh laws.

Table 1 Estimated parameters on the pictures reported in Fig. 2(a). n stands for the proportion of the two classes within the sonar image. k and p are the Gaussian parameters (shadow area). Min and a are the Rayleigh law parameters (reverberation area). ' represents the initial parameter estimates and the "nal W estimates are denoted ') W Initialisation of K-means procedure ' DGL?J

WQF?BMU ' DGL?J

WQC?\@MRRMK

0.04 L 0.96 L

36 I 39

55 N 1061 ?

ICE procedure ' DGL?J

WQF?BMU ' DGL?J

WQC?\@MRRMK

0.03 L 0.97 L

32 I 39

17 N 1591 ?

We calibrate the weight of the `stochastica aspect of the ICE by choosing n. When n increases, the `stochastica aspect of the algorithm decreases. The choice of a small value for n (n"1 in our application) can increase its e$ciency [14]. Fig. 3 represents the mixture of distributions of the sonar image reported in Fig. 2(a). The obtained results are given in Table 1. The quality of the estimations is di$cult to appreciate in absence of real values. We can roughly perform such evaluation by comparing the image histogram with the probability density mixture corresponding to the estimated parameters. Fig. 3(a) shows the resulting mixture solution in graphical form. The two dashed curves in the "gures represent the individual components P G(y/e ) 76 K with 0)m)K. The histogram is quite close to the mixture densities based on the estimated parameters, and a segmentation with these estimates can be done as shown in the following section.


2.2. Self-organizing map 2.2.1. Introduction Researches on neurobiology have shown that centers of diverses activities as thought, speech, vision, hearing, lie in speci"c areas of the cortex and these areas are ordered to preserve the topological relations between informations while performing a dimensionality reduction of the representation space. Such organization led Kohonen to develop the SOM algorithm [10]. This kind of competitive neural network is composed of one or two dimensional array of processing elements or neurons in the input space. All these neurons receive the same inputs from external world. Learning is accomplished by iterative application of unlabeled input data. As training process, the neurons evolve in the input space in order to approximate the distribution function of the input vectors. After this step, large-dimensional input vectors are, in a sense, projected down on the one or two-dimensional map in a way that maintains the natural order of the input data. This dimensional reduction could allow us to visualize and to use easily, on a one or two-dimensional array, important relationships among the data that might go unnoticed in a high-dimensional space. The model of SOM used in our application is a onedimensional array of n nodes. To each neuron N , G a weight vector w "(w , w ,2, w )31N is associated. G G G GN During learning procedure, an input vector x31N randomly selected among vectors of the training set, is connected to all neurons in parallel. The input x is compared with all the neurons in the Euclidean distance sense via variable scalar weight w . At the kth step, we assign the GH vector x to the winning or leader neuron N if: J #x!w I #"min #x!w I #. J G G

(14)

All the neurons within a certain neighborhood around the leader participate in the weight-update process. Considering random initial values for w (0)i)n), this G learning process can be described by the following iterative procedure:

1579

the convergence, it is necessary that H I P0 when kP¹, JG where ¹ is the total number of steps of the process [10]. Therefore, for the "rst step, a I should start with a value that is close to unity, thereafter decreasing monotonically [10]. To achieve this task, we use

k . ¹

a I "a 1!

(17)

Moreover, as learning proceeds, the size of the neighborhood should be diminished until it encompasses only a single unit. So, we applied for the width of the Kernel the monotonically decreasing function: p I "p

p 2\ I2\ p

(18)

The ordering of the map occurs during the "rst steps, while the remaining steps are only needed for the "ne adjustment of the weight values. 2.2.2. Iterative learning step The learning process is performed directly on the real image to be segmented. An input vector is "lled with the grey levels of the pixels contained in a 3;3 pixels window sliding over the image (cf. Fig. 4). Therefore, each neuron has nine weights allowing to locate it in the input space. At each step, the location of the window in the image is randomly chosen and the weights are modi"ed according to Eq. (15). Experiments have shown that this training strategy provides as good results as an ordered image scanning process while spending less processing time. p has a signi"cant impact on the quality of the convergence. We have to start with a fairly large value to globally order the map. The initial value p of p can be half the length of the network. During learning, p has to decrease monotonically until it reaches a small value. Experiments have shown that p "0.1 is a good 2\ choice and provides the minimum quantization error

w I> "w I #H I (x I !w I ). (15) G G JG G The lateral interactions among topographically close elements are modeled by the application of a neighborhood function or a smoothing Kernel de"ned over the winning neuron [10]. This Kernel can be written in terms of the Gaussian function

d(l, i) , H I "a I exp ! 2(p I ) JG

(16)

where d(l, i)"#l!i# is the distance between the node l and i in the array, a I (t) is the learning-rate factor and p I de"nes the width of the Kernel at the iteration k. For

Fig. 4. Model of the SOM used for the segmentation. An 3;3 sliding window is used to feed the SOM.

1580


de"ned by 1 , E " #x !w #, OS?LR N G J G

(19)

where the summation is over all the N windows of the image and w is the weight vector associated to the leader J neuron of the input vector x after learning step. G 3. Segmentation step 3.1. SOM segmentation The classi"cation task consists in running the sliding window over the image. For each location of the window, the corresponding input vector x is compared with all the neurons using Eq. (14). The winning neuron, the one which leads to the smallest distance, gives the class of the pixel located in the center of the window. However, before any classi"cation task, we have to calibrate the map in order to associate the label shadow or reverberation to each neuron. Assuming that the input vector x "(0 ,2, 0) should represent a window setting on a perfect shadow area, it is very useful to de"ne the distance graph representing the Euclidean distance in the nine-dimensional space between the point x and all the neurons. Such a graph is given in Fig. 5 and Fig. 6 respectively before and after learning for a hundred-neuron network. Both these "gures show that the maximal distance between two successive cells is widely smaller after learning than before. We can deduce that, after learning, neurons that are topologically close in the array are close

in the input space too. As a matter of fact, neurons that are physical neighbors should respond to a similar input vectors. The calibration of the map uses this topological property and the portion n estimated in Section 2.1 of the pixels labelled as shadow in the image. This process can be summarized as follows: (1) Initially, we a!ect the class reverberation to all neurons. (2) We seek the most evident prototype of the shadow class. This neuron is the winning unit according to Eq. (14) when inputting the vectors x . Then, we a!ect it to the shadow class. (3) We a!ect the shadow class to pixels for which the leader neuron belongs to the shadow class. We can deduce the intermediate shadow portion n provided GLR by the resulting image. (4) If n is smaller than n , we have to select an addiGLR tional prototype of a shadow class among neurons of the reverberation class. According to the topological preserving properties of the map, this additional neuron should be a direct neighbor of an already shadow labelled neuron. Among both the possible neighbors, we take the one which has the smallest Euclidean distance with the point x . Go to 3. (5) If n is larger than n , we stop the process. GLR Experiments have shown that E is a monotonically OS?LR decreasing function of the number of steps and reaches an asymptotic value for large value of ¹. One hundred times the number of network units seems to be a reasonable compromise solution between speed and quality of learning. In our application, 100 neurons have been chosen for the network.

Fig. 5. The distance graph between the 100 neurons of the SOM before learning.


1581

Fig. 6. The distance graph between neurons obtained after learning.

3.2. Markovian segmentation The segmentation of sonar images in two classes can be viewed as a statistical labelling problem according to a global Bayesian formulation in which the posterior distribution P (x/y) J exp [!;(x, y)] has to be 67 maximized [3]. In our case, the corresponding posterior energy ;(x, y) to be minimized is ;(x, y)" t (x , y ) Q Q Q QZ1

GFHFI ; (x,y)

# b [1!d(x , x )] QR Q R 1 2 QR

GFFHFFI ; (x,y)

(20)

where ; denotes the adequacy between observations and labels (( (x , y )"In [P (x /y )]) and ; exQ Q Q 6Q7Q Q Q presses the energy of the priori model. In order to minimize this energy function, we use a deterministic relaxation technique called ICM algorithm [3}5,15]. 3.3. Results on real scenes We compare the segmentation performance of the proposed SOM-based algorithm described in Section 3.1 with a ML segmentation and a classical Markovian segmentation using a deterministic relaxation technique such as the ICM [3]. All the segmentation results exploit the parameter estimation step presented in Section 2.1. This estimation step is used to estimate both the noise model parameters for the ML segmentation, the Mar-

Fig. 7. (a) A real sonar image involving a sandy sea #oor with the cast shadow of a tyre. Two-class segmentation results obtained with this image using: (b) ML segmentation, (c) Markovian segmentation with a deterministic relaxation technique as ICM, (d) the SOM based segmentation result (see Table 2 for the estimated parameters). The SOM segmentation exhibits a good robustness against the speckle noise (which induces false small shadow areas compared to the others approaches).

kovian segmentation and the proportion of the shadow class for the SOM segmentation. Figs. 6}9 show the segmentation results obtained with the di!erent methods. Example of the noise model

1582


Fig. 8. A real sonar image involving an object and a rock shadows. Two-class segmentation results obtained with (a), (b) ML segmentation, (c) Markovian segmentation with ICM technique, (d) SOM based segmentation method (see Table 3 for the estimated parameters). The ML and ICM do not permit to totally eliminate the speckle noise e!ect (creating shadow mislabelled isolated pixels).

Table 2 Estimated parameters on the picture reported in Fig. 6 ICE procedure ' DGL?J

WQF?BMU ' DGL?J

WQC?\@MRRMK

0.02 L 0.98 L

36 I 46

ICE procedure 0.03 L 0.97 L

25 I 35

Table 4 Estimated parameters on the picture reported in Fig. 8 ICE procedure ' DGL?J

WQF?BMU ' DGL?J

WPCTCP@CP?RGML

0.03 L 0.97 L

34 I 42

39 N 1412 ?

85 N 1878 ?

Table 3 Estimated parameters on the picture reported in Fig. 7

' DGL?J

WQF?BMU ' DGL?J

WQC?\@MRRMK

Fig. 9. (a) A synthetic sonar image of a sphere lying on the seabed. Segmentation results obtained with: (b) ML segmentation, (c) Markovian ICM technique, (d) SOM-based segmentation method.

32 N 1430 ?

parameters ' obtained with our scheme are given in W Tables 2}4. Experiments indicate that the SOM segmentation requires less computation than the Markovian segmenta-

tion (30 for the SOM estimation-segmentation, whereas roughly 100 are required for unsupervised scale causal Markovian modelization [16] on IBM 43P-200 MHz workstation). Besides, the ICM algorithm do not permit to decrease the number of false alarm (wrong detections) due to the speckle noise e!ect. The SOM segmentation performs better, exhibits a good robustness versus speckle noise (false alarms have been eliminated), and allows us to preserve the shadow shapes of little rocks. Manufactured objects or rock shadows are better segmented with our method than with the others (cf. Figs. 7}9) and their shape are close to the result we expected. The cast shadow of a manufactured object (a cylinder) reported in Fig. 10 has a geometric shape (contrary to the cast shadow of the rock) that will be discriminant for the classi"cation step.


1583

References

Fig. 10. Real sonar images of a cylindrical object (a) and of ridges of sand (c). Their corresponding SOM-based segmentation results are depicted, respectively, in (b) and (d).

4. Conclusion We have described an unsupervised segmentation procedure based on a parameter estimation step (which o!ers an appropriate estimation of the noise model) and a segmentation step well adapted for sonar image segmentation problem. The estimation step takes into account the diversity of the laws in the distribution mixture of a sonar image and can be used with the Kohonen SOM-based segmentation in order to solve the di$cult problem of unsupervised sonar image segmentation. This scheme is computationally simple and appears as an interesting alternative to existing complex hierarchical Markovian methods. This method has been validated on several real sonar images demonstrating the e$ciency and robustness of this scheme.

Acknowledgements The authors thank GESMA (Groupe d'Etude Sous Marine de l' Atlantique, Brest France) for having provided real sonar images and REGION BRETAGNE for partial "nancial support of this work.

[1] P. Galerne, K. Yao, G. Burel, Objects classi"cation using neural network in sonar imagery, Proceedings of the SPIE on New Image Processing Techniques and Applications, Vol. 3101, Munchen, June 1997, pp. 306}314. [2] J.W. Goodman, Some fundamental properties of speckle, J. Opt. Soc. Am. 66 (11) (1976) 1145}1150. [3] J. Besag, Spatial interaction and the statistical analysis of lattice systems, J. Roy. Statist. Soc. 36 (1974) 192}236. [4] C. Collet, P. Thourel, P. PeH rez, P. Bouthemy, Hierarchical MRF modeling for sonar picture segmentation, Proceedings of the Third IEEE International Conference on Image Processing, Vol. 3, Lausanne, September 1996, pp. 979}982. [5] P. Thourel, C. Collet, P. Bouthemy, P. PeH rez. Multiresolution analysis and MRF modelling applied to the segmentation of shadows in sonar pictures, Proceedings of the Second Asian Conference on Computer Vision, Vol. 2, Singapore, December 1996, pp. 81}85. [6] M. Mignotte, C. Collet, P. PeH rez, P. Bouthemy, Unsupervised hierarchical Markovian segmentation of sonar images. Proceedings of the ICIP, Vol. 3, Santa Barbara, CA, USA, October 1997. [7] F. Schmitt, L. Bonnaud, C. Collet, Contrast control for sonar pictures, Signal and Image Processing, SPIE'96* Technical Conference on Application of Digital Image Processing XIX Vol. 2847, August 1996, pp. 70}82. [8] P. Masson, W. Pieczynski, S.e.m. algorithm and unsupervised statistical segmentation of satellite images, IEEE Trans. GeH osci. Remote Sensing 3 (1993) 618}633. [9] F. Salzenstein, W. Pieczynski, Unsupervised Bayesian segmentation using hidden Markovian "elds, Proceedings of the ICASSP, May 1995, pp. 2411}2414. [10] T. Kohonen, Self Organizing Maps, Springer, Berlin, 1995. [11] S. Geman, D. Geman, Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images, IEEE Trans. Pattern Anal. Mach. Intell. 6 (6) (1984) 721}741. [12] F. Schmitt, M. Mignotte, C. Collet, P. Thourel, Estimation of noise parameters on sonar images, SPIE Statistical and Stochastic Methods for Image processing, Vol. 2823, Denver, 4}5 August 1996, pp. 1}12. [13] S. Banks, Signal Processing Image Processing and Pattern Recognition, Prentice-Hall, Englewood Cli!s, NJ, 1990. [14] B. Braathen, P. Masson, W. Pieczynski, Global and local methods of unsupervised Bayesian segmentation of images, Machine Graphics Vision 1 (1993) 39}52. [15] M. Mignotte, C. Collet, P. PeH rez, P. Bouthemy, Unsupervised Markovian segmentation of sonar images, Proceedings of the ICASSP, Vol. 4, Munchen, May 1997, pp. 2781}2785. [16] M. Mignotte, C. Collet, P. Perez, P. Bouthemy, Sonar image segmentation using an unsupervised hierarchical mrf model, IEEE Trans. Image Process. March 6, 1999, in press.

1584


About the Author*PASCAL GALERNE was born in Brest on August 20, 1970. He received the D.E.A (Postgraduate degree) in Electronic, Image and Signal Processing from the Bretagne Occidentale University in 1994. He is actually a Ph. D. student in Sonar image processing at the Signal Research Center of the French Naval academy. His current research interests include neural networks, image synthesis, image segmentation, classi"cation algorithms. About the Author*MAX MIGNOTTE received the Master of Sciences (Electronics and TeH leH communications) (1992) and the D.E.A (Postgraduate degree) in Digital Signal, Image and Speech Processing of the Institut National Polytechnique of Grenoble (1993). He is actually a Ph.D. student in computer science (Digital signal and image processing) at the Signal Research Center of the French Naval academy. His current research interests include image segmentation, parameters estimation, hierarchical Markovian models, statistical model and genetic algorithm. About the Author*KOFFI-CLED MENT YAO received the Ph.D. degree (1990) in Optical Signal Processing and Computer Science from the Louis Pasteur University of Strasbourg, France. From 1991 to 1992, he worked as Assistant Professor in the Department of Optics and Communication Systems at the Ecole Nationale SupeH rieure des TeH leH communications de Bretagne, France. Currently, He is Mam( tre de ConfeH rences of the Bretagne Ocidentale University and He teaches deterministic and statistical signal processing in Ecole Navale, the French Naval Academy. His present research interests include pattern recognition, neural networks, adaptive "ltering and higher order statistics for blind signals separation in Underwater Acoustics. About the Author*CHRISTOPHE COLLET was born in Paris in 1966. He studied in the Universities of Paris-Sud-Orsay and Toulon in the "elds of Electronic and Signal Processing. He received the Ph.D. in Image Processing in 1992. He is currently Mam( tre de confeH rences at the Bretagne Occidentale University and manages the Signal Research Center of the French Naval Academy. His research interests include nonlinear statistical estimation, Markov random "elds, texture analysis, 2D pattern recognition and hierarchical analysis. About the Author*GILLES BUREL was born in 1964. He obtained the diploma form the Ecole Superieure d'Electricite in 1988. Then, he received the Ph.D. degree in 1991 and the Habilitation a Diriger des Recherches in 1996 from the Bretagne Occidentale University. From 1988 until 1996 he was with Thomson CSF, Rennes, France, a leading company in professional electronics, where he worked in Image Processing and Digital Transmissions. He his now professor at the Bretagne Occidentale University. He is the author of 19 patents and 40 articles in the areas of Image Processing and Digital Transmissions.

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

No title

Recommend Documents