IMAGE SEGMENTATION Edited by PeiGee Peter Ho
Image Segmentation Edited by PeiGee Peter Ho
Published by InTech Janeza Trdine 9, 51000 Rijeka, Croatia Copyright © 2011 InTech All chapters are Open Access articles distributed under the Creative Commons Non Commercial Share Alike Attribution 3.0 license, which permits to copy, distribute, transmit, and adapt the work in any medium, so long as the original work is properly cited. After this work has been published by InTech, authors have the right to republish it, in whole or part, in any publication of which they are the author, and to make other personal use of the work. Any republication, referencing or personal use of the work must explicitly identify the original source. Statements and opinions expressed in the chapters are these of the individual contributors and not necessarily those of the editors or publisher. No responsibility is accepted for the accuracy of information contained in the published articles. The publisher assumes no responsibility for any damage or injury to persons or property arising out of the use of any materials, instructions, methods or ideas contained in the book. Publishing Process Manager Ivana Lorkovic Technical Editor Teodora Smiljanic Cover Designer Martina Sirotic Image Copyright Robert Gubbins, 2010. Used under license from Shutterstock.com First published March, 2011 Printed in India A free online edition of this book is available at www.intechopen.com Additional hard copies can be obtained from
[email protected] Image Segmentation, Edited by PeiGee Peter Ho p. cm. ISBN 9789533072289
free online editions of InTech Books and Journals can be found at www.intechopen.com
Contents Preface Part 1 Chapter 1
Chapter 2
Part 2
IX
Survey of Image Segmentation Algorithms
1
A Survey of Image Segmentation by the Classical Method and Resonance Algorithm Fengzhi Dai, Masanori Sugisaka and Baolong Zhang
3
A Review of Algorithms for Segmentation of Retinal Image Data Using Optical Coherence Tomography 15 Delia Cabrera DeBuc Image Segmentation Methods
55
Chapter 3
Image Segmentation through Clustering Based on Natural Computing Techniques 57 Jose Alfredo F. Costa and Jackson G. de Souza
Chapter 4
Segmentation with Learning Automata 83 Erik Cuevas, Daniel Zaldivar and Marco PérezCisneros
Chapter 5
Surround Suppression and Recurrent Interactions V1V2 for Natural Scene Boundary Detection 99 Francisco J. DíazPernas, Míriam AntónRodríguez, Isabel de la TorreDíez, Mario MartínezZarzuela, David GonzálezOrtega, Daniel BotoGiralda and J. Fernando DíezHiguera
Chapter 6
Using Emergence Phenomenon in Meaningful Image Segmentation for Contentbased Image Retrieval 119 Sagarmay Deb
Chapter 7
Dual Active Contour Models for Medical Image Segmentation 129 Gilson Giraldi, Paulo Rodrigues, Jasjit Suri and Sameer Singh
VI
Contents
Chapter 8
Image Segmentation Using Maximum Spanning Tree on Affinity Matrix 153 Qiang He and CheeHung Henry Chu
Chapter 9
Image Segmentation by Autoregressive Time Series Model PeiGee Peter Ho
161
Chapter 10
Evolutionarybased Image Segmentation Methods Licheng Jiao
Chapter 11
Segmentation of Handwritten Document Images into Text Lines 225 Vassilis Katsouros and Vassilis Papavassiliou
Chapter 12
IR Image Segmentation by Combining Genetic Algorithm and Multiscale Edge Detection 241 Li Zhaohui and Chen Ming
Chapter 13
Segmentation of Remotely Sensed Imagery: Moving from Sharp Objects to Fuzzy Regions Ivan Lizarazo and Paul Elsner
Chapter 14
Part 3
179
249
Colorbased Texture Image Segmentation for Vehicle Detection 273 Ricardo MejíaIñigo, María E. BarillaPérez and Héctor A. MontesVenegas Image Segmentation Applications
291
Chapter 15
An Enhanced Level Set Algorithm for Wrist Bone Segmentation 293 Donatello Conte, Pasquale Foggia, Francesco Tufano and Mario Vento
Chapter 16
Mineral Grain Boundary Detection With Image Processing Method: From Edge Detection Operation To Level Set Technique 309 Bibo Lu and Weixing Wang
Chapter 17
Multiscale Segmentation Techniques for Textile Images Xiqun Lu
Chapter 18
JSEG Algorithm and Statistical ANN Image Segmentation Techniques for Natural Scenes Luciano Cássio Lulio, Mário Luiz Tronco and Arthur José Vieira Porto
343
327
Contents
Chapter 19
Image Segmentation of ZiehlNeelsen Sputum Slide Images for Tubercle Bacilli Detection 365 R. A. A. Raof, M. Y. Mashor, R. B. Ahmad and S. S. M. Noor
Chapter 20
Image Segmentation Based on a TwoDimensional Histogram 379 Masmoudi Lhoussaine, Zennouhi Rachid and Mohamed EL Ansari
Chapter 21
Segmentation Methods for Biomedical Images Roberto Rodríguez Morales
Chapter 22
Algorithm Selection Based on a Region Similarity Metric for Intracellular Image Segmentation 419 Satoko Takemoto and Hideo Yokota
Chapter 23
Extraction of Estuarine/Coastal Environmental Bodies from Satellite Data through Image Segmentation Techniques 435 Ana Teodoro and Hernâni Gonçalves
Chapter 24
Rock Fracture Image Segmentation Algorithms Weixing Wang
Chapter 25
Image Segmentation Integrating Generative and Discriminative Methods 489 Yuee Wu and Houqin Bian
Chapter 26
PixonBased Image Segmentation 495 Hamid Hassanpour, Hadi Yousefian and Amin Zehtabian
Part 4 Chapter 27
Hardware Implementation
389
459
517
Hardware Implementation of a RealTime Image Segmentation Circuit based on Fuzzy Logic for Edge Detection Application 519 Angel Barriga
VII
Preface It was estimated that 80% of the information received by human being is visual. Image processing is evolving fast and continually. During the past 10 years, there has been a significant increase in knowledgebased image analysis, image recognition as well as image segmentation. To study a specific object in an image, its boundary can be highlighted by an image segmentation procedure. The objective of the image segmentation is to simplify the representation of pictures into meaningful information by partitioning into image regions. Image segmentation is a technique to locate certain objects or boundaries within an image. There are many algorithms and techniques have been developed to solve image segmentation problems, though, none of the method is a general solution. Among the best, these are neural networks segmentation, onedimensional signal segmentation, multiscale segmentation, model based segmentation, graphic partitioning, region growing and Kmean clustering segmentation methods. This book brings together many diﬀerent aspects of the current research on several fields associated to digital image segmentation. Four main parts have been defined and allowed gathering the 27 chapters around the following topics: Survey of Image Segmentation Algorithms, Image Segmentation methods, Image Segmentation Applications and Hardware Implementation of Image Segmentation. The book starts with a first set of chapters which addresses most recent general approaches in the image segmentation fields. One can find discussion about various new trends on image segmentation techniques. The evolutionary image segmentation algorithms and methods are presented next. Recently the most used approach in segmentation of medical images is the level set which is based on optimization mathematics. A segmentation of the image plane is computed by locally minimizing an appropriate energy functional E(C) by evolving the contour C of the region to be segmented starting from an initial contour. In general, this method may use either an explicit (parametric) or implicit representation of the contours. The active contour (also called snakes) image segmentation scheme is very popular in medical surgery these days. The basic idea of the dual snakes is to reject local minima by using two contours: one which contracts from outside the target and one which expands from inside. Such proposal makes possible to reduce the sensitivity to initialization through the comparison between the two
X
Preface
contours energy and positions. The newly developed combined Autoregressive time series imaging modeling with either region growing or Support Vector Machine as classifiers are detailed in one of the book chapter. In order to solve the image segmentation thresholding problem , a new way of using an optimization algorithm based on learning automata for multilevel thresholding is proposed in one chapter of this book. The pixon concept based on a set of disjoint regions with constant shapes and variable sizes was introduced in 1993 to decreases the computational time on image segmentation. A few innovative methods to improve the eﬃciency are also included. Nevertheless, the image segmentation applications that demand constrained response times, the specific hardware implementation is required. In this book, the chapter title “Hardware Implementation of a RealTime Image Segmentation Circuit based on Fuzzy Logic for Edge Detection Application” provides the hardware approach for image segmentation. Last, but not the least, we would like to thank all contributors to this book for their research, Intech publisher CEO Dr. Aleksandar Lazinica and Ms. Ivana Lorkovic for their publishing eﬀort. I am sure that you will enjoy reading this book and get many helpful ideas and overviews on your own study.
PeiGee Peter Ho, Ph.D. DSP Algorithm development group Naval Undersea Warfare Center at Newport RI, USA
Part 1 Survey of Image Segmentation Algorithms
1 A Survey of Image Segmentation by the Classical Method and Resonance Algorithm Fengzhi Dai1, Masanori Sugisaka2 and Baolong Zhang3 1Tianjin
University of Science and Technology Bunri University, ALife Robotics Co., Ltd 3Tianjin University of Science and Technology, GianTOPtics Technology Co., Ltd 1,3China 2Japan 2Nippon
1. Introduction Computer vision and recognition plays more important role on intelligent control (Chen & Hwang, 1998). For an intelligent system, it is necessary to acquire the information of the external world by sensors, to recognize its position and the surrounding situation. Camera is one of the most important sensors for computer vision. That is to say, the intelligent system endeavours to find out what is in an image taken by the camera: traffic signs, obstacles or guidelines. For image analysis, image segmentation is needed, which means to partition an image into several regions that have homogeneous texture feature. This process is usually realized by the regionbased, boundary based or edgebased method (Castleman, 1998). And from the viewpoint of clustering, it is divided into supervised and unsupervised texture segmentation. Since before segmentation, the intelligent control system seldom knows the feature of the image, e.g. which type and how many types of textures exist in an image, thus the unsupervised segmentation algorithm is always needed, although it is more difficult than the supervised method (Dai, Zhao & Zhao, 2007). In this chapter, the classical method (Agui & Nagao, 2000) and the resonance theory (Heins & Tauritz, 1995; He & Chen, 2000) are proposed respectively for image segmentation. The classical method is simple but practicable, which will be introduced in section 2. But for some situations, it is not suitable for complex image segmentation (e.g., the gradient variations of intensity in an image). We know that human vision can recognize the same texture that has gradient variations of intensity. And many image segmentation methods are proposed based on the change of intensity (Nakamura & Ogasawara, 1999; Deguchi & Takahashi, 1999). But they always fail to handle the wideranged gradations in intensity (Jähne, 1995). It is usually difficult to give a suitable threshold for pixelbased image processing methods to deal with this gradation. Resonance algorithm is an unsupervised method to generate the region (or feature space) from similar pixels (or feature vectors) in an image. It tolerates gradual changes of texture to some extent for image segmentation. The purpose of section 3 is to propose the resonancetheorybased method for image segmentation, which means that the same texture in an image will be resonated into one region by seed pixels. This method assumes that the
4
Image Segmentation
differences of feature between adjacent pixels of the same texture must be within a tolerable range. Thus the selection of feature distance to segment them is important. In this chapter, Section 2 introduces the classical method. Section 3 gives the resonance theory and the algorithm for image segmentation. Section 4 is the conclusion.
2. The classical method 2.1 The principle This section introduces the classical method for image segmentation, and the example is to recognize the simple characters (the digit and letter). Recognizing the whole word needs to segment it into the characters 0 to 9, a to z, or A to Z firstly. The principle (Agui, Nakajima & Kimi, 1990; Tanaka, 1989) and a result of segmentation is shown in Fig. 1. This section is simple but is the original meaning of image segmentation, which is useful when the reader wants to understand image segmentation clearly.
Fig. 1. The principle of segmentation Fig. 1. gives the example of the digit segmentation. In the image there are some digits but as the image style, not the character. If the computer or intelligent system wants to recognize these digits autonomously, first image segmentation is needed to search and locate each digit, and then recognize them. The method is that it calculates the values of the pixels by searching the Xaxis and Yaxis to find the neighboring distribution of the digits (looking for the break of the character). For each line (row and column) where there is a character, it appears that the values of pixels is not 0. If there are blanks, it means that they are the space between one character and another. Thus it gives the position of the break of the characters. In Fig. 1, there are four rows and four columns of digits. 2.2 Image segmentation If there are several targets in an image, image segmentation is necessary: locating and isolating the targets in an image and then identifying them. Once isolated, the targets can be measured and classified. The general image segmentation algorithm (Agui, Nakajima & Kimi, 1990) is shown in Fig. 2a. And Fig. 2b is the result of segmentation for the word ‘R05’. Table 1 gives the segmented result. The steps for character segmentation are divided into 2 steps:
5
A Survey of Image Segmentation by the Classical Method and Resonance Algorithm
line[0].starty
line[0].endy line[1].starty line[1].endy
(a) Principle
(b) Segmentation result
Fig. 2. The classical image segmentation algorithm Result for Fig. 2a There are two rows. row0 has two targets, row1 has two targets. (top, bottom, left, right) is
Result for Fig. 2b There is One row. row0 has three target.
row0, col0: F (line[0].starty, line[0].endy, line[0][0].startx, line[0][0].endx) row1, col0: 7 (line[1].starty, line[1].endy, line[1][0].startx, line[1][0].endx)
row0, col0: R (line[0].starty, line[0].endy, line[0][0].startx, line[0][0].endx)
row0, col1: L (line[0].starty, line[0].endy, line[0][1].startx, line[0][1].endx) row1, col1: 6 (line[1].starty, line[1].endy, line[1][1].startx, line[1][1].endx)
(top, bottom, left, right) is row0, col1: 0 (line[0].starty, line[0].endy, line[0][1].startx, line[0][1].endx)
row0, col2: 5 (line[0].starty, line[0].endy, line[0][2].startx, line[0][2].endx)
Table 1. Segmented result of Fig. 2a, b 1.
2.
Search the screen from top to bottom linebyline horizontally to find the start and the end line that contains how many rows of characters: the variables are line[0].starty, line[0].endy, line[1].starty and line[1].endy in Fig. 2a, which means that the image contains two rows of characters. The horizontal location of the first row of characters is from line[0].starty to line[0].endy, the second is from line[1].starty to line[1].endy, respectively. For each row of characters, search the image from left to right vertically to locate and calculate how many characters in each row. For example, in Fig. 2a for characters within
6
Image Segmentation
line[0].starty and line[0].endy, there are two letters in this row. They are the letter ‘F’ from line[0][0].startx to line[0][0].endx, and ‘L’ from line[0][1].startx to line[0][1].endx. Thus each character (letter or digit) can be segmented. 2.3 An experimental result Fig. 3 shows an experiment (including the image sampling, processing, segmentation and recognition), which is to segment the image and then recognize each character one by one.
Part 1
Part 3
Part 2 Fig. 3. An example of the classical image segmentation In Fig. 3, part 1 is the image sampling and segmentation. The topleft window in part 1 gives the image sampled directly from the camera. After image processing, the segmented result is shown in the topright window (Dai, Shimogaki & Fujihara, 2008). Part 2 in Fig. 3 shows some parameters for image processing. These parameters can be adjusted based on environment in real time, so that the result of image processing is good enough to segmentation. Part 3 is the process for character recognition. The feature vector of each segmented character is extracted and then matches it to the templates. By the template matching method (Snyder & Qi, 2004), each digit or letter can be recognized respectively, and then the whole meaning of the word can be understood by combination of the meaning of each character. 2.4 Problem of the classical method From the above explanation, we see that the classical method is really simple but practicable. It can segment the characters in an image by locating and calculating how many rows of characters, and how many characters in each row.
A Survey of Image Segmentation by the Classical Method and Resonance Algorithm
7
But this method cannot segment the characters in Fig. 4 correctly. Since the classical method cannot correctly divide the image into two rows of character (the numbers “127” and “305”) by any horizontal line (the dashed lines), the wrong result is appeared: all the characters in the image are recognized into one row. Of cause the correct number cannot be gotten.
line[0].starty
line[0].endy Fig. 4. An example for wrong segmentation by the classical method
3. Resonance theory and algorithm 3.1 Resonance theory The resonance theory (He & Chen, 2000) can be expressed in Fig. 5. Assumes that in a scene of space, each point has a mass m. These points are not isolated from each other but interconnected by interforce. In Fig. 5, two points (a) and (b) are given and the distance between them is d.
d m (a)
m (b)
Fig. 5. Resonance theory If an externally sinusoidal force F adds to the point (a), the movement of the point (a) can be expressed by
8
Image Segmentation
.. . m y + b y + ky = F cos wt
(1)
b, k are the parameters of the system and point, the values of which are not important to the following explanation. y and y means the first and secondorder derivatives with respect to the time t for the motive distance y. The frequency of force F is w. Its stable solution is y(t ) = A sin( wt + θ )
(2)
A = F / m2 ( w 2 − w02 )2 + b 2 w 2
(3)
where
is the oscillating amplitude. θ is the initial phase, w0 =
k / m is called the natural frequency.
From Eq. (3), if the frequency of the external force F (to the point (a)) is equal to the natural frequency w0, the amplitude A has the maximal value, and this case is called the resonance between the external force and the point (a). Since we have assumed that the points are not isolated, the motion of the point (a) will result in the spreading of its effect to other points that are around it. This is the resonance among the points, which is the theory for image segmentation proposed in section 3.2 to 3.4. From the analysis of the resonance mechanics (He & Chen, 2000), the amplitude of point (b) with the distance of d from (a) has the feature of A( distance = d ) ∝ A(point a) / d
(4)
From the above analysis, if we assume that point (a) is the source to resonate and another point (b) can be largely affected, then the difference between the external frequency and the natural frequency, and the distance d between those two points should be small sufficiently. These two conditions can be satisfied by: 1. A threshold is set to ensure the difference of external frequency from the natural frequency is small enough. 2. The resonance algorithm is used within the adjacent points (in image segmentation) to ensure the distance between them is small enough. 3.2 Resonance algorithm By the spreading of the resonance, the adjacent points that have the same or similar feature (e.g. texture in an image) are clustered into one region. It seems like the general region growing algorithm (Castleman, 1998; Jähne, 1995), but they are essentially different. Region growing method partitions an image by the threshold directly: in an image, defining the maximum and minimum thresholds for each region to segment them. If an image contains complex color (or gray level) gradation, the selection of threshold is difficult. Differently, the resonance algorithm emphasizes the similarity between the adjacent points, not the threshold for global usage. And the resonance can be spread from point to point. Thus the problem caused by gradation in intensity can be solved. Only the sudden change of features between adjacent points can be regarded as the boundary of different regions. Define Pδ ( a , b ) as the path between the point a and b (a and b need not to be adjacent) under the threshold δ (the value to estimate the difference of features between two adjacent points). If there are sequences of adjacent points connecting between a and b, all of which
A Survey of Image Segmentation by the Classical Method and Resonance Algorithm
9
have the same features or have different features between the adjacent points but below δ, they form the connected path by the determination of the path between point a and b. And if s is a point in an image, all points xi that satisfies Pδ (s , xi ) will form the region Rδ (s ) . The point s is always called as seed (Dai, Zhao & Zhao, 2007; He & Chen, 2000). Remember that the region Rδ (s ) is not centered by the point s, but refers to a region that all points in which have the same features to s, or the difference of features between adjacent points below δ. Now the principle of the resonance algorithm for image segmentation is clear (in image processing, the point is called pixel): From one or some seed pixels, the adjacent pixels that belong to the same region under δ are clustered until all the pixels are searched. From the above definitions and the transfer of resonance, we see that the selection of seeds does not influence the segmentation result in an image. Fig. 6 gives the expression of the resonant process. In Fig. 6, a is the seed pixel, from which to resonate all the space in the image. x1 a
x2
xi
xn
… y1
z1 z2
b
y2 c
d
Fig. 6. Resonance process By comparing with the difference of feature values between the point a and the adjacent pixels, all the pixels in Pδ ( a , xi ) are labeled to belong to one region, e.g. Rδ ( a) (Regin1) = {a, xi (i = 1,…,n)}. Next, from xn to the pixel b, the difference between them is larger than δ, b is defined as a pixel belonging to a new region, e.g. Rδ (b ) (Region2) = {b, yj (j = 1,…,m)}, all the pixels in which are Pδ (b , y j ) . The same is to Rδ (c ) (Region3) = {c, zk (k = 1,…,l)} that are Pδ (c , zk ) until all the pixels are labeled. The selection of seeds will not affect the result of segmentation. From Fig. 6, although the pixel d has the same feature to Region1, since it is far from Region1 and is segmented by other regions, by the resonance theory, the pixel d cannot be resonated by any pixel in Region1. Thus a new region creates from the point d. That is to say, the number of the segmented regions in an image does not absolutely equal to the number of real different texture types. In fact, this is not the weakness of the algorithm. Just as this case, it clearly shows that between Region1 and the pixel d, there must exist some other textures. Thus by the resonance theory, the resonance algorithm is determined by three important elements: 1. One or several seed points, 2. The features to determine the difference between points, 3. The parameter of the threshold δ.
10
Image Segmentation
And the steps of the resonance algorithm for image segmentation are: 1. Initialization. Rδ (0) . 2. Segmentation. Find new region Rδ (i ) . 3. Termination after having searched all pixels. 4.
M
Result. R = ∪ Rδ (i ) . i =0
Thus we see that this method is to find a harmonious threshold within adjacent points, not to estimate a global value. Fig. 7 gives the flow chart of the resonance algorithm for image segmentation. Start
Initialization and feature extraction
Have labelled all pixels
Yes
No
End
Select adjacent unlabeled pixel
distance >δ
Segmentation is finished
Yes
Y
Belongs to a new region
No Belongs to current region Fig. 7. Resonance algorithm 3.3 Selection of elements Selection of the three elements of the resonance algorithm will be considered in this section. First, we see that from the resonance theory and the transfer of resonance, the initial place of the seed does not influence the segmentation result in an image. Thus the seed can be selected randomly. The threshold δ is important in the resonance algorithm. Too large will include surplus regions into one while too small reject some points that belong to the same region. δ should
A Survey of Image Segmentation by the Classical Method and Resonance Algorithm
11
be larger than the distance in one region and less than the difference between two different regions. But different images, or different regions in one image, may have different thresholds. We propose an automatic selection method for δ. Since δ is used to partition the different regions in an image, it is rather a range of values (determine the maximum and minimum values) than a fixed value to ensure the points have the same or similar features in one region. If δ for different regions in an image are selected well, then all the regions are segmented stably. That is to say, when the correct δ varies by a small value, the change of regions in an image is not distinct. Other word is that if the selection of δ is incorrect, a small change of it may vary the area of region greatly. This is the influence of δ for the region segmentation, and comparatively, it can be used as criterion to estimate δ. Since the suitable threshold should be selected to ensure greater than the intraregion feature difference and less than the interregion feature differences (Castleman, 1998; He & Chen, 2000), from the initial seeds, the resonance begins from the current region to extend to other regions with the rise of δ. Fig. 8 is an example. In a unit square, three different regions are in it: the black, the gray and 1 1 1 white region. The area of black is Sblack = × = = 0.25 , the area of gray is Sgray = 2 2 4 3 3 5 3 3 7 × − Sblack = = 0.3125 , and the white Swhite = 1 − × = = 0.4375 . 4 4 16 4 4 16
Area of regions 0.25 0.5 0.75
1
Seed in topleft Seed in bottomright
0 (a)
4
8 Value of δ
12
(b)
Fig. 8. δ  area curve Fig.8b gives the δarea curves of Fig.8a for the seed point selected in the topleft and bottomright corner respectively, from which we see that the selection of seeds will affect the parameter δ, but not affect the segmentation result (He & Chen, 2000). Another element for resonance is the selection of features to estimate whether the distance between two adjacent points is below δ or not. In fact, it can be chosen by any features of the image, and the different selection of features results to different threshold δ. In this chapter, the eightconnectivity is used to connect the pixels of the object.
12
Image Segmentation
3.4 Image segmentation When there are multiobjects in an image, image segmentation is necessary: locating and isolating the objects from the image and then identifying them. Once isolated, the objects can be measured and classified. The correct segmentation should be that it divides the image S into several independent regions {S1, S2, …, Sn}, each region represents one kind of textures (Tanaka, 1989).
n (1) S = ∪ Si i=1 (2) Si ∩ S j = Φ (3) Pr (Si ) = 1
for all i ≠ j
(5)
for all i
Pr(x) is the probability of the existence of x and Φ is the null set. Fig. 9a (a strong light is given behind the right hand of the black rectangle) is always used to compare the results among different image processing algorithms, which is the scene of setting the experiment. Fig. 9b is the original experimental image extracted from Fig. 9a.
(a)
(b)
(c)
(d)
Fig. 9. Compared experiment By using the resonance algorithm proposed in the chapter, the segmentation result is given in Fig. 9c, in which the effect of light is removed, so that the background is in one region. A compared result is given in Fig. 9d by the reinforcement histogram algorithm (Castleman, 1998), which shows that the background is difficult to be clustered into one region. 3.5 The experiment In this section, compared to the conventional method, the natural image in real environment will be applied to analyze the resonance algorithm for image segmentation. Fig. 10 shows the source image and the segmentation result (Dai, Fujihara & Sugisaka, 2008). In the original image of Fig. 10a, sky and trees are two main regions, while the color of the sky is varied gradually by clouds. Fig. 10b is the result after segmented by the proposed resonance algorithm. We adopt the gray level as the feature and the seed pixel is selected from the topleft corner of the image. The image is divided into three parts. The sky is separated into part1 and 3 by a trunk of tree in the image. Part2 is the region of trees. The influence of clouds is greatly eliminated because the resonance algorithm can handle gradual changes of intensity.
A Survey of Image Segmentation by the Classical Method and Resonance Algorithm
13
Fig. 10. Image segmentation: (a) color image, (b) by the resonance algorithm, (c) by the histogram analysis Also if we do not satisfy that the sky is separated, the part1 and 3 can be combined by the later processing easily, because the features of them are the same. The compared result is given by Fig. 10c, which is produced by the general histogram analysis method. It shows that the influence of the clouds cannot be ignored.
4. Conclusion In this chapter, first the classical method for image segmentation is introduced. It is suitable for digit or letter segmentation and the program to realize the method is easy to be composed. But it still has some limitations. In the second part of the chapter, the resonance theory and algorithm, and the three important elements of resonance are introduced. The unsupervised resonance algorithm is proposed for complex image segmentation, which has the feature to eliminate the influence of gradual changes of texture in intensity to some extent. The resonance algorithm for image segmentation is to search the same or similar texture pixels (among the adjacent pixels) so as to cluster them into one region: the resonance is spread among all the pixels within the image. The compared result is also given and shows that the resonance algorithm emphasizes the similarity among the adjacent pixels rather than the global threshold values, and the segmentation result is satisfied. But there are some problems still existed to be solved: 1. How to select the parameter δ more correctly, 2. How to improve the program to fasten the algorithm.
14
Image Segmentation
5. Acknowledgment This work is sponsored in part by the Scientific Research Foundation of Tianjin University of Science and Technology (China) for the Introduced Talented Scholars, No. 20100408.
6. References Agui, T.; Nakajima, M. & Kimi, J. (1990). Image Processing with C: PC9801 Series MSDOS (in Japanese). Shokodo press, ISBN 4785630760, Japan Agui, T. & Nagao, T. (2000). Introduction to image processing using programming language C. Shokodo press (in Japanese), ISBN 4785631244, Japan Castleman, K. (1998). Digital Image Processing, Original edition published by Prentice Hall, Inc., a Simon & Schuster Company, Press of Tsinghua University, ISBN 7302028281/TN.88, China Chen, P. & Hwang Y. (1998). SANDROS: A dynamic graph search algorithm for motion planning, IEEE Transactions on Robotics and Automation, Vol. 14, No. 3, pp. 390403, ISSN 1042296X, NEW YORK Dai, F.; Fujihara, Y. & Sugisaka, M. (2008). Image Segmentation by Resonance Algorithm, Proceedings of the International Conference on Instrumentation, Control and Information Technology, pp. 20182022, ISBN 9784907764296, Tokyo, Japan, August 2008, SICE, Tokyo Dai, F.; Shimogaki, D. & Fujihara, Y. (2008). Research on Digit Recognition by Image Processing for Autonomous Mobile Robot. Proceedings of the Thirteenth International Symposium on Artificial Life and Robotics, pp. 827830, ISBN 9784990288020, Oita, Japan, January 31  February 2, 2008, Shubundo Insatsu Co. Ltd., Japan Dai, F.; Zhao, J. & Zhao, H. (2007). The Application of Resonance Algorithm for Image Segmentation, Journal of Applied Mathematics and Computation, Vol. 194/2, pp. 453459, December 2007, ISSN 00963003 Deguchi, K. & Takahashi, I. (1999). Imagebased simultaneous control of robot and target object motion by directimageinterpretation, Proceedings of the 1999 IEEE/RSJ International Conference of Intelligent Robot and System, Vol. 1, pp. 375380, 99CH36289, Kyongju, Korea, October 1999 He, H. & Chen, Y. (2000). Unsupervised texture segmentation using resonance algorithm for natural scenes, Pattern Recognition Letters, Vol. 21, 2000, pp. 741757 Heins, L. & Tauritz, D. (1995). Adaptive Resonance Theory (ART): An Introduction, Department of Computer Science, Leiden University, IR9535, May/June, 1995. “http://www.wi.leidenuniv.nl/art” Jähne, B. (1995). Digital Image Processing  Concepts, Algorithms, and Scientific Applications. The Third Edition, SpringerVerlag, ISBN 3540592989, Germany Nakamura, T. & Ogasawara, T. (1999). Online visual learning method for color image segmentation and object tracking, Proceedings of the 1999 IEEE/RSJ International Conference on Intelligent Robots and Systems, Vol. 1, pp. 222228, 99CH36289, Kyongju, Korea, October 1999 Snyder, W. & Qi, H. (2004). Machine Vision, the Press of Syndicate of the University of Cambridge, ISBN 052183046X, UK Tanaka, H. (1989). Image Processing and Application Technology (in Japanese), Kogyo Chosakai Publishing Co., Ltd., ISBN 4769310722, Japan
2 A Review of Algorithms for Segmentation of Retinal Image Data Using Optical Coherence Tomography Delia Cabrera DeBuc, PhD
Bascom Palmer Eye Institute, University of Miami Miller School of Medicine United States of America 1. Introduction In the context of biomedical imaging analysis and computerassisted diagnosis, segmentation analysis is an intense field of research and development. The most difficult part of medical image analysis is the automated localization and delineation of structures of interest. Automated data evaluation is one way of enhancing the clinical utility of measurements. In particular, medical image segmentation extracts meaningful information and facilitate the display of this information in a clinically relevant way. A crucial role for automated information extraction in medical imaging usually involves the segmentation of regions of the image in order to quantify volumes and areas of interest of biological tissues for further diagnosis and localization of pathologies. Optical coherence tomography (OCT) is a powerful imaging modality used to image various aspects of biological tissues, such as structural information, blood flow, elastic parameters, change of polarization states and molecular content (Huang et al., 1991). OCT uses the principle of low coherence interferometry to generate two or three dimensional imaging of biological samples by obtaining highresolution crosssectional backscattering profiles. A variety of successful algorithms for computeraided diagnosis by means of OCT image analysis are presented in the literature, but robust use in clinical practice is still a major challenge for ongoing research in OCT image analysis. There are, therefore, efforts being made to improve clinical decision making based on automated analysis of OCT data. Particularly, in ophthalmology, efforts have been made to characterize clinically important features, such as damage to the fovea and optic nerve, automatically. The transfer of image analysis models from algorithmic development into clinical application is currently the major bottleneck due to the complexity of the overall process. For example, the process to establish an application for OCT medical image analysis requires difficult and complex tasks that should considers the following actions: 1) to define the OCT image data structures representing relevant biomedical features and the algorithms determining a valid example for given image values, 2) to select meaningful values for all technical parameters of the image data structures and algorithms and, as a result, to configure such a method to operate on specific OCT clinical data, 3) to run the algorithm with the selected parameters to find the individual model instance that best explains the input image and 4) to validate the procedure to ensure a trustworthy result from an automated segmentation algorithm even if a gold standard is unavailable.
16
Image Segmentation
This chapter is intended to give a broad, but by no means complete, overview over common segmentation methods encountered in OCT retinal image processing. To do this, some algorithms that are representative for each class in some detail are described. In addition an understanding of the original derivation and motivation of each algorithm is provided, instead of merely stating how each method functions. This is of high importance in order to get an idea where and under what circumstances a method can function and when one can expect an algorithm to fail. To briefly motivate why one should consider different segmentation algorithms, consider the example of a 2D OCT image in Fig. 1. Simple thresholding can be used to mark the locations of the inner and outer boundaries of the retina in this OCT image. But some boundary sections are not properly identified due to poor contrast or low resolution, making it impossible to identify the exact extent of the retina in this image (see Fig.1A). Since these boundaries are found by a threshold procedure, their estimated locations could be sensitive to relative differences in reflectance between the outer and deeper retinal structures. By choosing a different segmentation algorithm (see Fig. 1B), identification of the retinal boundaries can be improved (Cabrera Fernández et al., 2005b). All segmentation methods that have been proposed in the literature aim at improving retinal image segmentation in this or other aspects. The causes for problems such as the ones in Fig. 1 can by manifold, many times being inherent to the respective image acquisition method itself.
Fig. 1. Segmentation results showing the performance of the Stratus OCT custom builtin algorithm compared to the results using a custom algorithm. A) Macular scan obtained from a healthy eye. Note the misidentification of the outer boundary of the retina outlined in white. B) Results obtained for the same eye using a custom algorithm. Note that the custom algorithm was able to correctly detect the outer boundary of the retina. The chapter is organized as follows. The next section continues with an outline of current retinal imaging modalities. Section 3 explains the physical principles and technical details of how OCT works. The interpretation of the OCT image along with the current and future technology development of OCT systems is also presented in this section. Section 4 provides the necessary background about medical image segmentation approaches. A review of algorithms for segmentation of retinal image data using OCT is presented in Section 5. All
A Review of Algorithms for Segmentation of Retinal Image Data Using Optical Coherence Tomography
17
published (within the author's awareness) papers related to retinal image segmentation are gathered into this single compilation. Section 6 offers some concluding remarks.
2. Current retinal imaging modalities Millions of people worldwide live with retinal disease and the accompanying threat of severe vision loss or blindness. During the last few years, the retinal research field has undergone a dramatic change in terms of diagnostic tools and therapies that have resulted in substantial benefits for patients suffering from retinal disease. Traditionally the retina has been observed either directly via an ophthalmoscope or similar optical devices such as the fundus camera. The field of ophthalmology was revolutionized in 1851 with the invention of the ophthalmoscope by Hermann von Helmholtz (von Helmholtz, 1851) as for the first time detailed examinations of the interior of the eye could be made in living patients. The ophthalmoscope and later the fundus camera remained the primary methods of ocular examination into the 1960’s, and they are standard tools still effective and in use today, although they are not without limitations, and both require trained users to operate and make diagnoses. With advances in medical technology, more powerful techniques were introduced. In 1961 fluorescein angiography was developed by Novotny and Alvis, a procedure in which sodium fluorescein is injected into a vein, and under filtered light the sodium fluorescein within the blood fluoresces, glowing brightly and providing easily observed patterns of blood flow within the eye (Novotny & Alvis, 1961). This allows the arteries, capillaries and veins to be easily identified and photographed, and from this, large amounts of information concerning the health or otherwise of the circulatory system can be determined. During the 1990’s the indocyanine green dye angiography technique was developed; similarly to the flourescein angiography a dye is injected into the bloodstream, however the indocyanine green dye glows in the infrared section of the spectrum. The indocyanine green dye approach only came into widespread use when digital cameras sensitive into the infrared became commonly available, and it complements fluorescein angiography by highlighting different aspects of the vasculature of the eye. In particular it enhances the structure of the choroid, which is the layer of blood vessels beneath the retina. These two techniques can be used together to gain a more thorough understanding of the structure and pathologies affecting an eye. They can illustrate patterns of blood flow, haemorraging and obstructions within the vascular system, but, like the ophthalmoscope, both require trained medical staff to perform the procedure, and a clinical environment where the images can be taken and analyzed. In addition to these methods for observing the vasculature of the eye there are a range of other, more advanced, methods of mapping structures and changes within the eye, including ultrasound, OCT and laserbased blood flowmeters in development and in use. All of these can be used to scan the eye and make observations and diagnoses on the eye and circulatory system. Specifically, the introduction of OCT imaging in daily routine have resulted in some of the central changes to retinal disease understanding and management. Figure 2 shows the operational range of the OCT technology compared to standard imaging. OCT is a rapidly emerging medical imaging technology that has applications in many clinical specialties. OCT uses retroreflected light to provide micronresolution, crosssectional scans of biological tissues (Hee et al., 1995; Huang et al., 1991; Izatt et al., 1994a).
18
Image Segmentation
Fig. 2. OCT vs. standard imaging. The first micronresolution OCT system for imaging human retina in vivo was introduced in 1991 (Huang et al., 1991). In ophthalmology; OCT is a powerful medical imaging technology because it enables visualization of the cross sectional structure of the retina and anterior eye with higher resolutions than any other noninvasive imaging modality (Huang et al., 1991). The depth resolution of OCT is extremely fine, typically on the order of 0.01mm or 0.4 thousandth of an inch. An OCT image represents a crosssectional, micron scale picture of the optical reflectance properties of the tissue (Huang et al., 1991). This image can either be used to qualitatively assess tissue features and pathologies or to objectively make quantitative measurements. While this is just a brief introduction to some of the diagnostic tools available to obtain retinal images, to draw diagnoses from these images requires specialist training, and to adequately extract and track the retinal damage from the images often takes extensive image processing and analysis. Once automatic image analysis is possible, those at risk of numerous diseases and problems of the retinal tissue can be rapidly identified and referred for further treatment. The development of this methodology would also allow automated tracking of the progress of such health problems as diabetic retinopathy, and track changes in the eyes as the subject ages. This would have numerous health benefits, including providing an early prediction of retinal diseases.
3. Optical coherence tomography background The clinical potential of OCT technology in ophthalmology was originally recognized in the early 1990s. OCT is an extension of optical coherence domain reflectometry to imaging in two or three dimensions (Brezinski et al., 1996). This imaging technique generates a crosssectional image by recording axial reflectance profiles while the transverse position of the optical beam on the sample is scanned. Thus, the longitudinal location of tissue structures are determined by measuring the timeofflight delays of light backscattered from these structures. The optical delays are measured by low coherence interferometry. Light reflected from deeper layers has a longer propagation delay than light reflected from more superficial layers. Conventional or time domain OCT (TDOCT) is based on the principle of low coherence interferometry which is a powerful tool to section a transparent object. Low coherence
A Review of Algorithms for Segmentation of Retinal Image Data Using Optical Coherence Tomography
19
means that the system employs a wide range of wavelengths. The most straightforward and currently the most common interferometer for OCT is a simple Michelson interferometer (see Fig 3) (Michelson & Morley, 1887). A lowcoherence source illuminates the interferometer. The light is split by a 50/50 beamsplitter into a sample and a reference path. Light retroreflected from the reference and the sample is recombined at the beamsplitter and half is collected by a photodetector in the detection arm of the interferometer. Half of the light is returned towards the source, where it is lost. In addition, the reference arm light is typically attenuated by orders of magnitude in order to improve signal to noise ratio.
Fig. 3. Schematic drawing of the principle of OCT emphasizing how it is essentially a Michelson intereferometer. The outgoing light paths are solid lines, while reflected light is drawn as dashes lines. The axial resolution of an OCT image depends on the coherence length which is a fundamental property of the light source, whereas transverse resolution for OCT imaging is determined by focused spot size, as in microscopy. By rapidly varying the reference arm mirror and synchronously recording the magnitude of the resulting interference signal, a single axial profile or Ascan is obtained which is a graph of the optical reflectivity versus distance in the eye. A sequence of such Ascans is obtained by scanning the probe beam across the entire retina which forms a Bscan tomogram. As a result, a crosssectional view of the structure similar to a histology section is obtained. OCT can be used for retinal imaging and anterior segment imaging. The OCT for ophthalmic examination is similar to a slit lamp for anterior segment imaging and a fundus camera for retinal imaging. The instrumentation includes a video display for operator viewing of the anterior segment or fundus while obtaining the OCT images and a simultaneous computer display of the tomograms. Images are stored via computer for the diagnostic record (Puliafito, 1996). 3.1 Interpreting OCT images The OCT signal from a particular tissue layer is a combination of its reflectivity and the absorption and scattering properties of the overlying tissue layers. Strong reflections occur at the boundaries between two materials of different refractive indices and from a tissue that has a high scattering coefficient along with a disposition to scatter light in the perfectly backward direction (Huang et al., 1991; Puliafito, 1996 ). Thus, an OCT image is a map of the reflectivity of the sample. In most tissues, main sources of reflection are collagen fiber bundles, cell walls, and cell nuclei. Dark areas on the image represent homogeneous material with low reflectivity, such as air or clear fluids. The imaging light is attenuated in the sample, so there is an
20
Image Segmentation
exponential decrease in the intensity of the image with depth. Blood attenuates the signal faster than collagenous tissues, fat and fluids attenuate the signal the least. In OCT images, the signal strength is represented in false color. High backscatter appears redorange and low backscatter appears blueblack (see Fig. 4). Thus, tissues with different reflectivity are displayed in different colors. It is important to note that OCT image contrast arises from intrinsic differences in tissue optical properties. Thus, coloring of different structures represents different optical properties in false color image and it is not necessarily different tissue pathology (see Fig. 4). The exact relationship between the histology of the tissue and the OCT map is still under investigation. Relative high reflectivity layers correspond to areas of horizontal retinal elements such as the nerve fiber layer at the retinal surface or deeper plexiform layers and a single layer of retinal pigment epithelium (RPE) and choroid. Relative low reflectivity layers correspond to the nuclear layers and a single layer of photoreceptor inner and outer segments. Warm colors (red to white) represent areas of relative high reflectivity, while cold colors (blue to black) represent areas of relative low reflectivity. In the retina, the vitreoretinal interface is demarcated by the reflections from the surface of the retina. The retinal pigment epithelium (RPE) and choriocapillaris layer (ChCap) is visualized as a highly reflective red layer and represents the posterior boundary of the retina. Below the choriocapillaris weakly scattered light returns from the choroid and sclera because of attenuation of the signal after passing through the neurosensory retina, RPE, and ChCap. The outer segments of the rods and cones appear as a dark layer of minimal reflectivity anterior to the RPE and ChCap. The intermediate layers of the retina exhibit moderate backscattering (see Fig. 4). The fovea appears as a characteristic thinning of the retina. The lateral displacement of the retina anterior to the photoreceptors is evident (see Fig. 4).
Fig. 4. OCT image of the normal human macula. (A) Stratus OCT image showing the various cellular layers of the retina. (B) Comparison of the OCT image (same as shown in A) to a histologic micrograph of the normal human macula.
A Review of Algorithms for Segmentation of Retinal Image Data Using Optical Coherence Tomography
21
3.2 Quantitative measurements of retinal morphology OCT can aid in identifying, monitoring and quantitatively assessing various posterior segment conditions including macular edema, ageandnonage related macular degeneration, full and partialthickness macular hole, epiretinal membrane, intaretinal exudate, idiopathic central serous chorioretinopathy, RPE detachment, detachment of the neurosensory retina, and macular lesions associated with optic nerve head pits or glaucoma Figure 5 shows exemplary images of two of the above cited pathological cases obtained with a RTVue FDOCT system (Optovue Inc., Freemonth, CA).
Fig. 5. OCT images showing two OCT Bscans (6 mm length) from pathological retinas. A) Macular hole, B) Epiretinal membrane. As a matter of fact, OCT can demonstrate the presence of edema where it is not seen on biomicroscopy or angiographically. A very important feature of the OCT system is that it provides information on the retinal structures. For example, the location of fluid accumulation in relation to the different retinal layers may be determined and the response to treatment without the need to perform invasive studies such as fluorescein angiography may be objectively monitored. At the same time it may be possible to explain why some patients respond to treatment while others do not. OCT has significant potential both as a diagnostic tool and particularly as a way to monitor objectively subtle retinal changes induced by therapeutic interventions. Thus, OCT may become a valuable tool in determining the minimum maintenance dose of a certain drug in the treatment of retinal diseases, and may demonstrate retinal changes that explain the recovery in some patients without angiographically demonstrable improvement and lack of recovery in others. In the clinical routine, measurement of retinal thickness by the OCT software depends on the identification of the internal limiting membrane and the hyperreflective band believed to correspond to the retinal pigment epithelium – choriocapillaris interface (or, more precisely, the photoreceptor innerouter segment border in the case of third generation OCTs). The OCT software algorithms calculates the distance between these 2 boundaries across all of the sampled points and interpolates the retinal thickness in the unsampled areas between these lines. However, once the various layers can be identified and correlated with the histological structure of the retina, it may seem relevant to measure not only the entire thickness of the retina, but the thickness of the various cellular layers. Moreover, measuring the reflectance of the various retinal layers on OCT images may also be of interest. Drexler et al. have shown in in vitro and in vivo (Bizheva et al., 2006; Hermann et al., 2006) studies that physiological processes of the retina lead to optical density changes that can be observed by a special Mmode OCT imaging, known as optophysiology. Thus, it also seems rational that quantitative analysis of reflectance changes may provide clinically relevant information in retinal patophysiology.
22
Image Segmentation
3.3 Recent developments in OCT technology The emergence of ultrabroad bandwidth femtosecond laser technology has allowed the development of an ultrahigh resolution OCT, which has been demonstrated to achieve axial resolutions of 3 µm during in vivo imaging of the human retina, which is two orders of magnitude higher than what can be achieved by conventional ultrasound imaging. Figure 6 shows the ultrahigh resolution OCT cross section of a normal human macula showing all of the major layers and internal structures of the retina. The ultrahigh resolution OCT will in effect be a microscope capable of revealing certain histopathological aspects of macular disease in the living eye. As it was previously explained, in the conventional or time domain OCT (TDOCT) system the length of the reference arm in an interferometer is rapidly scanned over a distance corresponding to the imaging depth range. The mechanism of scanning largely limits the acquisition speed and makes realtime imaging impossible. In recent years a new model OCT based on Fourier domain interferometry has emerged, and it has been called spectral domain OCT (SDOCT) or Fourier domain OCT (FDOCT) (Fercher et al., 1995; Fercher et al., 2003; Hausler & Lindner, 1998). SDOCT can avoid scanning of the reference, thus it can reach very high acquisition speed. As a matter of fact, in time domain OCT the location of scatters in the sample is observed by generation of interferometric fringes at the detector as the reference reflector position is axially translated. In contrast, Fourier domain OCT required the reference arm to be held fixed, and the optical path length difference between sample and reference reflections is encoded by the frequency of the interferometric fringes as a function of the source spectrum. Two configurations have prevailed in Fourier domain systems: spectral domain (SD) OCT uses a grating to spatially disperse the spectrum across an arraytype detector, and in swept source (SS) OCT a narrow band laser is swept across a broad spectrum, encoding the spectrum as a function of time. SDOCT offers a significant sensitivity advantage over TDOCT (Choma et al., 2003; de Boer et al., 2003, Leitgeb et al., 2003; Mitsui, 1999).
Fig. 6. Ultrahigh resolution OCT cross section of a normal human macula with 3 microns resolution (Courtesy "James Fujimoto," (Fujimoto et al., 2003)). New technology has also been developed to improve resolution in the transverse dimension. In the current commercial application of OCT, the transverse resolution is limited by the intrinsic ocular aberrations of the eye. The transverse resolution can be significantly improved by correcting the aberrations across a large pupil using adaptive optics (AO). A high axial (3 µm) and improved transverse (5–10 µm) resolution AOOCT system was demonstrated for the first time in in vivo retinal imaging (Hermann et al., 2004).
A Review of Algorithms for Segmentation of Retinal Image Data Using Optical Coherence Tomography
23
The instrument uses a wavefront sensor that measures the aberrations in patient’s eyes and then adjusts the surface of an adaptive mirror to correct the visual flaws. An improvement of the transverse resolution of two to three times, compared with ultrahigh resolution OCT systems used so far, was obtained by using adaptive optics A significant signaltonoise ratio improvement of up to 9 dB in corrected compared with uncorrected OCT tomograms was also achieved. Zawadzki et al. also demonstrated the ability to image microscopic blood vessels and the cone photoreceptor mosaic using an AOOCT instrument with high 3D resolution(4x4x6 μm) (Zawadzki et al., 2005). AOOCT is currently the only option for studying living retinal tissue at the cellular level and clinical trials of the instrument are still being performed. Another limitation of OCT technology has been the difficulty in accurately localizing the crosssectional images and correlating them with a conventional en face view of the fundus. One way to localize and visually interpret the images would be to integrate a scanning laser ophthalmoscope (SLO) into the OCT, thereby mapping OCT pixels to the conventional en face view of the SLO. This rationale was used by Ophthalmic Technologies Inc (Toronto, Canada) to develop the Spectral OCTSLO in 2004 (Podoleaunu et al., 2004). The system simultaneously produces SLO and OCT images that are created through the same optical path, and therefore correspond pixel to pixel. OCTSLO, offers multiple views from a single scan with perfect registration of images. OCTSLO imaging offers very accurate localisation of pathology with enhancement of the vitreoretinal interface. Its ability to align serial topographies and to fuse other modalities, with realtime, ultrahigh resolution capability and multiplanar anterior segment imaging should make it an invaluable addition to the diagnostic arsenal of the vitreoretinal surgeon. A number of other instruments have also been built based on variations of the basic OCT system. For instance, polarizationsensitive optical coherence tomography (PSOCT) uses polarizationaltering optics in the arms of the interferometer to determine the sample birefringence from the magnitude of the backreflected light (de Boer et al., 1997; de Boer et al., 2003). This instrument can be used to assess effects such as retinal nerve fiber layer thickness (Cense et al., 2002), early osteoarthritic changes in cartilage (Hermann et al., 1999) or burn depth in thermally damaged tissue (de Boer et al., 1998). Optical coherence microscopy (OCM) is a hybrid instrument that uses a system of high numerical aperture to achieve resolutions comparable to confocal microscopy but with increased depth of penetration (Izatt et al., 1994b). This instrument has been applied to gastrointestinal tissues and promises to enable endoscopically based cellular imaging (Izatt et al., 1996; Aguirre et al., 2003). Doppler optical coherence tomography (Doppler OCT) is an augmentation capable of simultaneous blood flow mapping and spatially resolved imaging (Chen et al., 1997; Izatt et al., 1997; Westphal et al., 2002; Wong et al., 2002; Yazdanfar et al., 2003; Zhao et al., 2000; Ding et al., 2002; Ren et al., 2002). Doppler flow measurements can be performed by measuring the Doppler shift of light scattered from blood. Doppler OCT has been used to explore the human retinal flow dynamics (Yazdanfar et al., 2003); and it is a promising imaging technology for quantitatively assessing capillarity density and angiogenesis (Fujimoto et al., 2003). Functional OCT imaging is another emerging modality that facilitates the assessment of functional or biochemical properties of the investigated tissue. Spectroscopic OCT imaging using broadband light sources enables the spectrum of the backscattered light from each pixel to be measured (Morgner et al., 2000). This extension of OCT is closely related to classical Fourier transform infrared spectroscopy and has the advantage that the
24
Image Segmentation
spectroscopic information can be acquired at multiple wavelengths across the available bandwidth of the light source in a single measurement (Boppart et al., 1999). The potential of Spectroscopic OCT in developmental and cellular biology is really promising. As it can be seen, a wide range of OCT imaging platforms with rapidly emerging applications spanning a range of fields has been developed. The rapid advances in OCT imaging are likely to alter the practice of ophthalmology dramatically in the next several years. Increased resolution and imaging speeds, wavefront correction, improved multiple functionality of the OCT systems; and the possibility of quantitative 3D modeling are just a few of the features to look for in the future. Further advances may transform the OCT from an ancillary procedure to a common and necessary “optical biopsy”. Indeed, future ophthalmologists will use the next generation of OCT devices as a broad based tool for comprehensive ophthalmic examinations; and may even diagnose macular disorders exclusively by digital imaging, without a funduscopic examination.
4. Medical image segmentation approaches Imaging operations may be broadly classified according to four categories: preprocessing, visualization, manipulation and analysis. Segmentation is a common used operation in preprocessing approaches and an essential operation for most visualization, manipulation and analysis tasks in image processing (see Fig. 7). Segmentation is, therefore, the most critical among all imaging procedures, and also the most challenging. Segmentation purpose is to identify and delineate objects. Here, an object refers to any physical object such as an anatomical organ or a pathological entity such as a tumor or cyst (see Fig. 8). Segmentation is defined as the partitioning of an image into nonoverlapping, component regions which are homogeneous with respect to some characteristic such as intensity or texture (Haralick et al., 1985; Gonzalez & Woods, 1992; Pal & Pal, 1993). Typically, image segmentation consist of two related tasks: recognition and delineation. Recognition consists of determining approximately the objects' location in the image. For example, in Figure 9, this task involves determining the location of the RNFL, GCL, IPL, etc. This does not involve the precise specification of the region occupied by the object. Delineation involves determining the objects' precise spatial extent and composition including gradation of intensities. In Figure 9 again, if retinal tissue is the object structure of interest, then delineation consists of the spatial extent of the RNFL and GCL separately, and for each element (i.e. pixels for 2D and voxels for 3D) in each object, specifying a characteristic value of the object (for example, RNFL thickness or volume). Once the objects are defined separately, the RNFL and GCL can be individually visualized, manipulated and analyzed. While automatic and humanassisted are the only two approaches for recognition tasks, numerous methods are available for delineation. Approaches to delineation can be classified as: 1) boundarybased and 2) regionbased (Kim & Hori, 2000). Numerous approaches regarding image segmentation techniques are available in the literature. Some of these techniques use only the gray level histogram, some use spatial details while others use fuzzy set theoretic approaches. Most of these techniques are not suitable for noisy environments. In particular, segmentation approaches can be classified according to the methodology used in the segmentation strategy (see Fig. 10): 1. Classical segmentation methods: These approaches classically partition an image into nonoverlapping segments which are homogeneous with respect to some characteristic such as intensity or texture (Haralick et al., 1985; Gonzalez & Woods, 1992; Pal & Pal, 1993).
A Review of Algorithms for Segmentation of Retinal Image Data Using Optical Coherence Tomography
25
Fig. 7. Classification of image operations.
Fig. 8. OCT image segmentation results showing isolated retinal features of interest. A) OCT Bscan showing multiple lesions in the central retinal area of a patient with age related macular degeneration. Macular cysts and the subretinal fluid area were segmented using a deformable model (Cabrera Fernández et al., 2005a). B) Segmentation result showing the intraretinal layers outlined on an OCT Bscan section obtained from a healthy subject . 2.
Pixel classification methods: These methods basically do not require the constraint that regions be connected. Thresholding, classifier, clustering, and Markov random field (MRF) approaches can be considered pixel classification methods. Thresholding is the most intuitive approach to segmentation (Sahoo et al., 1988). Specifically, algorithms based on threshold create a partitioning of the image based on quantifiable features, like image intensity or gradient magnitude. These algorithms map and cluster pixels in a feature space called a histogram. Thresholds are chosen at valleys between pixel clusters so that each pair represents a region of similar pixels in the image. The segmentation is then achieved by searching for pixels that satisfy the rules defined by the thresholds. Thresholds in these algorithms can be selected manually according to a priori knowledge or automatically through image information. Thresholding algorithms can be divided into edgebased ones (e.g. Canny edge detector and Laplacian edge detector), regionbased ones (e.g. region growing algorithms) and hybrid ones (e.g. watershed algorithms). Edgebased algorithms attempt to find edge pixels while eliminate the noise influence. Thresholds in the edgebased algorithms are related with the edge information. Regionbased algorithms exploit the fact that pixels
26
Image Segmentation
inside a structure tend to have similar intensities. Region growing algorithms, once initial seeds are selected, search for the neighboring pixels whose intensities are inside the intervals defined by the thresholds and then merge them to expand the regions. Hybrid algorithms fuse region information with a boundary detector to complete the segmentation. Typical hybrid algorithms are the level set method with regularizers and the watershed algorithm. Particularly, the watershed algorithms (Yezi et al., 1999) combine the image intensity with the gradient information. In addition, the methodology of using MRF based methods to the problem of segmentation has received a great deal of attention in the past decade (Tamás et al., 2000). MRF modeling itself is not a segmentation method but a statistical model which can be used within segmentation methods. As shown in Fig. 10, MRF is a regionbased approach. MRF models spatial interactions between neighboring or nearby pixels. Hence, the classification of a particular pixel is based, not only on the intensity of that pixel, but
Fig. 9. Segmentation results for an OCT Bscan obtained from a healthy normal eye. The layers have been labeled as: ILM: inner limiting membrane, RNFL: retinal nerve fiber layer, GCL+IPL complex: ganglion cell layer and inner plexiform layer, INL: inner nuclear layer, OPL: outer plexiform layer, ONL: outer nuclear layer, OS: outer segment of photoreceptors, and retinal pigment epithelial layer (RPE). We note that the sublayer labeled as ONL is actually enclosing the external limiting membrane (ELM) and IS, but in the standard 10 µm resolution OCT image this thin membrane cannot be visualized clearly, making the segmentation of the IS difficult. Therefore, this layer classification is our assumption and does not reflect the actual anatomic structure. Also, observe that since there is no significant luminance transition between GCL and IPL, the outer boundary of the GCL layer is difficult to visualize in the Stratus OCT image shown. Thus, a combined GCL+IPL layer is preferable.
A Review of Algorithms for Segmentation of Retinal Image Data Using Optical Coherence Tomography
3.
4.
5.
6.
27
also on the classification of neighbouring pixels. These local correlations provide a mechanism for modeling a variety of image properties (Li, 1995). MRF is often incorporated into clustering segmentations such as Kmeans under a Bayesian prior model (Held et al., 1997; Pappas et al., 1992; Rajapakse et al. 1997). Pattern recognition methods: Since structures in medical images can be considered as patterns, pattern recognition techniques can be used in segmentation procedures. There are supervised and unsupervised classification methods used to perform segmentation. Supervised classification algorithms are pattern recognition techniques that require training data that are manually segmented and then used as references for automatically segmenting new data. Typical supervised algorithms include artificial neural network (ANN), support vector machine (SVM) and active appearance models (AAM) (Alirezaie et al., 1997; Cootes et al., 2001; Wang et al., 2001). Unsupervised classification algorithms, also known as clustering algorithms, do not require a training set but they do require initial parameters to perform segmentation. Commonly used unsupervised classification algorithms include Fuzzy Cmeans algorithm (FCM), iterative selforganizing data analysis technique algorithm (ISODATA) and unsupervised neural network (Cheng et al., 1996; Mohamed et al., 1998; Wong et al., 2002). Deformable model methods: These are modelbased techniques for delineating region boundaries using closed parametric curves or surfaces that deform under the influence of internal and external forces. Deformable models can be classified into parametric and geometric models depending on the contour representation. These algorithms are usually interpreted as a modeling of curve evolution because they delineate an object boundary in an image by placing a closed curve or surface near the desired boundary and then allowing to undergo an iterative relaxation process. The parametric deformable models sample contours as discrete points and track them according to their respective moving equations. The moving equation for the parametric deformable models can be derived through either energy functional or dynamic forces. A priori knowledge can be easily incorporated to the procedures of parametric models. The geometric deformable models are based on the level set method (Osher & Sethian, 1988), which embed the moving contour into a higher dimensional level set function and view the contour as its zero level set. Then, instead of tracking the contour points, the zero level set of the level set function are tracked. The advantage of doing so is that topological changes can be naturally handled and the geometric properties of the contour such as normal vector and curvature can be calculated implicitly. Consequently, the computational complexity is decreased. Global optimization methods: A growing number of image segmentation approaches use energy minimization techniques (Boykov & FunkaLea, 2006; Kolmogorov & Zabih, 2004). Among all the various energy minimization techniques for segmentation, graph cuts are based on partitioning a graph by a minimum cut / maximum flow optimization algorithm (Greg et al., 1986; Ford & Fulkerson, 1956). The image is represented using an adjacency graph. Each vertex of the graph represents an image pixel, while the edge weight between two vertices represents the similarity between two corresponding pixels. Usually, the cost function to be minimized is the summation of the weights of the edges that are cut. Registration Methods: The standard method used is the atlasguided approach which treats segmentation as a registration problem (Maintz & Viergever, 1998). Typically, the atlas is generated by compiling information on the anatomy that requires segmentation
28
7.
8.
Image Segmentation
and then used as a reference frame for segmenting new images. This approach is theoretically similar to classifier methods but it is implemented in the spatial domain of the image rather than in a feature space. This approach uses a procedure known as atlas warping which first finds a onetoone transformation that maps a presegmented atlas image to the target image that requires segmenting. The warping can be performed using linear transformations (Andreasen et al., 1996; Lancaster et al., 1997; Talairach and P. Tournoux, 1988). Modelfitting methods: These approaches usually fits a simple geometric shape such as an ellipse or parabola to the locations of extracted image features in an image (Pathak et al., 1998). It is a technique which is specialized to the structure being segmented but is easily implemented and can provide good results when the model is appropriate. A more general approach is to fit spline curves or surfaces to the features (Kim & Hori, 2000). LEGION based: These approaches are a biologically plausible computational framework for image analysis based on a biologically inspired oscillator network, called the locally excitatory globally inhibitory oscillator network (LEGION) (von der Malsburg, 1981;
Fig. 10. Classification of segmentation approaches.
A Review of Algorithms for Segmentation of Retinal Image Data Using Optical Coherence Tomography
29
Wang & Terman, 1995; Wang & Terman, 1997). The network was proposed based on theoretical and experimental considerations that point to oscillatory correlation as a representational scheme for the working of the brain. The oscillatory correlation theory assumes that the brain groups and segregates visual features on the basis of correlation between neural oscillations (von der Malsburg, 1981; Wang & Terman, 1995). As a final point, it is noteworthy to mention that years of research in segmentation have demonstrated that significant improvements on the final segmentation results may be achieved by using notably more sophisticated feature selection procedures, more elaborate segmentation techniques, taking into account prior distribution on the labels, region processes, or the number of classes, finally, involving (e.g. in the case of energybased segmentation models) more costly optimization techniques.
5. Review of algorithms for segmentation of retinal image data using OCT In opthalmology, a number of segmentation approaches of retinal image data using OCT have been proposed to enhance the clinical benefit of the OCT technology in the process of clinical decision making. The success of OCT in the investigation and treatment of retinal diseases might be best illustrated by the progress in automated analysis and the recent advancement of this technology from time domain to spectral domain (Cense et al., 2004; Drexler & Fujimoto, 2008; Fercher et al., 1995; Huang et al., 1991; Wojtkowski et al., 2003). The segmentation of the retinal structure is a challenging topic that faces major problems. First, OCT images suffer from the intrinsic speckle noise, which decreases the image quality and complicates the image analysis. This particular noise is the foundation of existing problems in the precise identification of the boundaries of the various cellular layers of the retina and other specific retinal features present in the OCT tomograms. Second, since the intensity pattern in OCT images results from absorption and scattering of light in the retinal tissue, intensity of a homogeneous area decreases with increasing imaging depth deterministically. This complicates segmentation algorithms which are commonly based on the assumption that intensity variations of homogeneous regions are only due to noise and not intrinsic to the imaging modality. The third problem is the low optical contrast in some regions of the OCT images due to the optical shadows of the retinal blood vessels resulting from the high haemoglobin absorption of light. Finally, motion artifacts and suboptimal imaging conditions affecting the quality of the OCT tomograms also cause failure in the segmentation approaches or reduce their accuracy. In this section, a number of approaches that have appeared in the literature on OCT image segmentation are described. This review concentrates on automated and semiautomated algorithms developed to segment the various cellular layers of the retina, structural irregularities associated to retinal disease (e.g. drusen and fluidfilled regions), and particular features of the optic nerve head in OCT images. Specifically, automation in OCT image analyses requires the establishment of parameters and features obtained from quantitative measurements of OCT data such as optical and structural parameters, scattering properties and refractive index of biological tissues. Thus, Table 1 gives an overview of the approaches discussed in this section based on the details of the above quantitative parameters and features. Although retinal blood vessel segmentation methods often consist of applying segmentation algorithms to fundus images, and more recently to
30
Image Segmentation
advanced OCT images (e.g. SDOCT and UHR OCT) using the vessel shadows (Wehbe et al., 2007), I consider this application of segmentation to be a separate branch of research and do not included it in this review. The segmentation methods that will be reviewed can be classified into three groups based on the dimension (D) of the image analyzed. In particular, a number of different methods have been reported under the 1D, 2D and 3D categories (see Table 1). Segmentation approaches of OCT retinal images differ depending on the number of retinal features (e.g. number of intraretinal layers and fluidfilled regions) to be segmented, and on their robustness in the presence of inherent speckle noise, retinal blood vessel shadows and structural irregularities at the fovea, macula and optic nerve in pathological retinal tissue. Most of the initial segmentation algorithms are based on information retrieved from either gradient or conventional intensity data. However, some recent segmentation methods are based on more complex models, such as active contours and optimal graph search methods. It is worthy of mention that all existing published segmentation approaches have been basically introduced to overcome the limitation of the commercial OCT softwares and most of them have provided additional quantitative information of the retinal structure. As it can be seen in Table 1, a number of segmentation approaches have been proposed to segment the retinal structure. The initial segmentation method proposed by Hee et al. was based on intensity variation and demonstrated the potential of OCT as a quantitative tool to measure total retina and RNFL thickness (Hee et al., 1995a, 1995b; Hee, 1997). This very first method used a 1D edge detection kernel approach, which is independent of the absolute signal level in the image, to compute the derivative of reflectivity versus axial distance for each Ascan in the OCT image. Thus, it is more effective than threshold identification. Specifically, the detection kernel identified the strongest two edges in each Ascan using peakdetection in more than 90% of the Ascans. Huang et al. used a similar approach to characterize the retina and outer retina choroid complex in hereditary retinal degenerations in experimental animals and humans (Huang et al., 1998). This early work, to the knowledge of the author, represents the first study to characterize and quantify OCT signals in relation to the optical properties of retinal layers. In contrast, George et al. used a dual threshold to segment the retina and choriocapillaries structure from OCT images (George et al., 2000). Unfortunately, very little information is available about this work. However, simple thresholding is sensitive to noise and intensity inhomogeneities in OCT images because it does not take into account the spatial characteristics of the image. Koozekanani, et al. introduced a Markov random field (MRF) model for extracting the inner and outer retinal boundaries from radial scans of the macula (Koozekanani et al., 2001). This autoregressive model showed to be more robust on the macular region of normal retinas than standard columnwise thresholding methods. Particularly, retinal thickness was calculated with an error comparable to the 10µm resolution of the OCT system used, representing a substantial improvement over clinical measurements provided by the Humphrey 2000 OCT builtin algorithm. Although a difficulty associated with MRF models is the proper selection of parameters controlling the strength of spatial interactions, Koozekanani's model is entirely independent and involves no critically tuned parameters. However, the main problem of this model is to find reliable “seed” points for OCT images of retinal pathologies. On the other hand, since this model relies on simply connecting 1D points, makes it sensitive to
A Review of Algorithms for Segmentation of Retinal Image Data Using Optical Coherence Tomography
31
noise. Thus, it demands to apply special rules to correct for errors in the extracted layer borders because the model per se is sensitive to noise. Since the problem with thresholding lies in the selection of the threshold, Herzog et al. proposed a method based on edge maximization and smoothness constraints to choose an optimal threshold to utomatically extract the optic nerve and retinal boundaries from axial OCT scans through the optic nerve head (ONH) (Herzog et al., 2004). A method to automatically segment the curve to extract the nerve head profile was also proposed. An interesting aspect is that in this approach the boundaries are obtained by maximizing the number of edges that lie on the boundary while minimizing the boundary’s average rate of change. The algorithm generally identified the correct vitrealretinal boundary in the images except in regions where the OCT signal was severely attenuated due to shadowing. This study is the first published work on ONH segmentation using TDOCT data. Two years later, a more efficient methodology to segment OCT nerve head images and extract the necessary parameters for clinical measurements such as the cuptodisk ratio and RNFL thickness was proposed by Boyer et al. (Boyer et al., 2006) The improved methodology is esentially a parabolic model of the cup geometry and an extension of the Markov model introduced by Koozekanani et al. This study is the first published work on clinical parameter extraction taking advantage of the optic nerve head crosssectional geometry in TDOCT images. Recently, Shrinivasan et al. also used a modification of the Koozekanini algorithm to perform quantitative measurements of the outer retinal morphology using UHR OCT (Shrinivasan et al., 2008). In this study the thick scattering region of the outer retina previously attributed to the RPE is shown to consist of distinct scattering bands corresponding to the photoreceptor outer segment tips, RPE, and Bruch’s membrane. Gregori et al. presented the first algorithm that was able to locate automatically and/or interactively the complex geometry and topology typical of many macular pathologies in TDOCT images (Stratus OCT system), and lately in SDOCT images (Cirrus HDOCT unit) (Gregori et al., 2004; Gregori et al., 2005; Gregori et al., 2008). Unfurtunately, this is a proprietary algorithm that has not been described in detail because has been licensed to Carl Zeiss Meditec, and it is currently part of the commercial Cirrus HDOCT instrument. However, it is a robust algorithm able to locate the boundaries of the major anatomical layers internal to the retina with great accuracy not only in eyes presenting abnormal and unusual anatomy but also in poor quality images. Table 1 includes a summary of all the results that have been presented in the ARVO meetings since 2004. In 2005, algorithms based only on intensity variation were also presented (Shahidi et al., 2005; Ishikawa et al., 2005; Cabrera Fernández et al., 2005b). In general, these algorithms overcame the limitations of the commercial OCT3/Stratus OCT software and also provided additional quantitative information. For example, Shahidi et al. segmented three retinal segments by using a simple search of peaks corresponding to high and lowintensity bands, and an improved edge detection approach using the correlation between axial Ascans was presented in a more recent study (Bagci et al., 2007). Ishikawa et al. used a modified median filter and an adaptive thresholding method based on the reflectivity histogram of each Ascan line to segment four layer structures within the retina (Ishikawa et al., 2005). A similar adaptive thresholding approach along with an intensity peak detection procedure was also employed by Ahlers et al. to segment data from patients with RPE detachments (Ahlers et
32
Image Segmentation
al., 2008). This study was also based on the work by Gregori et al. (Gregori et al., 2005). It is worthy of mentioning that the work of Ishikawa et al. was the first report demonstrating that the thickness of the innermost layers in the macula had diagnostic power comparable with that of circumpapillary nerve fib er layer (cpNFL) in glaucoma studies (Ishikawa et al., 2005). Later on, Tan et al. using a 2D gradient approach in a dynamic programming framework also confirmed that glaucoma primarily affects the thickness of the inner retinal layers (RNFL, GCL, IPL) in the macula (Tan et al., 2008). Cabrera Fernández et al. used complex diffusion filtering to reduce speckle noise without blurring retinal structures and a peak finding algorithm based on local coherence information of the retinal structure to determine seven intraretinal layers in a automatic/semiautomatic framework (Cabrera Fernández et al., 2005b). This algorithm searches for edges in a map obtained by calculating the first derivative of the structure coherence matrix using the denoised image. Although, good results were obtained for some pathological Stratus OCT images, the algorithm in its original development was prone to failure and allowed detected boundaries to overlap. This algorithm worked reliably for data from 72 OCT Bscans from healthy normal subjects. The automatic/semiautomatic framework developed by Cabrera Fernández et al. was used to demonstrate for the first time the potential of OCT quantification for early DR damage (Cabrera Fernández et al., 2008; Cabrera DeBuc et al., 2010). The early segmentation work of Gregori et al., Ishikawa et al. and Cabrera Fernández et al. allowed the automated generation of 2D thickness maps of individual retinal layers and, therefore, also a more local analysis of the retinal morphology using Stratus OCT data before the introduction of advanced OCT systems (Gregori et al.,2005; Ishikawa et al., 2005; Cabrera Fernández et al., 2005b). A different approach using active contour algorithms has been used to quantify structural irregularities in OCT retinal images. For example, Cabrera Fernández et al. applied for the first time a deformable model to TDOCT images of retinas demonstrating cystoids and subretinal fluid spaces using a semiautomatic framework (Cabrera Fernández et al., 2004; Cabrera Fernández et al., 2005a). Specifically, this method used a nonlinear anisotropic diffusion filter to remove strong speckle noise and a gradient vector flow (GVF) snake model to extract fluidfilled regions in the retinal structure of AMD patients. Extension of this deformable model framework to a daily routine image analysis might prove to be difficult and unpractical since the algorithm requires manual interaction to place an initial model and choose appropriate parameters. Mujat et al. used deformable splines to assess the thickness of the RNFL in SDOCT images. (Mujat et al., 2005). Although all the model parameters were set based on a large number of OCT scans in different retinal areas, contour initialization is still a major problem because it must be close to the true boundary locations. In addition, though sensitivity to initialization was not reported in this study, the approach was highly vulnerable to the existence of blood vessels and other morphological retinal features. However, the advantage of this automated snake methodology is that it is able to provide larger area maps of the RNFL thickness facilitating the correct registration of ROIs with visual field defects which could allow better longitudinal evaluation of RNFL thinning in glaucoma studies. In 2009, Yazdanpanah et al. presented a modified Chan– Vese’s energyminimizing active contour algorithm in a multiphase framework to segment SDOCT data from rodent models. This approach incorporated a circular shape prior based on expert anatomical knowledge of the retinal layers, avoiding the need for training
A Review of Algorithms for Segmentation of Retinal Image Data Using Optical Coherence Tomography
33
(Yazdanpanah, et al., 2009). Although the sensitivity of the algorithm with respect to model parameters and initialization was not tested, the experimental results showed that this approach was able to detect with good accuracy the desired retinal layers in OCT retinal images from rats compared to the ground truth segmentation used in the evaluations performed. Moreover, the algorithm was not evaluated for images including the foveal pit. Later on, Mishra et al. also presented a modified active contour algorithm based on a sparse dynamic programming method and a twostep kernel based optimization scheme (Mishra et al., 2009). Although this effective algorithm achieves accurate intraretinal segmentation on rodent OCT images under low image contrast and in the presence of irregularly shaped structural features, results on images including the foveal pit region were not given and no quantitative evaluation using a large data set was provided. Baroni et al. used a multistep approach to extract the boundaries of the vitreoretinal interface and the inner and outer retina by maximizing an edge likelihood function (Baroni et al., 2007). Interestingly, the effect of intravitreal injection of triamcinolone acetonide for the treatment of vitreoretinal interface syndrome was evaluated using a set of measures such as thickness measurement, densitometry, texture and curvature extracted from the identified retinal layers. This study was the first report, to the knowledge of the author, that demonstrated the potential of texture information in TDOCT retinal images as a complimentary information of retinal features to aid diagnosis. Another intensity variation based approach to segment the posterior retinal layers, which is resistant to discontinuities in the OCT tomogram, was presented by Szulmowski et al (Szulmowski et al., 2007). Furthermore, the quantitative analysis has been largely limited to total retinal thickness and/or inner and outer retinal thickness in early studies exploring the correlation between histology and OCT in rodents (e.g. see Kocaoglu et al., 2007 & Ruggeri et al., 2007). Recently, more intensity variation based approaches have also been presented (see Table 1 for details) (Fabritius et al., 2009; Tumlinson et al., 2009; Koprowski et al., 2009 ; Lu et al., 2010 and Yang et al., 2010) Among them, it is worthy to mention that Fabritius et al. incorporated 3D intensity information to improve the intensity based segmentation and segmented the ILM and RPE directly from the OCT data without massive preprocessing in a very faster manner. (Fabritius et al., 2009). Likewise, Yang et al. presented a fast, efficient algorithm that simultaneously utilized both local and global gradient information (Yang et al., 2010). This approach skillfully used an Ascan reduction technique to reduce the execution time to 16 seconds per volume (480x512x128 voxels) without remarkably degrading the accuracy or reproducibility of the results. In addition, an alternative promising method was introduced by Mayer et al., who used a fuzzy Cmeans clustering technique to automatically segment RNFL thickness in circular OCT Bscans without the need of parameter adaptation for pathological data (Mayer et al., 2008). In contrast to the edge detection approaches mentioned above, a multiresolution hierarchical support vector machine (SVM) was used in a semiautomatic approach to calculate the thickness of the retina and the photoreceptor layer along with the volume of pockets of fluid in 3D OCT data (Fuller et al., 2007). In this approach, the SVM included scalar intensity, gradient, spatial location, mean of the neighbors, and variance. Although this SVM method performed well on both healthy and diseased OCT data, a major drawback was that some voxels were misclassified resulting in scattered noise in the thickness maps. In addition, this method requires that the user paints the areas of interest in
34
Image Segmentation
any slice of the volume. Thus, the training data set grows through painting increasing the complexity of the SVM, and as a result more time is required to complete the segmentation task. A different approach was presented by Tolliver et al., who used a graph partitioning algorithm that assumes that different regions of the OCT image correspond to different modes of oscillation. The oscillation steps that represent the retinal edges are then determined by an eigenvector calculation (Tolliver et al., 2008). By using the eigenvector from the prior step as a starting point, for finding the new eigenvector, the approach works in only a small number of steps. In this study, the accuracy range for the detected boundaries was good and the algorithm performed well in the presence of retinal pathology. On the contrary to the vast majority of the studies cited so far, Götzinger et al. used PSOCT to segment the RPE layer employing polarization scramble features. Even though the two algorithms presented facilitated a better visualization and quantification of RPE thickening and RPE atrophies when compared to algorithms based on intensity images, a PSOCT system is needed to acquire polarization data (Götzinger et al., 2008). A more complex approach to OCT retinal layer segmentation using gradient and/or intensity information in a 3D context was presented by Haecker et. al., who generated a composite 3D image from radial linear TDOCT 2D scans and performed a 3D graphsearch (Haecker et al., 2006). The basic idea of this graph approach is to break a graph into paths or fragments, which are utilized as filtering features in graph search. The early development of Haecker et al.'s algorithm extracted only 2 intratretinal layers and was evaluated on data from 18 controls and 9 subjects with papilledema (Haecker et al., 2006). This approach was further developed into a multilayer segmentation (Garvin et al., 2008) showing superior results for high quality OCT data. This graphsearch approach potentially increased the accuracy of segmentation by using weights describing both edge and regional information to segment the volume. However, assumptions on the layers, as Garvin et. al. made, may be violated in pathological cases, or parameters have to be adapted for either normal subjects or pathological patients. Even though this elegant method can guarantee to find the global minimums when compared to deformable models, its computational complexity can really increase the computation time if more complex constraints are required to segment diseased retinal images showing common structural irregularities and a less ideal foveal pit. In 2009, Abramoff et al. combined a multiscale 3D graph search algorithm and a voxel column classification algorithm using a kNN classifier to segment the ONH cup and rim (Abramoff, et al., 2009). This preliminary study showed for the first time a high correlation between segmentation results of the ONH cup and rim from SDOCT images and planimetry results obtained by glaucoma experts on the same eye. Later on, Lee et al. presented an improved and fully automatic method based on a similar methodology using graph search combined with a kNN classifier that employed contextual information combined with a convex hullbased fitting procedure to segment the ONH cup and rim (Lee et al., 2010). In general, the methodology showed good performance but additional processing steps to compensate for the presence of vessels in and around the ONH would be required to reduced misclassified Ascans on the vessels and increase the accuracy of the ONH rim or cup contour segmentation. Similarly, Hu et al. used a graphtheoretic approach to segment the neural canal opening (NCO) and cup at the level of RPE/Bruch’s membrane complex (Hu et al., 2010). Qellec et al. presented a promising method for detecting footprints of fluidfilled regions in SDOCT images from AMD patients (Quellec et al., 2010). This approach also used
A Review of Algorithms for Segmentation of Retinal Image Data Using Optical Coherence Tomography
35
a multiscale 3D graph search method to identify automatically a total of 10 intraretinal layers. The segmented layers were characterized by their thickness and 3D textural features. As in the Baroni et al. study, this report confirmed that useful 3D textural information can be also extracted from SDOCT scans to aid local retinal abnormality detection. In addition, Chiu et al. reported a skillful approach based on graphbased theory and dynamic programming that significantly reduced the processing time required for image segmentation and feature extraction (Chiu et al., 2010). This methodology is able to address sources of instability such as the merging of layers at the fovea, uneven tissue reflectivity, vessel hyporeflectivity and the presence of pathology. Interestingly, the approach incorporates an automatic initialization that bypasses the need for manual endpoint selection. The development of SDOCT systems has also made possible a better visualization and identification of the RPEBruch's membrane providing the ability to image drusen. As a result, segmentation algorithms have been recently presented to quantify drusen area and volume in AMD patients. For example, Farsiu et al. presented the DOCTRAP algorithm that is based on a modified implementation of the GVF snake model to accurately segment drusen in SDOCT images of AMD eyes (Farsiu et al., 2008 ). This methodology also included a semisupervised approach to correct for segmentation errors such as false regions marked as drusen in images showing RPE elevation unrelated to drusen. The approach presented by Fuller et al. and described above also facilitates the semiautomatic segmentation of drusen in SDOCT images (Fuller et al., 2007). Gregrori et al. has also measured drusen area and volume using quantitative descriptors of drusen geometry in three dimensional space (Gregori et al., 2008). In addition, Yi et al. characterized ONH drusen using a commercial available software (see Table 1 for details) (Yi et al., 2009). Kajić et al. presented a promissing novel statistical model based on texture and shape able to capture the variance of the training data used to segment unseen data (Kajic et al., 2007). As the authors themselves stated, this guarantees that the segmentation will be close to the ground truth and less sensitive to noise. This algorithm successfully segments 8 intraretinal layers on 3D OCT data even under conditions which prove extremely difficult for some preexisting segmentation approaches cited in the literature. It also has the potential of segmenting choroid layers and the ONH. In this study, for the first time, an error measure is computed from a large manually segmented data set which was certainly segmented twice by different operators.
6. Concluding remarks In contrast to OCT technology development which has been a field of active research since 1991, OCT image segmentation has only being fully active explored during the last decade. However, it continues to be one of the more difficult and at the same time most commonly required steps in OCT image analysis, therefore, there does not and can not exist a typical segmentation method that can be expected to work equally well for all tasks. The works cited in this review spread from the 1997’s until September 2010. Of course, the citation in this review is by no means complete. For example, an early active research topic such as manual tools for image segmentation has not been covered. It is also worthy the mentioning that it was difficult to assess the robustness of the various segmentation approaches because of many authors have used different OCT imaging setups and reported limited quantitative
36
Image Segmentation
validation. Accordingly, a careful evaluation of different available academic and commercial segmentation methods using common test datasets is required to choose the one that best solves the given image processing task. Current research in the segmentation of OCT images is striving towards improving the accuracy, precision, and computational speed of segmentation methods, as well as reducing the amount of manual interaction. On the other hand, most of the reported computation times of segmentation methods on 2D and 3D OCT datasets (see Table 1) are not really practical for general clinical use. However, segmentation methods will be particularly valuable in areas such as computer assisted surgery, where realtime visualization of the anatomy is a crucial component. For increasing computational efficiency, multiscale processing and parallelizable methods appear to be promising approaches (Sylwestrzak et al., 2010). As a matter of fact, the current expanding use of 3D OCT systems along with the advances in volume rendering techniques, is now shifting slowly the focus of segmentation to volume segmentation. In addition, the potential of OCT image segmentation to evaluate therapeutic or adverse effects of experimental interventions in timecourse experiments might prove to be even more important to translate insights from bench to bedside in a proficient and timely manner. Since OCT allows realtime data acquisition, future research will strive towards improving automation and data evaluation in near real time to support retinal disease screening and diagnosis. Automated segmentation still remains one of the most difficult problems in the world of OCT retinal image segmentation. This dificulty mainly arises due to the sheer size of the datasets coupled with the complexity and variability of the pathological retinal anatomy. The situation is worsened by the shortcomings of OCT imaging systems, such as sampling artifacts, noise, low contrast etc. which may cause the boundaries of retinal structures to be indistinct and disconnected. Recently, Liu et al. introduced a very effective approach for automated macular pathology identification in retinal OCT images (Liu et al., 2010). This method uses a machine learning approach that has the potential to provide unsupervised objective classifications for automated OCT data analysis in real time. Computational efficiency is particularly important in realtime processing applications for computer aided diagnosis and surgical planning. As a matter of fact, segmentation algorithms do have the capability to run in parallel with the OCT scanning method and to provide a concrete support for clinical decision making in real time. Finally, it is worthy of mention that automated segmentation methods will never replace physicians but they will likely become crucial elements of medical image interpretation. Thus, there are numerous challenges to improve clinical decision making based on automated processing of OCT data, as outlined throught this chapter, for engineers, mathematicians, physicists and physicians working to advance the field of OCT image analysis.
7. Acknowledgement I would like to acknowledge my amazing husband, Arthur, who was infinitely supportive and always poised to provide external motivation when I needed it most writing this chapter with our new baby boy around. Recognition is also due to my beautiful baby boy, Arthur Anthony, who provides me with joy and a healthy perspective on life that inspires me to work.
Retinal and Imaging data RNFL thickness from various pathological retina
Lowpass filtering: 2D centerweighted kernel
Stratus OCT: Nonlinear 40 OCT Banisotropic scans filter
ONH and retinal Exemplary Median filter boundaries OCT Bscans (4x4 applied from axial OCT taken at the twice) scans through ONH the ONH
Gregori, G., 2004, Global TDOCT (Stratus boundaries OCT ) and SDOCT (ILM, RPE),
Herzog, A., 2004, TDOCT (Humphrey 3000 OCT)
Thickness of the OCT data 2D lineal retina and outer from human smoothing retina choroid subjects (3x3 centercomplex (normal & weighted (ORCC=photore with Gaussian ceptor hereditary smoothing layer+RPE+anter retinal filter applied 5 ior choroid); and degenerations times) the reflectivity ) and posterior to degenerative ORCC avian and swine retinas George, A., 2000/ Thickness and Information Median filter TDOCT volume of the not and image (Humphrey 2000 retina and available(I/N homogenizatio OCT) choriocapillaries /A) n using Nagao filter Koozekanani, D., Inner and outer 1450 images 4x4 median TDOCT (2001, borders of the (Bscans) filter (applied Humphrey 2000 retina twice) OCT)
Huang, Y., 1998, TDOCT (Humphrey 2000 OCT)
Hee, MR., 1997/ TDOCT (Humphrey 2000 OCT)
Computation Time
Adaptive threshold algorithm based on edge maximization and smoothness constraints Iterative boundary detection
Automated /SemiAutomated
Automated
None
Agreement with boundaries determined by
Automatic and Algorithm offered Not reported manual performance measurements significantly differed by less than superior to 25 μm, and in 89% of the Humphrey 2000 the tests the OCT's builtin difference was less algorithm than 10 μm (near the resolution limit) Not reported None First published results on ONH segmentation and geometric characterization from OCT data
First segmentation Not reported algorithm that was able to locate
(I/N/A)
Manual correction for evaluation purposes and linear interpolation to correct segmentation errors None
(I/N/A)
(I/N/A)
(I/N/A)
First report of image Not reported processing techniques and methods of extracting quantitative information from OCT images Not reported First report of quantitative OCT results in hereditary retinal degenerations in experimental animals and humans
Remark
Dual threshold Automated to segment the retina and choriocapillarie s structures Automated 1D edge detection kernel/Markov Boundary Model
Not reported
Error Correction/ Validation (key Refinement points) Method Interpolation and Reproducibility of local edgethe RNFL thickness detection in 10 glaucomatous eyes
None
Automated
Automation
Algorithm Automated based on signal amplitude and slope calculated using a 1stderivative weighted filter
1D edgedetection kernel/peakdetection approach
First Author/Year/ Segmentation Feature Studied Sample Size Preprocessing Method OCT system
A Review of Algorithms for Segmentation of Retinal Image Data Using Optical Coherence Tomography
37
Mujat, M., 2005, experimental SDOCT
RNFL thickness Two Anisotropic volumetric filtering data sets of the same eye from a single subject
Median filter
algorithm (Structural SDOCT: 150 eyes (full Correlation Algorithm for range of Speckle retinal diseases), 30 Removal glaucomatous (SCASR)) eyes and 30 normal eyes
Deformable splines (snake algorithm)
Peak search algorithm (simple search of peaks corresponding to high and lowintensity bands) Ishikawa, H., 2005, Various retinal Fortyseven Modified Adaptive TDOCT (Stratus layers/segments subjects (23 mean filter thresholding OCT) technique (4) normal and (kernel size 24 with 7x5) (cutoff glaucoma) threshold was calculated based on reflectivity histogram of each Ascan line) Cabrera Cystoids and OCT images Nonlinear Gradient vector Fernández, D., subretinal fluid from 7 anisotropic flow snake 2005, TDOCT filled regions patients with diffusion filter model (Stratus OCT) AMD
NFL, GCL+IPL, boundaries of epiretinal membranes and other vitreous structures, the boundaries of cystic spaces, drusen, and RPE detachment
Shahidi, M., 2005, Thickness of the OCT data TDOCT (OCT3 total retina and 3 from 10 CarlZeiss Meditec) retinal segments healthy subjects
(Cirrus HDOCT)
Algorithm performed well in 350 frames with only few and isolated boundary detection errors
Manual correction for evaluation purposes
SemiAutomated
None
Mean distance between all extracted contours using the snake algorithm and the expert’s contours was below 3 pixels.
None
Automated
Automated
Algorithm failure of at least one detected border in approximately 10% of the goodquality images.
None
Automated
First deformable 8492 seconds model applied to (on average) OCT images of for a set of six AMD patients radial OCT Bdemonstrating scans in a cystoids and Pentium 4 CPU, 2.26 GHz subretinal fluid spaces. Provided large area 62 seconds for maps of the RNFL a single image thickness facilitating (1000 Ascans) the correct on a 3.2 GHz registration of ROIs Pentium 4 in glaucoma processor longitudinal studies
First segmentation Not reported algorithm that demonstrated the thickness of the innermost layers in the macula had diagnostic power comparable with that of cpNFL.
visual inspection automatically agreed on at least and/or interactively the complex 99% of the pixels and never diverges geometry/topology by more than 10% of typical of many macular the correct local pathologies. retinal thickness. Automated detected boundaries were within 20 microns from manually drawn boundaries on well over 90% of pixels Reproducibility of Algorithm Not reported thickness averaged measurements thickness profiles in the transversal showed that changes of 28 to 36 direction producing μm can be detected less refined with 95% confidence thickness profiles
38 Image Segmentation
Two 1D filters were used: 1) median filtering (5 pixels) along the Ascans; 2) Gaussian kernel in the longitudinally direction
Baroni,M., 2007, ILM, TDOCT (OCT2 RNFL, inner CarlZeiss Meditec) retina (IR), including GCL and IPL; and outer retina (OR), including OPL and inner photoreceptor layers (IPL) 16 pathologic OCT images from patients with crinkled cellophane maculopathy
2D median filter (4x4 applied twice) along with a Palladian of Gaussian (1D) edge detector
Various retinal 72 OCT BComplex layers/segments scans from diffusion filter (7) normal subjects and 4 Bscans from pathological eyes
Boyer, K., 2006, ONH (cuptoOCT data TDOCT (OCT 3000 disk ratio) and from 59 axial ZeesHumphrey) RNFL thickness OCT nerve head scans
Cabrera Fernández, D., 2005, TDOCT (Stratus OCT)
Automated 2D dynamic programming where edge detection uses a Gaussian gradient applied crosssectionally, as a 1D filter (detection of peaks using cumulative horizontal histograms and boundaries obtained by
Parabolic Automated model of cup geometry and an extension of the Markov model introduced by Koozekanani, et al. to segment the retinalnerve head surface, identify the choroidnerve head boundary, and the extent of the optic cup
OCTRIMA Automated software which /Semiuses a peak Automated search algorithm based on local coherence information of the retinal structure.
None
A glaucoma specialist marked the cup end points in 59 images in order to compare cup endpoints selected by the ophthalmologist to those selected automatically by the algorithm.
Manual correction to eliminate segmentation pitfalls (if any) during the automatic procedure
24 seconds for filteringsegmenting each OCT Bscan (1024x512pixel) on a personal computer (Pentium 4 CPU, 2.26 GHz). A correlation First published Total coefficient for cup results on clinical processing diameter above 0.8 parameter time of 9.42 and above 0.9 for the extraction seconds (7.62 disk diameter taken advantage of seconds for the ONH crossextracting the sectional geometry retinalvitreal in OCT images boundary requires, 0.793 seconds segmenting the cup limits, and 1.004 seconds for segmenting the disk limits) on a Pentium M, 1.8 GHz processor with 1 GByte of RAM None First multistep Not reported segmentation approach that demonstrated the potential of texture information in TDOCT images as a complimentary information of retinal features to aid diagnosis
First segmentation All instances of algorithm algorithm failure that demonstrated were in regions the potential of which had OCT's extremely low reflectivity or almost quantification for no structural early DR damage. information.
A Review of Algorithms for Segmentation of Retinal Image Data Using Optical Coherence Tomography
39
Fuller, AR., 2007, experimental 3D OCT
OCT data Not reported from normal and diseased rodent eyes
3D segmentation algorithm that detects the ILM and RPE boundaries by means of an iterative procedure by which an initial guess is repeatedly evaluated and improved
Evaluation of Not reported the quality of segmentation requires operator intervention.
None Automated for None extracting the inner and outer borders of the retina. Manual segmentation was performed on each crosssectional OCT image to obtain the boundaries of the tumor in the retinoblastoma model. Thickness of the Data from Noise is MultiSemiManual 68% of the thickness retina and the patients with handled by resolution Automated segmentation tool differences between photoreceptor AMD and considering Hierarchical for evaluation the SVM layer along with retinal voxel’s mean Support vector purposes segmentation the volume of detachment value and machine (SVM) and the manual pockets of fluid variance across segmentation fell multiple below 6 voxel units resolutions in the SVM approach
Ruggeri, M., 2007, ILM, RPE layers experimental along with SDOCT tumor volume in small animals
Szkulmowski, M., Posterior retinal OCT data None 2007, experimental layers from normal SDOCT eyes and eyes with a selection of retinal pathologies
maximizing an edge likelihood function) Variation of a Semimultiple Automated thresholding algorithm that identifies regions in the tomogram that have similar intensity and variance
SVM approach with global awareness by considering statistical characteristics at multiple levels of resolution
Less than two minutes (on average) for the full thickness segmentation performed on a computer with 3GB of RAM ( dual processor 3GHz Intel Xeon)
About 5 minutes for full processing of a data volume containing 200 images with 600 Ascans/image on a personal computer with a Pentium 4, 2 GHz processor Algorithm facilitates Not reported the thickness map of the rodent retina Segmentation method is resistant to discontinuities in the tomogram
40 Image Segmentation
Various retinal layers/ segments (6) but main focus on the outer retina segmentation
Inner and outer OCT data from 22 borders of the retina along with patients with the volume and fPED area of fPEDs
Shrinivasan, VJ., 2008, UHROCT
Ahlers, C., 2008, SDOCT (Cirrus HDOCT)
OCT data from 43 healthy subjects
Morphological filtering to eliminate thin vitreous membranes from the thresholded image
Adaptive Automated thresholding technique and intensity peak detection
None
Linear interpolation to correct for segmentation errors
Average of three experts’ tracings as a reference standard was used for evaluation purposes
Optimal 3D Automated graph search (graphs are constructed from edge/regional image information and a priori surface smoothness and interaction constraints) Median filter Modification of Automated (3 pixels) in the Koozekanani transverse algorithm direction
2D spectral OCT data reducing from 12 patients (24 anisotropic diffusion filter 3D image datasets) with unilateral anterior ischemic optic neuropathy (AION).
Various retinal layers/ segments (5)
Garvin, M., 2008, TDOCT (Stratus OCT)
Progressive segmentation
Gradient Automated approach using dynamic programming (2D approach)
149 Gaussian glaucomatous smoothing patients and filtering 47 normal patients
Various retinal layers/ segments
Tan, O., 2008, TDOCT (Stratus OCT)
First reported approach for the automated 3D segmentation of intraretinal layers.
Mean segmentation time (after alignment/reg istration) was 4.1 ± 0.9 min (using a Windows XP workstation with a 3.2GHz Intel Xeon CPU).
Segmentation Not reported results confirmed that glaucoma primarily affects the thickness of the inner retinal layers (RNFL, GCL, IPL) in the macula.
Segmentation Quantitative Not reported accuracy of the outer measurements of retinal layers was the outer retinal assessed by morphology were used to aid comparing interpretation of the automated vs. manually assisted scattering bands measurements posterior to IS/OS juntion visualized on UHR OCT images Quality control of Automatic Not reported the automatic segmentation segmentation facilitated a revealed reasonable thickness map results in over 90% showing the configuration of of examinations intraretinal fluid in much higher detail. Algorithm is able to track retinal and subretinal changes in patients with RPE detachments
Repeatability of thickness measurements showed the potential of using the thickness of both retina and inner retinal layers for tracking glaucoma progression. Overall mean unsigned border positioning error was 6.1 ± 2.9 μm, a result comparable to the interobserver variability (6.9 ± 3.3 μm)
A Review of Algorithms for Segmentation of Retinal Image Data Using Optical Coherence Tomography
41
OCT data Lowpass from 6 AMD filtering eyes (a total of 228 SDOCT Bscans)
OCT data Fixed pattern from healthy noise removal volunteers (1) and patients with AMD (1) and pseudovitellif
Farsiu, S., 2008, Drusen area, SDOCT (Bioptigen RPE and RNFL Inc., Durham, NC) inner border
Gotzinger, E., 2008, PSOCT
RPE
N/A OCT data from 9 pathological subjects and 2 normative subjects
Tolliver, D., 2008, Difficult SDOCT (3D OCT boundary Cirrus) contours (4)
Directional filtering (2D filter with a wedgeshaped pass band)
Data from healthy subjects: TDOCT (15) and SDOCT (10)
Automated
Two Automated algorithms: 1) Based on retardation data 2)Based on local variations of the polarization
DOCTRAP Automated algorithm /Semibased on a Automated modified implementation of the GVF based deformable snake method
Spectral rounding
Automated 2D edge detection algorithm based on the first derivative of Gaussian in the vertical direction
2D mean filter Fuzzy Cmeans Automated (kernel size clustering 7x5) for technique speckle denoising along with complex diffusion filtering to aid segmentation.
Bagci, AM., Various retinal 2008/TDOCT layers/ (Stratus OCT) & segments (6) SDOCT (RTVue 100 OCT, Optovue, Freemont, CA)
Mayer, M., 2008, RNFL thickness OCT data SDOCT (Spectralis from 5 HRA+OCT, normal and 7 Heidelberg glaucoma Engineering) eyes 97% of the upper and 74% of the lower RNFL layer boundary points lied within a 2 pixel range from the manual segmentation of the evaluation data set
None
Not reported
Difference between automated and manual segmentation was ≤ 4.2 μm, and almost identical to the difference between manual measurements by three observers (≤4.4 μm) For evaluation The aggregate purposes, manual accuracy range for the detected segmentation was performed boundaries was [85for 4 randomly 99]% selected scans per subject Manual Not reported correction tool to modify the automated segmentation results. AMD images were also manually segmented by 2 experts for evaluation purposes.
None
None
About 6.5 seconds to automatically segment, display and record drusen locations (image size: 512x1000 pixels) on a Intel CentrinoDuo 2.4 GHz CPU Algorithm 1: Algorithms facilitated a better 8.3 minutes (without visualization and preprocessing quantification of RPE thickening and and cornea RPE atrophies when compensation) compared to for volumes
Algorithm takes advantage of differences of drusen substructures revealed by SDOCT facilitating drusen area and volume measurements
Algorithm is able to Not reported track complex boundary contours in the presence of retinal pathology
NFL segmentation 45 seconds on method of circular a 2Ghz Pentium IV for OCT scans that is applicable to normal a 512x496 circular Bscan as well as pathological data, different patients and varying scanner settings without parameter adaptation Edge detection uses: Not reported 1) the correlation between adjacent Ascans, 2) graylevel mapping technique to overcome uneven tissue reflectivity and variance across subjects
42 Image Segmentation
state calculated using Stokes vector analysis
Automated / SemiAutomated (user initialization prior to segmentation)
Automated
Intensity signal Automated based thresholding segmentation
Speckle noise Modified active and other contour typical artifacts algorithm by in OCT images using 1) sparse are handled by dynamic using an programming adaptive method and 2) vectorvalued twostep kernel kernel function based in the precise optimization layer boundary scheme optimization step 20 OCT None Modified images from Chan–Vese’s rodent energymodels (4) of minimizing retinal active contour degeneration algorithm (augmented with shape prior and weight terms)
OCT data from healthy and diseased rodent retina
Fabritius, T., 2009, Inner and outer Data from a No denoising experimental borders of the healthy required SDOCT retina volunteer and from patients with ARMD (1) and PCV(1)
Yazdanpanah, A., Various retinal 2009, experimental layers (5) FDOCT
Mishra, A., 2009, Various retinal experimental HR layers OCT (high speed)
orm dystrophy (1)
First multiphase Not reported framework to segment OCT data that incorporates a circular shape prior based on expert anatomical knowledge of the retinal layers, avoiding the need for training Manual Error smaller than 5 Incorporated 3D About 1721 segmentation for pixels in 99.7 (99.2) intensity seconds for evaluation % of scans for the 140 frames information to purposes RPE (ILM) with 1022 improve the segmentation intensity based depth scans segmentation. ILM and RPE can be segmented directly from the OCT data without massive preprocessing in a very faster manner.
Not reported
Manual An average dice segmentation for similarity coefficient evaluation of 0.85 was obtained purposes when measuring the area similarity between the manual and automated segmentation
None
Algorithm 2: 31 minutes using an AMD Athlon 64X2 Dual Core Processor 4800+, 2.41 GHz, 2 GB RAM Algorithm achieves 5 seconds per accurate intraimage on an retinal segmentation Intel Pentium on retinal OCT 4 2.4 GHz images under low machine with image contrast and 1 GB of RAM. in the presence of irregularly shaped structural features
algorithms based on with 60 Bintensity scans (100 Aimages. scans each)
A Review of Algorithms for Segmentation of Retinal Image Data Using Optical Coherence Tomography
43
Various retinal OCT data Median filter layers/segments from a dark(8) adapted human subject
Retinal contours Various Median filter (inner, outer and artificial others) images and one OCT scan
Tumlinson, 2009, experimental FDOCT
Koprowski, R., 2009, SDOCT (Copernicus)
OCT data None (200x200x102 4 voxels) from 34 glaucoma patients
ONH cup and rim
Abramoff, 2009, SDOCT (CirrusOCT)
OCT data Anisotropic from one diffusion filter exenteration patient and 4 glaucoma patients
ONH drusen
Yi, K., 2009, experimental SDOCT
Thresholded Automated edgefinding algorithm that first applies directionally biased filtering, finds positive and negative edges using derivative filters, and assigns those edges to layer boundaries based on a set of rules. Random Automated Contour detection algorithm based on area analysis
Commercially Automated available software from Amira interfaced with ITK's open source C++ algorithms Multiscale 3D Automated graph search algorithm to segment three retinal surfaces and a voxel column classification algorithm using a kNN classifier
Not reported
None
First algorithm that Not reported shows a high correlation between segmentation results of the ONH cup and rim from SDOCT images and planimetry results obtained by glaucoma experts on the same eye
First segmentation Not reported approach that is able to facilitate 3D imaging of the shape, size and location of ONH drusen
Algorithm exploited Comparable to information in the calculation adjacent Bscans of the fast making simpler the Fourier segmentation task transform used to investigate (FFT) for for the first time translation of depthresolved slow raw spectral retinal intrinsic data into optical signals structural tomograms (volume data: 512x 512x1024 on a Condor computer cluster) Error analysis only Contours extraction Not reported. reported for artificial depend on But authors images parameter selection mentioned computation time is a major drawback of the algorithm
A nine kNN The correlation of classifier is used algorithm c/d to refine/smooth ratio to experts 1, 2, the appearance of and 3 segmentation was 0.90, 0.87, and results. In 0.93, respectively. addition, the correlation between algorithmdetermined cup todisc (c/d) ratio and planimetryderived c/d by 3 experts was calculated for evaluation purposes None None
None
44 Image Segmentation
Various retinal layers (8)
466 Bscans Dualtree from 17 eyes complex wavelet (DTCW) denoising
Various retinal layers (7)
ONH cup and rim
Chiu, SJ., 2010/SDOCT (Bioptigen Inc., Durham, NC)
Lee, K., 2010, SDOCT (Cirrus HDOCT) 27 SDOCT scans (200x200x102 4 voxels) from 14 glaucoma patients
Median filtering and averaging Based smoothing
Multiscale 3D Automated graph search algorithm to segment four retinal surfaces and a
Automated
Statistical Automated model based on texture and shape that captures the variance of the training data used to segment unseen data. graphtheoretic Automated approach
Data from 10 Gaussian filter Graphbased algorithm healthy along with and dynamic subjects rectangular averaging filter programming (3 × 19 pixels )
Zhihong, H., 2010, neural canal None OCT data SDOCT (Cirrus opening (NCO) from HDOCT) and cup at 34 patients the level of (68 eyes) with RPE/Bruch s glaucoma or glaucoma membrane suspicion complex
Kajic, V. , 2010, 3DOCT Evaluation against a large set of manual segmentations (a difference of only 2.6% against the interobserver variability)
Local fitting method using the convex hulls of the segmentation to refine/smooth the contours of the ONH rim and cup. In addition, two glaucoma experts annotated the cup and rim area in stero color photographs using planimetry to create the reference standard for evaluation
9.74 seconds per image (on average) for 108 Bscans (64bit OS, Intel Core2 Duo CPU at 2.53 GHz, and 4 GB RAM). Unsigned error for Fast and fully About 132 the optic disc cup automatic method S (80 seconds was 2.52+/0.87 to segment the optic to segment 4 pixels (0.076+/0.026 disc cup intraretinal mm) and for the and rim in 3D SD surfaces using neuroretinal rim OCT volumes the was 2.04+/0.86 multiscale 3D pixels (0.061+/0.026 graph search mm). approach, and 52 seconds for the feature extraction and classification) on a PC (Microsoft Windows XP Professional x64 edition,
Not reported
First time, Not reported an error measure is computed from a large manually segmented data set (segmented twice by different operators)
Bspline used to Mean unsigned and Algorithm is able to signed border detect natural ONH smooth/refine anatomic structures the NCO and cup differences of 2.81 ± 1.48 pixels of and optic cup boundaries. In (0.084 ± 0.044 mm) addition, computeraided and 0.99 ± 2.02 planimetry was pixels (0.030 ± 0.061 performed by 3 mm) respectively for experts to create a NCO segmentation reference standard Manual Fully automatic Automatic segmentation for algorithm differed initialization that evaluation from one of the bypasses the need purposes manual graders by for manual an average of 0.95 endpoint selection pixels.
None
A Review of Algorithms for Segmentation of Retinal Image Data Using Optical Coherence Tomography
45
Table 1. Overview of OCT segmentation approaches.
Quellec , G., 2010, Various retinal SDOCT (Cirrus, layers (10) along HDOCT) with fluidfilled regions OCT data Wavelets from 13 normal eyes and from 23 eyes with CNV, intra, and subretinal fluid and pigment epithelial detachment
OCT data (38 None Scans) from 38 individuals, 19 glaucoma patients and 19 controls
Yang, Q., 2010, SDOCT (Topcon 3D OCT1000)
Various retinal layers (9)
Bilateral filter OCT data and median from 4 filter normal healthy eyes
Lu, S., 02010, Various retinal SDOCT (Spectralis layers (5) OCT, Heidelberg Engineering)
Multiscale 3D graph search approach
Automated
Algorithm Automated based on the Canny's edge detector for the nonvessel sections. Linear interpolation is used for the layer boundaries in the vessel sections Algorithm Automated based on a dualscale gradient information and a shortest path Search using dynamic programming Algorithm is able to segment lowintensity and lowcontrast OCT images in a very short time without degrading the accuracy In addition, preextraction of vessel locations, which is not a trivial operation, is unnecessary
Algorithm first detects the retinal blood vessels and then split the OCT image into multiple vessel and nonvessel sections.
Intel Core 2 Duo CPU at 3.00 GHz, 4GB RAM) Not reported
About 45 seconds for each 3D volume (480x512x128 voxels) in normal segmentation processing mode and 16 seconds in fast segmentation mode using Ascan reduction technique Manual Mean unsigned Confirmed that 70 seconds per segmentation for surface positioning useful 3D eye for 10 evaluation errors were less than textural information layer detection purposes 6 μm. can be also extracted on a from SDOCT scans 200x1024x200 to aid local retinal voxel volume using a abnormality standard PC at detection. 2.4 GHz; (800 MB of RAM)
Segmentation errors for the RNFL were less than 5 μm on average
Manual Overall the ICC of segmentation for each boundary was evaluation above 0.94, purposes the mean coefficient of variation was less than 7.4%, and the mean standard deviation was less than 2.8μm.
Manual segmentation for evaluation purposes (16 OCT images)
purposes
46 Image Segmentation
A Review of Algorithms for Segmentation of Retinal Image Data Using Optical Coherence Tomography
47
8. References Abràmoff, MD., Lee, K., Niemeijer, Wallace, L., Alward, M., Greenlee, EC., Garvin, MK., Sonka, M. & Kwon, YH. (2009). Automated Segmentation of the Cup and Rim from Spectral Domain OCT of the Optic Nerve Head. Invest. Ophthalmol. Vis. Sci. 2009 50: 57785784 Ahlers, C., Simader, C., Geitzenauer, W., Stock, G., Stetson, P., Dastmalchi, S & SchmidtErfurth, U. (2008). Automatic segmentation in threedimensional analysis of fibrovascular pigmentepithelial detachment using highdefinition optical coherence tomography. Br. J. Ophthalmol. 92, 197203 Aguirre AD., Hsiung, P., Ko, P., Hartl, TH.& Fujomoto JG. (2003) Highresolution optical coherence microscopy for highspeed in vivo cellular imaging. Optics Lett. 20642066 28 Alirezaie, J., Jernigan, M E., Nahmias, C., (1997). Neural NetworkBased Segmentation of Magnetic Resonance Images of the Brain, IEEE Trans. on Nuclear Science, 44:1948 Andreasen, NC., Rajarethinam, J., Cizadlo, T. Arndt, S., Swayze, VW., Flashman, LA., O'Leary. DS, Ehrhardt, JC. & Yuh, WT. (1996) Automatic atlasbased volume estimation of human brain regions from MR images. J. Comp. Assist. Tom., 20:98–106 Bagci, AM., Shahidi, M., Ansari, R., Blair, M., Blair, NP & Zelkha, R. (2008). Thickness profiles of retinal layers by optical coherence tomography image segmentation. Amer. J. Ophthalmol., vol. 146, no. 5, pp. 679–687 Baroni, M., Fortunato, P. & Torre, AL. (2007). Towards quantitative analysis of retinal features in optical coherence tomography. Medical Engineering and Physics 29(4), 432–441 Bizheva, K., Pflug, R., Hermann, B., Povazay, B., Sattmann, H., Qiu, P., Anger, E., Reitsamer, H., Popov, S., Taylor, JR., Unterhuber, A., Ahnelt, P. & Drexler, W. (2006). Optophysiology: depthresolved probing of retinal physiology with functional ultrahighresolution optical coherence tomography. Proc. Natl. Acad. Sci. USA 28, 103, 506671 Boppart, SA., Drexler, W., Morgner, U., Kirtner, FX. & Fujimoto, JG. (1999). Ultrahigh Resolution and Spectroscopic OCT Imaging of Cellular Morphology and Function, Proceedings of InterInstitute Workshop on In Vivo Optical Imaging at the National Institutes of Health, A. H. Gandjbakhche, Ed., pp. 56–61 Boykov, Y., & FunkaLea, G. (2006). Graph cuts and efficient ND image segmentation. Int. J. Comput. Vision, vol. 70, pp. 109–131 Boyer, KL., Herzog, A., Roberts, C. (2006). Automatic recovery of the optic nervehead geometry in optical coherence tomography. IEEE Trans Med Imaging. May; 25(5):55370. PubMed PMID: 16689260. Brezinski, ME., Tearney, GJ., Bouma, BE., Izatt, JA., Hee, MR., Swanson, EA., Southern, JF., & Fujimoto, JG. (1996). Optical coherence tomography for optical biopsy: properties and demonstration of vascular pathology. Circulation, 93 (6): pp. 12061213. Cabrera Fernández, D., Villate, N., Puliafito, CA. & Rosenfeld, PJ. (2004). Comparing total macular volume changes measured by Optical Coherence Tomography with retinal lesion volume estimated by active contours. Invest. Ophthalmol. Vis. Sci. 45: EAbstract 3072.
48
Image Segmentation
Cabrera Fernández, D. (2005a). Delineating fluidfilled region boundaries in optical coherence tomography images of the retina,” IEEE Trans. Med. Imag.,vol. 24, no. 8, pp. 929–945 Cabrera Fernández, D. Salinas, HM & Puliafito, CA. (2005b). Automated detection of retinal layer structures on optical coherence tomography images. Optics Express 13(25), 10,200–10,216 Cabrera Fernández, Somfai, GM., Tátrai, E., Ranganathan, S., Yee, DC., Ferencz, M. & Smiddy, WE. (2008) Potentiality of Intraretinal Layer Segmentation to Locally Detect Early Retinal Changes in Patients With Diabetes Mellitus Using Optical Coherence Tomography Invest. Ophthalmol. Vis. Sci. 49: EAbstract 2751. Cabrera DeBuc, D. & Somfai, GM. (2010). Early detection of retinal thickness changes in diabetes using Optical Coherence Tomography,” Med. Sci. Monit. 16(3), MT15– MT21 Cense, B., Chen, TC., Hyle, P.B, Pierce,MC &de Boer, JF. (2002). In vivo depthresolved birefringence measurements of the human RNFL by PSOCT. Optics Lett. 27, 16101612 Cense, B., Nassif, N., Chen, T., Pierce, M., Yun, SH., Park, B., Bouma, B., Tearney, B. & de Boer, JF (2004). Ultrahigh resolutionhighspeed retinal imaging using spectraldomain optical coherence tomography, Opt. Express. 12(11), 2435–2447 (2004). Chen, Z., Milner, TE., Dave, D. & Nelson JS. (1997). Optical Doppler tomographic imaging of fluid flow velocity in highly scattering media. Optics Lett. 22, 6466 Cheng, K S., Lin, J S. & Mao, CW. (1996). The Application of Competitive Hopfield Neural Network to Medical Image Segmentation. IEEE Trans. on Med. Img., 15:5607 Chiu, SJ.,Li, CT., Nicholas, P.,Toth, CA., Izatt, JA, & Sina Farsiu. (2010). Automatic segmentation of seven retinal layers in SDOCT images congruent with expert manual segmentation. Optic Express. Vol. 18, No. 18. pp. 1941319428 Choma, MA., Sarunic, MV., Yang, C., Izatt, JA. (2003). Sensitivity advantage of swept source and Fourier domain optical coherence tomography. Optics Express. 11: 21832189 Cootes, T F., Edwards, G J., Taylor, C J. (2001). Active Appearance Models. IEEE Trans. Pattern Analysis and Machine Intelligence, 23: 68185 deBoer, JF., Milner, TE., vanGemert, MJC. & Nelson, JS. (1997). Twodimensional birefringence imaging in biological tissue by polarizationsensitive optical coherence tomography. Optics Letters, 22(12), 934936 deBoer, JF., Shrinivas, SM., Malekafzali, A., Chen, Z. & Nelson, JS. (1998). Imaging thermally damaged tissue by polarization sensitive OCT. Optics Express. 3, 212218 de Boer, JF., Cense, B., Park, BH., Pierce, MC., Tearney, GJ., & Bouma, BE. (2003). Improved signaltonoise ratio in spectraldomain compared with timedomain optical coherence tomography. Optics Letters, 28 (21), 20672069 Ding, Z., Zhao, Y.,Ren, H., Nelson, JS. & Chen, Z. (2002). realtime phaseresolved OCT and optical Doppler tomography. Optic Express. 10, 236245 Drexler, W. & Fujimoto, JG. (2008). Stateoftheart retinal optical coherence tomography, Prog. Retin. Eye Res. 27(1), 45–88 Fabritius, T., Makita, S., Miura, M., Myllyla, R. & Yasuno, Y. (2009). Automated segmentation of the macula by optical coherence tomography. Opt. Express 17(18), 1565915669
A Review of Algorithms for Segmentation of Retinal Image Data Using Optical Coherence Tomography
49
Farsiu, S., Chiu, SJ., Izatt, JA., Toth, CA. (2008). Fast detection and segmentation of drusen in retinal optical coherence tomography images. Photonics West, Ophthalmic Technologies. San Jose, CA; 68440D6844168412 Fercher, AF., Hitzenberger, CK., Kamp, G. & Elzaizt, SY. (1995). Measurement of intraocular distances by backscattering spectral interferometry. Opt. Commun. 117(12), 43–48 Fercher, AF., Drexler, W., Hitzenberger, C.K. (2003). Optical coherence tomography. Principles and applications. Rep Prog Phys. 66: 239303 Ford, L. & Fulkerson, D. (1956). Maximal flow through a network. Canadian Journal of Mathematics. 8. 399404 Fuller, AR., Zawadzki, RJ., Choi, S., Wiley, DF., Werner, JS. & Hamann, B. (2007). Segmentation of Threedimensional Retinal Image Data. IEEE Transactions on Visualization and Computer Graphics 13(6), 1719–1726 Fujimoto, JG. (2003). Optical coherence tomography for ultrahigh resolution in vivo imaging. Nature Biotechnology, Vol. 21, No. 11, pp 136167 Garvin, MK., Abramoff, MD., Kardon, R., Russell, SR., Wu, X. & Sonka, S. (2008). Intraretinal Layer Segmentation of Macular Optical Coherence Tomography Images Using Optimal 3D Graph Search. IEEE Transactions on Medical Imaging 27(10), 1495 – 1505 George, A., Dillenseger, JA., Weber, A. & Pechereau, A. (2000). Optical coherence tomography image processing. Investigat. Ophthalmol. Vis.Sci., vol. 41, pp. S173– S173 Gonzalez, RC. & Woods, RE. (1992) Digital Image Processing. AddisonWesley Götzinger, E., Pircher, M., Geitzenauer, W., Ahlers, C., Baumann, B., Michels, S., SchmidtErfurth, U. & Hitzenberger, CK. (2008). Retinal pigment epithelium segmentation by polarization sensitive optical coherence tomography. Opt. Express 16(21), 16410–16422 Greig, D., Porteous, B. & Seheult, A. (1989). Exact maximum a posteriori estimation for binary images. Journal of the Royal Statistical Society 51(2) (1989) 271279 Gregori, G. & Knighton, RW. (2004). A Robust Algorithm for Retinal Thickness Measurements using Optical Coherence Tomography (Stratus OCT). Invest. Ophthalmol.Vis.Sci.45:EAbstract3007 Gregori, G., Knighton, RW., Jiao, S., Huang, X., Rosenfeld, PJ. & Puliafito, CA. 3–D OCT Maps of Retinal Pathologies. (2005) Invest. Ophthalmol. Vis. Sci. 2005 46: EAbstract 1055. Gregori, G., Gregori, NZ., Knighton, RW., Lujan, BJ. Puliafito, CA. & Rosenfeld, PJ. (2008). Imaging Drusen With Spectral Domain Optical Coherence Tomography. Invest. Ophthalmol.Vis.Sci.49:EAbstract4234 Greig, D., Porteous, B., Seheult, A. (1989). Exact maximum a posteriori estimation for binary images. Journal of the Royal Statistical Society. 51(2), 271279 Haeker, M., Abràmoff, M., Kardon, R., Sonka, M. (2006). Segmentation of the surfaces of the retinal layer from OCT images. Med Image Comput Comput Assist Interv. 9 (Pt 1):8007. PubMed PMID: 17354964. Haralick, RM. & Shapiro, LG. (1985). Image segmentation techniques. Comput. Vis. Graph. Im. Proc., 29:100–132 Hausler, G. & Lindner, MW. (1998). Coherence Radar and Spectral Radar. New tools for dermatological diagnosis. J Biomed Opt 1998; 3: 2131
50
Image Segmentation
Hee, MR., Izatt, JA., Swanson, EA., Huang, D., Schuman, JS., Lin, CP., Puliafito, CA. & Fujimoto, JG. (1995a). Optical coherence tomography of the human retina. Arch. Ophthalmol. 113, 32532 Hee, MR., Izatt, JA., Swanson EA., Huang, D., Schuman, JS., Puliafito, CA. & J. G. Fujimoto. (1995b). Optical coherence tomography for micronresolution ophthalmic imaging. IEEE Eng. Med. Biol., vol. 14, pp. 67–76 Hee, MR. (1997). Optical Coherence Tomography of the eye. Thesis (PhD). Massachusetts Institute of Technology, Source DAIB 58/04, p. 1952. Held, K., Kops, ER., Krause, BJ., Wells, WM. Kikinis, R., & MullerGartner, HW. (1997). Markov random field segmentation of brain MRI images. IEEE T. Med. Imag., 16(6) Hermann JM., Pitris, C., Bouma, BE., Boppart, SA., Jesser, CA., Stamper, DL., Fujimoto, JG., Brezinski, ME. (1999). Highresolution imaging of normal and osteoarthritic cartilage with OCT. J. Rheumatol. 26, 627635 Hermann, B., Fernandez, EJ., Unterhubner, A. Sattmann,H., Fercher, AF., Drexler, Prieto, WPM., & Artal, P. (2004). Adaptiveoptics ultrahighresolution optical coherence tomography. Opt. Lett. 29, 21422144 Hermann, B., Považay, B., Unterhuber, A. Lessel, M., Sattmann, H., SchmidtErfurth, U. & Drexler, W. (2006). Optophysiology of the Human RetinaWith Functional Ultrahigh Resolution Optical Coherence Tomography. Invest. Ophthalmol. Vis. Sci. 2006 47: EAbstract 1672 Herzog, A., Boyer, KL. & Roberts, C. (2004). Robust Extraction of the Optic Nerve Head in Optical Coherence Tomography. CVAMIAMMBIA, LNCS 3117, pp. 395–407 Hu, Z., Abramoff, MD., Kwon, YH., Lee, K., Garvin, M. (2010). Automated Segmentation of Neural Canal Opening and Optic Cup in 3D Spectral Optical Coherence Tomography Volumes of the Optic Nerve Head. Invest Ophthalmol Vis Sci. Jun 16. [Epub ahead of print] PubMed PMID: 20554616 Huang, D., Swanson, EA., Lin, CP., Schuman, JS., Stinson, GW., Chang, W., Hee, MR., Flotte,T., Gregory, K., Puliafito, CA. & Fujimoto JG. (1991). Optical coherence tomography. Science 254, 1178–81 Huang, Y., Cideciyan, AV., Papastergiou, GI., Banin, E., SempleRowland, SL., Milam, AH., Jacobson, SG. (1998). Relation of optical coherence tomography to microanatomy in normal and rd chickens. Invest. Ophthalmol. Vis. Sci. 39: 24052416. Ishikawa, H., Piette, S., Liebmann, JM. & Ritch, R. Detecting the inner and outer borders of the retinal nerve fiber layer using optical coherence tomography. (2002). Graefe’s Archive for Clinical and Experimental Ophthalmology. May;240(5):36271 Ishikawa, I., Stein, DM., Wollstein, G., Beaton, S., Fujimoto, JG. & Schuman, JS. (2005). Macular Segmentation with Optical Coherence Tomography. Investigative Ophthalmology and Visual Science (IOVS) 46(6), 2012–2017 Izatt, JA., Hee, MR., Swanson, EA., Lin,CP., Huang, D., Schuman, JS., Puliafito, CA. & Fujimoto, JG. (1994a). Micrometerscale resolution imaging of the anterior eye in vivo with optical coherence tomography. Arch. Ophthalmol. 112, 15849 Izatt, JA., Hee, MR., Owen, GM.,Swanson, EA. & Fujimoto, JG. (1994b). Optical coherence microscopy in scattering media. Optics Lett. 19. 590592 Izatt, JA., Kulkami, MD., Wang, HW., Kobayashi, K., & Sivak MV. (1996). Optical coherence tomography and microscopy in gastrointestinal tissues. IEEE J. Selected Topics Quantum Electron. 2, 10171028
A Review of Algorithms for Segmentation of Retinal Image Data Using Optical Coherence Tomography
51
Izatt, JA., Kulkami, MD.,Yazdanfar, S., Barto, JK. & Welch, AJ. (1997). In vivo bidirectional color Doppler flow imaging of picoliter blood volumes using OCT. Optics Lett. 22, 14391441 Kajić, V., Považay, B., Hermann, B., Hofer, B., Marshall, D., Rosin, PL. & Drexler, (2010). W. Robust segmentation of intraretinal layers in the normal human fovea using a novel statistical model based on texture and shape analysis. Opt. Express 18(14), 14730– 14744 Kocaoglu, OP., Uhlhorn, SR., Hernandez, E., Juarez, RA., Will, R Parel, JM., & Manns, F. (2007). Simultaneous Fundus Imaging and Optical Coherence Tomography of the Mouse Retina.Invest. Ophthalmol. Vis. Sci. 48: 12831289. Kolmogorov, V. & Zabih, R. (2004). What energy functions can be minimized via graph cuts?, IEEE PAMI, vol. 26, pp. 147–159 Koozekanani, D., Boyer, KL.,Roberts, C. (2001). Retinal Thickness Measurements in Optical Coherence Tomography Using a Markov Boundary Model. Medical Imaging, IEEE Transactions on 20(9), 900–916 Koprowski, R. & Wrobel, Z. (2009). Layers recognition in tomographic eye image based on random contour analysis. Computer recognition Syst. 3, AISC 57, Kurzynski, M. & Wozniak, M. (Eds), pp. 471478, SpringerVerlag, Berlin Heidelberg Lancaster, JL., Rainey, LH., Summerlin, JL., Freitas, CS., Fox, PT., Evans, AC., & Mazziotta, JC. (1997). Automated labeling of the human brain: A preliminary report on the development and evaluation of a forwardtransform method. Human Brain Mapping., 5:238242 Lee, K., Abramoff, MD., Niemeijer, M., Garvin, MK., Sonka, M. (2010). 3D segmentation of retinal blood vessels in spectraldomain OCT volumes of the optic nerve head. Proc. of SPIE Medical Imaging: Biomedical Applications in Molecular, Structural, and Functional Imaging, vol. 7626, p. 76260V, 2010 Leitgeb, RA., Schmetterer, I., Drexler, W., Fercher, AF., Zawadzki, RJ. & Bajraszewskit, J. (2003). Realtime assessment of retinal blood flow with ultrafast acquisition by color Doppler Fourier domain optical coherence tomography. Optics Express; 11: 31163121 Li, SZ. (1995) Markov random field modeling in computer vision. SpringerVerlag, ISBN:4431701451, London, UK Liu,Y., Chen, M., Ishikawa, H., Wollstein, G., Schuman, JS. & Rehg, JM. (2010). In: Automated Macular Pathology Diagnosis in Retinal OCT Images Using MultiScale Spatial Pyramid with Local Binary Patterns.T. Jiang et al. (Eds.), pp. 1–9, MICCAI 2010, Part I, LNCS 6361, SpringerVerlag, Berlin, Heidelberg Lu, S., Cheung, C., Liu, J., Lim, S., Leung, C. & Wong, T. (2010). Automated Layer Segmentation of Optical Coherence Tomography Images. IEEE Trans Biomed Eng.; 57(10), 26058 Maintz, JBA. & Viergever., MA. (1998). A survey of medical image registration. Med. Im. Anal., 2:1–36 Mayer, MA., Tornow, RP., Bock, R., Hornegger, J. & Kruse, FE. (2008). Automatic Nerve Fiber Layer Segmentation and Geometry Correction on Spectral Domain OCT Images Using Fuzzy CMeans Clustering. Invest. Ophthalmol. Vis. Sci. 49: EAbstract 1880 Michelson AA. & Morley EW. (1887). Philos. Mag. S.5, 24 (151), 449463
52
Image Segmentation
Mishra, A., Wong, A., Bizheva, K. & Clausi, DA. (2009). Intraretinal layer segmentation in optical coherence tomography images. Opt. Express 17(26), 23719–23728 Mitsui, T. (1999). Dynamic range of optical reflectometry with spectral interferometry. Jap J Appl Phys (Part 1 Regular Papers Short Notes & Review Papers); 38: 61336137 Mohamed, NA., Ahmed, MN., Farag A. (1998). Modified Fuzzy Cmean in Medical Image Segmentation. Proceedings of the 20th Annual International Conference of the IEEE, 3:137780. Morgner, U., Drexler, W., Kartner, FX., Li, XD., Pitris, C., Ippen, EP., Fujimoto, JG. (2000). Spectroscopic optical coherence tomography. Optics Lett. 25, 111113 Mujat, M., Chan, R., Cense, B., Park, B., Joo, C., Akkin, T., Chen, T. & de Boer, J. (2005). Retinal nerve fiber layer thickness map determined from optical coherence tomography images. Optics Express 13(23). 9480–9491 Novotny, HR & Alvis, DL. (1961). A method of photographing fluorescence in circulating blood in the human retina. Circulation. 24(1):82–86 Osher, S. & Sethian, J. (1988). Fronts Propagating with curvaturedependent speed: algorithms based on HamiltonJacobi formulations. J. Comp. Phys., 79: 1249 Pal, NR. & Pal., SK. (1993). A review on image segmentation techniques. Patt. Rec., 26:1277– 1294 Pappas, TN. (1992). An adaptive clustering algorithm for image segmentation. IEEE Trans. Signal Processing. 40(4):901914 Pathak, SD., Grimm, PD., Chalana, V. & Kim, Y. (1998). Pubic arch detection in transrectal ultrasound guided prostate cancer therapy. IEEE T. Med. Imag., 17:762–771 Podoleanu, AG., Dobre, GM., Cucu, RG., Rosen, R., Garcia, P., Nieto, J., Will, D., Gentile, R., Muldoon, T., Walsh, J., Yannuzzi, L.A., Fisher, Y., Orlock, D., Weitz, R., Rogers, J.A., Dunne, S., Boxer, A. (2004). Combined multiplanar optical coherence tomography and confocal scanning ophthalmoscopy. J. Biomed. Opt. 9, 86–93 Puliafito CA. (1996). Optical Coherence Tomography of Ocular Diseases. Thorofare, NJ: SLACK Inc Quellec, G., Lee, K., Dolejsi, M., Garvin, MK., Abràmoff, MD. & Sonka, M. (2010). Threedimensional analysis of retinal layer texture: identification of fluidfilled regions in SDOCT of the macula. IEEE Trans. Med. Imaging 29(6), 1321–1330 Rajapakse, C., Giedd, JN. & Rapoport, JL. (1997). Statistical approach to segmentation of singlechannel cerebral mr images. IEEE T.Med. Imag., 16:176186 Ren, H. (2002). Phaseresolved functional Optical Coherence Tomography: simultaneous imaging of in situ tissue structure, blood flow velocity, standard deviation, birefringence, and Stokes vectors in human skin. Optics Lett.27, 17021704 Ruggeri, M., Wehbe, H., Jiao, S., Gregori, G., Jockovich, ME., Hackam, A., Duan, Y. & Puliafito, CA. (2007) In Vivo ThreeDimensional HighResolution Imaging of Rodent Retina with SpectralDomain Optical Coherence Tomography. Invest. Ophthalmol. Vis. Sci. 48: 18081814 Sahoo, PK., Soltani, S.,Wong, AKC. (1988). A survey of thresholding techniques. Comput Vision Graphics Image Process 41:233–260 Sarkar, S., Boyer, K. (1991). Optimal impulse response zero crossing based edge detectors. Computer Vision Graphics Image Process: Image Understanding 54, 224–243
A Review of Algorithms for Segmentation of Retinal Image Data Using Optical Coherence Tomography
53
Shahidi, M., Wang, Z. & Zelkha, R. (2005). Quantitative Thickness Measurement of Retinal Layers Imaged by Optical Coherence Tomography. American Journal of Ophthalmology 139(6), 1056 – 1061 Smith RT, Chan JK, Nagasaki T, Sparrow JR, Barbazetto I. (2005). Automated detection of macular drusen using geometric background leveling and threshold selection. Arch Ophthalmol 2005;123:200206 Sylwestrzak, M., Szkulmowski, M., Szlag, D. & Targowski, P. (2010). Realtime imaging for SDOCT with massively parallel data processing. Photonics Letter of Poland. vol. 2(3), 137139. Sziranyi, T., Zerubia, J., Czuni, L., Geldreich, D. & Kato, Z. (2000). Image Segmentation Using Markov Random Field Model in Fully Parallel Cellular Network Architectures. RealTime Imaging. 6, 195211 Szulmowski, M., Wojtkowski, M., Sikorski, B., Bajraszewski, B., Srinivasan, VJ., Szkulmowska, A., Kaluzny, JJ., Fujimoto, JG & Kowalczyk, A. (2007). Analysis of posterior retinal layers in spectral optical coherence tomography images of the normal retina and retinal pathologies. Journal of Biomedical Optics 12(4) Talairach, J. & Tournoux, P. (1988). CoPlanar Stereotaxic Atlas of the Human Brain. 3Dimensional Proportional System: An Approach to cerebral Imaging. Thieme Medical Publisher, Inc., Stuttgart, NY Tan, O., Li,. G., Lu, AT., Varma, R., Huang, D. (2008). Advanced Imaging for Glaucoma Study Group. Mapping of macular substructures with optical coherence tomography for glaucoma diagnosis. Ophthalmology. Jun;115(6):94956 Tolliver, DA., Koutis, Y., Ishikawa, H., Schuman, JS., & Miller, GL. (2008). Automatic Multiple Retinal Layer Segmentation in Spectral Domain OCT Scans via Spectral Rounding. Invest. Ophthalmol. Vis. Sci. 49: EAbstract 1878 von der Malsburg, C. (1981). The correlation theory of brain function. Technical report. MaxPlanckInstitute Biophysical Chemestry von Helmholtz, H. (1851) In World Encyclopedia. Oxford University Press, 2005. Wang, DL. & Terman. D. (1995). Locally excitatory globally inhibitory oscillator networks. IEEE Transactions on Neural Networks. 6(1):283286 Wang, DL. & Terman, D. (1997). Image segmentation based on oscillatory correlation. Neural Computing.9:805836. Wang, S., Zhu, W Y., Liang, Z P. (2001). ShapeDeformation: SVM Regression and Application to Medical Image Segmentation. Eight International Conference on Computer Vision, 2:20916 Westphal, V., Yazdanfar, S, Rollins, AM., Izatt, JA. (2002) Real time, highvelocity resolution color Doppler Optical Coherence Tomography. Optics Lett. 27, 3436 Wojtkowski M., Bajraszewski T., Targowski P., Kowalczyk A. (2003). Realtime in vivo imaging by highspeed spectral optical coherence tomography. Opt Lett. 2003 Oct 1;28(19):17457 Wong, RC. (2002). Visualization of subsurface blood vessels by color Doppler Optical Coherence Tomography in rats: before and after hemostatic therapy. Gastrointest. Endosc. 55, 8895 Yang, Q., Reisman, CA., Wang, z., Fukuma, Y., Hangai, m., Yoshimura, N., Tomidokoro, A., Araie, M., Raza, AR., Hood, DC. & Chan, K. (2010). Automated layer segmentation
54
Image Segmentation
of macular OCT images using dualscale gradient information. Optics Express. Vol. 18, No. 20, pp. 2129421307 Yazdanfar, S., Rollins, AM., Izatt, JA. (2003). In vivo imaging of human retinal flow dynamics by color Doppler Optical Coherence Tomography. Arch. Ophthalmol. 121, 235239 Yazdanpanah, A., Hamarneh, G., Smith, B., Sarunic, M. (2009). Intraretinal layer segmentation in optical coherence tomography using an active contour approach. Med Image Comput Comput Assist Interv. 12(Pt 2):64956 Yezzi ,A., Tsai, A., Willsky, A. (1999). A statistical approach to snakes for bimodal and trimodal imagery. Proceedings of the seventh IEEE international conference on computer vision, pp. 898–903.Washington, DC: IEEE Computer Society Yi, K., Mujat, M., Park, BH., Sun, W., Miller, JW., Seddon, JM., Young, LH., de Boer, JF. & Chen, TC. (2009). Spectral domain optical coherence tomography degeneration for quantitative evaluation of drusen and associated structural changes in nonneovascular agerelated macular. Br J Ophthalmol. 93: 176181 Zawadzki, RJ., Jones, SM., Olivier, SS., Zhao, M., Bower, BA., Izatt, JA., Choi, S., Laut, S. & Werner, JS. (2005). Adaptiveoptics optical coherence tomography for highresolution and highspeed 3D retinal in vivo imaging. Optics Express. Vol. 13, No. 21, pp. 85338546 Zhao, Y. (2000) Phaseresolved Optical Coherence Tomography and Optical Doppel Tomography for imaging blood flow in human skin with fast scanning speed and high velocity sensitivity. Optics Lett. 25, 114116
Part 2 Image Segmentation Methods
3 Image Segmentation through Clustering Based on Natural Computing Techniques Jose Alfredo F. Costa and Jackson G. de Souza Federal University of Rio Grande do Norte Brazil
1. Introduction Natural Computing (NC) is a novel approach to solve real life problems inspired in the life itself. A diversity of algorithms had been proposed such as evolutionary techniques, Genetic Algorithms and Particle Swarm Optimization (PSO). These approaches, together with fuzzy and neural networks, give powerful tools for researchers in a diversity of problems of optimization, classification, data analysis and clustering. Clustering methods are usually stated as methods for finding the hidden structure of data. A partition of a set of N patterns in a pdimensional feature space must be found in a way that those patterns in a given cluster are more similar to each other than the rest. Applications to clustering algorithms range from engineering to biology (Xu & Wunsch II, 2005; Xu & Wunsch, 2008; Jain et al., 1999). Image segmentation techniques are based on Pattern Recognition concepts and such a task aims to identify behavior in a data set. In the context of image segmentation, the data set represents image data, coded as follows: the light intensity value (the pixel data) represents a pattern, an item in the data set, and the color information is represented by columns (the feature vectors). Clustering techniques represent the nonsupervised pattern classification in groups (Jain et al., 1999). Considering the image context, the clusters correspond to some semantic meaning in the image, which is, objects. More than simple image characteristics, these grouped semantic regions represent information; and image segmentation is applicable in an endless list of areas and applications, for example: computeraided diagnosis (CAD) being used in the detection of breast cancer on mammograms (Doi, 2007), outdoor object recognition, robot vision, contentbased image, and marketplace decision support. Among the many methods for data analysis through clustering and unsupervised image segmentation is: Nearest Neighbor Clustering, Fuzzy Clustering, and Artificial Neural Networks for Clustering (Jain et al., 1999). Such bio and socialinspired methods try to solve the related problems using knowledge found in the way nature solves problems. Social inspired approaches intend to solve problems considering that an initial and previously defined weak solution can lead the whole population to find a better or a best so far solution. This chapter presents concepts and experimental results of approaches to data clustering and image segmentation using (NC) approaches. The main focus are on Evolutionary Computing, which is based on the concepts of the evolutionary biology and individualtopopulation adaptation, and Swarm Intelligence, which is inspired in the behavior of individuals, together, try to achieve better results for a complex optimization problem.
58
Image Segmentation
Genetic and PSO based Kmeans and fuzzy Kmeans algorithms are described. Results are shown for data clustering using UCI datasets such as Ruspini, Iris and Wine and for image texture and intensity segmentation using images from BrainWeb system. The remainder of the chapter is organized in the following form: section 2 describes Data Clustering and Image Segmentation; section 3 presents the stateoftheart in Image Segmentation techniques; section 4 presents Natural Computing; section 5 focuses on clustering using Natural Computing methods. Section 6 presents experimental results and discussion and section 8 gives the conclusions and final considerations.
2. Image segmentation and data clustering Digital Image Processing is an extremely important and fundamental task to image analysis, whose main task is the separation or isolation of image regions, reducing the data space to be analyzed. On monochromatic images, image segmentation algorithms are based on the following image gray level properties (Gonzalez & Woods, 2003): a. Discontinuity: the objective is to find hard changes on gray level, using this information as the method to edge detection; and b. Similarity: closest pixels are very similar. Some of the main challenges to the scientific community are related to the development of techniques that realize the automatic or unsupervised image segmentation. In controlled environment the image segmentation process is easily achieved than in a noncontrolled environment, where light and other circumstances affect physical process of image acquisition. Image segmentation applications contemplate many areas of Computer Graphics. In the case of Computer Vision, one of the objectives is make robots move in a semi or noncontrolled environment, and realize tasks like find and interact with specific objects. Another area of interest is the automatic vehicle guiding. On Image Understanding and Analysis there is Content Based Image Retrieval, that aims to develop efficient search engines that can find items on an image database by using a reference image, detecting similarities. The mathematical formulation of segmentation is defined as follows (Raut et al., 2009): Let I be the set of all image pixels, then by applying segmentation we obtain different unique nonoverlapping regions {S1 , S2 , S3 ,..., Sn } which, when combined, form I: n
∪ Si = I
i = 1, n
where Si ∩ S j = ∅
(1)
where: a. Si is a connected region, i = 1, 2, …, n b.
P ( Si ) = TRUE for i = 1, 2, …, n
c.
P Si ∪ S j =FALSE for i ≠ j
d.
P ( Si ) is a logical predicate defined over points in set Si .
(
)
Eq. 1 is a condition that indicates that segmentation must be complete: every pixel in the image must be covered by segmented regions, which must by disjoint.
Image Segmentation through Clustering Based on Natural Computing Techniques
59
2.1 Data clustering In a very simple level of abstraction, the image segmentation process is very close to the clustering problem. To find clusters in a data set is to find relations amongst unlabeled data. The "relation" means that some data are in some way next to another that they can be grouped. It is found in (Jain et al., 1999) that the components of a clustering task are: 1. Pattern representation includes: feature selection, which identifies the most effective subset of the original features to use in clustering; and feature extraction, which is the preprocessing of the input features. 2. A Distance measure is used to determine pattern proximity. A simple, and, perhaps, the most used, distance function is the Euclidean Distance. 3. Clustering relates to finding the groups (or, labeling the data) and it can be hard (an element belongs to one group only) or fuzzy (an element belongs to one group following a degree of membership). 4. Data abstraction is an optional phase and extracts a simple and compact representation of a data set and, in the case of data clustering, some very representative patterns are chosen: the centroids. 5. Assessment of output is the process of evaluating the clustering result. Cluster validation techniques are, also, a traditional approach to dynamic clustering (Omram et al., 2006). Two classical clustering algorithms are used in this work: Kmeans (Forgy, 1965) and Fuzzy CMeans (Zadeh, 1994). 2.1.1 Kmeans Kmeans objective if minimize the J function, which represents the minimization of the distance between objects (patterns) and clusters: K
J K − means = ∑
∑ d2 (x j , ck )
(2)
k = 1 j∈Sk
where: a. k is the number of clusters evaluated (in a space defined by Sk) b. xj is the pattern j evaluated in relation to the centroid ck c. d2(xj, ck) is the distance between pattern xj and centroid ck The algorithm performs as follows: a. Initialize K centroids (for example, randomly) b. Until a stop criterion is not satisfied a. Calculate the distances between all elements in the dataset and the K centroids. Elements closer to centroids form clusters b. Centroids are updated (assume the clusters values) The main advantages of this algorithm are (Turi, 2001): a. Is easy to implement b. The complexity is O(Np), which makes it very applicable to large datasets. The main disadvantages are (Davies, 1997): a. It is dependent on the dataset b. It is a greedy algorithm, which depends upon initial conditions that can lead to suboptimal solutions c. The number of clusters (K) must be informed by the user.
60
Image Segmentation
2.1.2 Fuzzy Cmeans The Fuzzy Cmeans (FCM) algorithm is defined by (Bezdek et al., 1987) as follows: let c ≥ 2 be an integer; let X = {x1 ,..., xn } be a finite dataset which contains at least c < n distinct points; and let Rcn be the set of all real matrices c × n . A partition of the X set is represented by a matrix U = [ uik ] ∈ Rcn whose elements satisfy the following equations: ui k ∈ [0,1],
1 ≤ i ≤ c; 1 ≤ k ≤ n
(3)
1≤ k ≤n
(4)
1≤i≤c
(5)
c
∑ uik = 1, i =1 n
∑ uik > 0,
k =1
where vi is the centroid of cluster i (most representative element). Partitions and centroids are chosen from the minimization of the functional J: n
c
J m (U , v ) = ∑ ∑ ( μik ) x k − vi m
2
(6)
k =1 i =1
where 1 ≤ m ' < ∞ is the fuzzyficator parameter and . is a distance measure. Yet, the following condition is necessary for every i: n
vi =
∑ ( uik )
m
i =1 n
∑ ( uik )
xk
(7)
m
k =1
and, for every k, in such a way that if dik = xk − vi for every i:
2
> 0 for every i, then the following is true
1
uik =
1
(8)
⎛ d ⎞ m−1 ∑ ⎜⎜ dik ⎟⎟ i = 1 ⎝ jk ⎠ c
This chapter understands Clustering and Image Segmentation as a similar task. We make no distinction between them, in the view of the experiments. For a review on clustering techniques, please refer to (Jain et al., 1999; Xu & Wunsch, 2005; Hruschka et al., 2009).
3. The stateoftheart Some image segmentation techniques are presented by (Raut et al., 2009) and they can be classified in: a. Thresholdbased techniques: are generally used for gray level images. A threshold value T is defined to split the image in two parts: foreground and background based on pixel value
Image Segmentation through Clustering Based on Natural Computing Techniques
b.
61
Histogrambased techniques: the histogram of all the pixels is calculated, and according to peaks and valleys different clusters are formed c. Edge detection techniques: first and second order derivatives are used for detection of edges. Edges are divided in two categories: intensity edges and texture edges d. Regionbased techniques: uses region growing and region splittingmerging procedures. Region growing procedure groups pixels or sub regions into large regions based on predefined criteria. Region splitmerge divides image into disjoint regions and then either merge and/or split to satisfy prerequisite constraints e. Watershed Transformation techniques: considered to be more stable than the previous techniques, it considers the gradient magnitude of an image (GMI) as a topographic surface. Pixels having the highest GMI correspond to watershed lines, which represent region boundaries. Water placed on any pixel enclosed by a common watershed line flows downhill to a common local intensity minima (LMI). Pixels draining to a common minimum form a catchments basin, which represent the regions. Clustering can be formally considered as a particular kind of NPhard grouping problem (Hruschka et al., 2009). This assumption has stimulated much research and use of efficient approximation algorithms. Many variations of approaches have been introduced over last 30 years, and image segmentation remains an opensolution problem. Recently there has been an increase in the presence of optimizationbased techniques. (Angus, 2007) proposed a technique for a Populationbased Ant Colony Optimization (PACO) to Multiobjective Function Optimization (MOFO). (Raut et al., 2009) proposed an approach used for prediction using segmentation. They use a GraphPartitioning technique which has some bases on Ontology. In summary, image features may contain concepts (definitions of things) and relations between concepts. This makes up a knowledge database used for object prediction. Important to note about the almost obvious result in the use of optimization techniques and how much it differs from, for example, the much well known Kmeans algorithm: the optimization technique will, theoretically, always find a better solution. Let single be an algorithm that finds one solution; let multi be an algorithm based on single that executes it about 100 times; from the 100 times, multi finds the better solution. It is possible that the single’s solution is the same found by multi, but optimization techniques tend to actually see the problem by the worst side, i.e. if there is a local best maybe there is a global best. This behavior demonstrates the expectationexploitation dilemma. As we will see in Section 4, most of the Natural Computing techniques are based on some common facts: a. A population can achieve better results than one individual [of that population]; b. Every population needs some sort of change in its life. It is called progress or evolution; c. The evolution can obey a random process, sometimes called mutation, and it can occur when a population tend to remain unchanged for a long period of time; d. Every population has an individual that knows a very good solution. Sometimes, this individual can be crossed over another individual (that knows a good solution too) to generate another, eve better individual; e. It is also a good approach to select the most capable individuals from one population (parents), cross over them, and create the next generation of individuals (descendants). It is assumed that every generation is better than the previous one; f. There is a method to calculate how good an individual is, to measure it. It is often called fitness function.
62
Image Segmentation
This chapter is located in this context of optimization techniques. We present some techniques to solve clustering and image segmentation problems and discussion about experiments and results.
4. Natural computing According to (Castro, 2007) Natural Computing is the computational version of the process of extracting ideas from nature to develop computational systems, or using natural materials to perform computation. It can be classified in (Castro, 2007): a. Computing inspired by nature: algorithms take inspiration from nature to solve complex problems; b. The simulation and emulation of nature by means of computing: a synthetic whose product mimics natural phenomena; c. Computing with natural materials: the use of novel materials to perform computation to substitute or complement siliconbased computers. d. Next section presents some of the most representative approaches. 4.1 Artificial Neural Networks (ANN) An Artificial Neural Network, as found in (Haykin, 1998), is a massively distributed parallel builtin processor composed of simple processing units (the neurons) that act, naturally, to store useful knowledge which is acquired through a learning process that yields better results when the processing units work in a network interconnected form (the neural network). The learning process, realized through a learning algorithm, resembles brain in two aspects: a. Knowledge is obtained by the network from its environment through a learning process, which means the network does not acts in an unknown environment. ANN fits in a class of algorithms that need an instructor, a professor, who identifies and models the domain, presents data to the network and evaluate obtained results; b. Forces connecting neurons, the synapse, are used to store achieved knowledge. Some useful properties of ANN are: 1. Nonlinearity 2. Mapping between InputOutput 3. Adaptability 4. Faulttolerance 5. Uniformity of analysis and project 4.2 Evolutionary computing The ideas of evolutionary biology and how descendants carry on knowledge from their parents to be adaptive and better survive are the main inspiration to develop search and optimization techniques for solving complex problems. Evolutionary Algorithms have their bases on biology and, specifically, Evolutionary Theory and adapting organisms. (Castro, 2007) says that this category of techniques are based on the existence of a population of individuals that are capable of reproduction and are subject to genetic variation, followed by selection of new more adapted individuals in its environment. There are many variations in the concept of Evolutionary Algorithms:
Image Segmentation through Clustering Based on Natural Computing Techniques
63
a. Genetic Algorithms b. Evolutionary Strategies c. Evolutionary Programming and d. Genetic Programming Although, they are all based on the following principles: a. A population of individuals reproduces and transmits characteristics to other generations (inheritance). This concept determines that every individual, called chromosome carries a potential solution to the optimization problem in question. The solution represents the genetic trace of the individual, the chromosomes’ components, the alleles, and it’s encoded and structured in some way. These individuals are capable of reproduction, which is, a combination between two individuals and, after this process, future generation carry characteristics of previous ones. b. Genetic variation: the individual reproduction mechanism generates modifications in the genetic trace of the next population’s individuals. A process known as mutation allows the exploration of new solutions inside the search space. c. Natural selection: the living environment for individuals is competitive, for only one of them will give a most adequate and useful solution to a given problem. So, it’s necessary to define some way to verify how much an individual is able to participate in the process of generation of new individuals. The evaluation is realized through a performance evaluation function, known as fitness function. It is important to remember that some characteristics of living organisms are not present in the formulation of evolutionary methods. (BarCohen, 2006) presents some of them: a. In nature, the occurrence of climate variations and environmental situations changes the characteristics of species through time and are fundamental to the verification of how much skilled an organism is. Evolutionary algorithms, otherwise, consider that the fitness function is constant in time. b. In natural evolution, individuals of different species can battle and only one will survive. In evolutionary algorithms there is only one species. In summary, with bases in (Krishna & Murty, 1999) an evolutionary algorithm is composed of the following steps: 1. Initialization of Population or Initial Generation: is often a random process to generate individuals for the initial population. 2. Selection: chromosomes of a previous population are selected to be part of the reproduction process. In general, a probabilistic distribution is used and the selection is based in the value of the fitness function for every individual. 3. Mutation: the individual’s encoded solution, the allele, generated in the reproduction process, is exchanged in some way to make the algorithm don’t stay stuck on local optima, but, through an exploration process, stay next to the global optima. This process of generation of new individuals and population modification or update is repeated several times, until a stop criterion is satisfied. Some applications of evolutionary algorithms are: • Planning (i.e.: routing and scheduling) • Design (i.e.: signal processing) • Simulation and identification • Control • Classification (i.e.: machine learning, pattern recognition)
64
Image Segmentation
4.3 Swarm intelligence Optimization based on swarm intelligence corresponds to methods that have become target of recent scientific researches. (Brabazon & O’Neill, 2006) indicates that there are two variations of this swarm model: a. The first is inspired in bird flock social behavior b. The second is based on behavior of insects, like ants. The term “swarm intelligence” can have many definitions. (Castro, 2007) quotes some of them: • Swarm intelligence is a property of nonintelligent agent systems with limited individual capabilities that exhibit collective intelligent behavior (White & Parurek, 1998). • Swarm intelligence includes every effort to design algorithms or distributed devices to solve problems inspired in collective behavior or social insects and other animal societies (Bonabeau et al., 1999). Ant Colony Optimization (ACO) was designed in 1997 by Dorigo and collaborators. They showed how the behavior of ants following pheromone could be used to optimize Travelling Salesman Problem (TSP) (Kennedy & Eberhart, 2001). For a detailed presentation of this method, please refer to (Brabazon & O’Neill, 2006). Particle Swarm Optimization (PSO) (Kennedy & Eberhart, 2001) is a population based stochastic algorithm, modeled after the observation and bird flock behavior simulation. Even being very similar to other evolutionary approaches, PSO defines that each individual (called particle) benefits from its own previous solutions (a notion of history) (Omram, 2004). The theory that delineates PSO design is under the Adaptive Culture Model and three fundamental principles are taken into account: a. To evaluate: learning is based on the analysis that every individual make of its own responses to external stimuli. b. To compare: individuals are stimulated to compare themselves to other individuals, mainly that ones who have better performance and success. c. To imitate: the logical consequence of the previous principles, it directs the individuals on their learning process. 4.3.1 The algorithm The classical PSO design is that each particle, amongst the multitude of individuals (the swarm), flies through the search space (Omram, 2004) and carries on a potential solution to the optimization problem (Omram et al., 2006). The movement of each particle, which is, the changing of position, is determined by an equation that considers he current position of the particle and a velocity vector (Omram, 2004; Omram et al., 2006): xi = xi (t ) + v i (t + 1)
(9)
(
v i (t + 1) = ϖ v i (t ) + c 1r 1 ( p i (t ) − xi (t )) + c 2r 2 p g (t ) − xi (t )
)
where, according to (Omram et al., 2006): a. ω is the inertia weight, which controls the impact of the previous velocity b. c1 and c2 are acceleration constants c. r1~U(0,1) and r2~U(0,1) d. U(0,1) is a uniform distribution between 0 and 1
(10)
Image Segmentation through Clustering Based on Natural Computing Techniques
65
e.
pi (t ) is the cognitive component, which represents the experience of particle i about where is the best solution. It considers the memory of particle’s previous solutions p g (t ) is the social component, which represents the experience of the whole swarm about f. where is the best solution A user defined maximum velocity can be used to constraint the velocity update (Kennedy & Eberhart, 2001). The performance of the particle is measured using a fitness function which depends on the optimization problem. The PSO algorithm is summarized as follows: 1. For each particle, randomly position it in the search space and randomly initialize its velocity vector 2. Repeat while until a stop criterion is satisfied a. For each particle i. Evaluate its quality (using the fitness function) ii. Update its best position iii. Update swarm’s best position iv. Update its velocity (Eq. 10) v. Update its position (Eq. 9)
4.4 Artificial Immune Systems Artificial Immune Systems (AIS) is a term to adaptive systems, emerging in 1980’s, that extract ideas and metaphors from the biologic immune system to solve computer problems (Castro, 2007). The main idea is inspired in following understanding (Castro, 2007): a. that every living organism have the ability to resist over illness caused by pathogenic agents (virus or bacteria) b. the first rule of the immune system is to protect the body or structure of the living organism; the cells of the immune system are capable to recognize molecular patterns (some sort of molecular signature) that is present within pathogens c. once the pathogen is recognized, cells send each other signals that indicates the need for fight against the illness This framework of immunologic engineering is composed by (Castro, 2007): a. a representation of the system’s components b. a set of mechanisms to evaluate the interaction between individuals and their environment. The environment is simulated by a series of stimuli (input patterns), one or more evaluation functions (fitness) c. adaptive procedures rule the system dynamics, which is, how its behavior changes over the time. As can be seen, there is a very large set of naturally inspired approaches, each one needing its own chapter to be clearly detailed. This chapter will focus on Genetic Algorithms and Particle Swarm Optimization.
5. Clustering and image segmentation based on natural computing This section presents two clustering methods based on GA and PSO, both used in clustering and image segmentation.
66
Image Segmentation
5.1 Genetic Kmeans algorithm Genetic Algorithms have been applied to many function optimization problems and are shown to be good in finding optimal and near optimal solutions (Krishna & Murty, 1999). Aiming to solve the partitional clustering algorithm problem of finding a partition in a given data, with a number of centroids (or clusters), Genetic KMeans Algorithm (GKA) is introduced by (Krishna & Murty, 1999); it establishes an evaluation criterion based on the minimization of the Total Within Cluster Variation (TWCV), an objective function that is defined as follows (Doi, 2007; Lu et al., 2004): given X, the set of N patterns, and Xnd the dth feature of a pattern Xn, Gk the kth cluster and Zk the number of patterns in Gk, the TWCV is defined as: TWCV =
N
D
K
1
D
∑ ∑ Xnd 2 − ∑ Z ∑ SFkd 2
n=1 d =1
k =1
(11)
k d =1
where SFkd is the sum of the dth features of all patterns in Gk. The TWCV is also known as squareerror measure (Krishna & Murty, 1999). The objective function, thus, tries to minimize the TWCV, finding the clustering that has centroids attending concepts of (Omram et al., 2006) compactness (patterns from on cluster are similar to each other and different from patterns in other clusters) and separation (the clusters’ centroids are wellseparated, considering a distance measure as the Euclidean Distance). It is found in (Bandyopadhyay & Maulik, 2002) another method for genetic algorithm based clustering that uses another fitness function, the DaviesBoudin index, which is a function of the ration of the sum of withincluster scatter to betweencluster separation. As will be seen later, other validation indexes may be used and despites the objective function, GKA main aspects are: 1. Coding. Refers to how to encode the solution (the chromosome); one way of doing this is the stringofgroupnumbers encoding where for Z coded solutions (partitions), represented by strings of length N, each element of each string (an allele) contains a cluster number. 2. Initialization. The initial population P0 is defined randomly: each allele is initialized to a cluster number. The next population Pi+1 is defined in terms of the selection, mutation and the Kmeans operator. 3. Selection. Chromosomes from a previous population are chosen randomly according to a distribution. 4. Mutation. The mutation operator changes an allele value depending on the distances of the cluster centroids from the corresponding pattern. 5. KMeans Operator (KMO). This operator is used to speed up the convergence process and is related to one step of the classical Kmeans algorithm. Given a chromosome, each allele is replaced in order to be closer to its centroid. Another approach, KMeans Genetic Algorithm (KGA), is presented in (Bandyopadhyay & Maulik, 2002) and shows a slight modification to the definitions presented before: the crossover operator is added to the algorithm and it is a probabilistic process that exchanges information between two parent chromosomes for generating two new (descendant) chromosomes. 5.2 Clustering using Particle Swarm Optimization Different approaches are found that implement clustering based PSO algorithms, such as (Omram et al., 2006) and (Omram, 2004). A PSObased Clustering Algorithm (PSOCA) can be defined as follows (Omram, 2004; Omram et al., 2006): in the context of data clustering, a
67
Image Segmentation through Clustering Based on Natural Computing Techniques
single particle represents de set of K cluster centroids, in other words, each particle represents a solution to the clustering problem and, thus, a swarm represents a set of candidate data clusterings. The main steps are: a. Initialize particle position and velocity (for each particle); b. While a stop criterion is not found, for each particle: a. Calculates particle’s quality b. Finds particle’s best and global best c. Updates particle’s velocity.
6. Experiments and results The experiments rely on evaluate numerical results of clustering algorithms based on Genetic Algorithms and PSO. As previously seen, both methods are modeled to allow a switch of the traditional and basic clustering algorithm. Thus, this allows us to define the following algorithms variations: a. Genetic Kmeans Algorithm (GKA) b. Genetic Fuzzy Cmeans Algorithm (GFCMA) c. PSObased Kmeans Algorithm (PSOKA) d. PSObased Fuzzy Cmeans Algorithm (PSOFCMA) The datasets used in data clustering experiments are the following: a. Ruspini: twodimensional dataset with 75 patterns. Has four classes easily separable b. Wine: thirteen dimensions and 178 patterns. Has three classes c. Iris: fourdimensional dataset with 150 patterns. Has three classes Implementation was made in Matlab and used the Fuzzy Clustering and Data Analysis Toolbox (Balasko et al., 2005). To best evaluate the results, considering classification error, in each dataset was added another dimension, corresponding to the cluster number associated to the pattern. Cluster Validation Indexes (CVI) was used to obtain numerical results and guide the possible best solution found by the algorithms: DaviesBouldin (DB), SC (separation and compactness), S (separation), and XieBeni (XB). For a review on CVI please refer to (ElMelegy et al., 2007). To compare the effectiveness of GA and PSObased approaches, Table 1 presents Kmeans and FCM clustering results for Ruspini, Wine and Iris datasets. It can be seen that FCM performs better than Kmeans considering the CVI and Error of classification. Table 2 presents GKA and GFCMA clustering results for Ruspini, Wine and Iris datasets. It can be seen that, in general, GKA got better results than Kmeans, FCM and GFCMA.
CVI DB SC S XB Error (%)
Min 0.61212 0.46308 0.00446 3.41458 11.33
KMeans Mean 0.69081 0.48372 0.00465 4.47178 19.60 (a)
Max 0.77991 0.51758 0.00497 4.93836 42.67
FCM 0.62613 0.62798 0.00638 3.97634 10.67
68
Image Segmentation
CVI DB SC S XB Error (%)
Min 0.29046 0.27625 0.00407 2.88926 0.00
KMeans Mean 0.44717 0.35350 0.00665 7.18313 30.43
Min 1.10551 0.95495 0.00664 1.90714 2.81
KMeans Mean 1.26082 0.97880 0.00682 1.96253 6.19
Max 0.87690 0.51807 0.01296 8.57401 100.00
FCM 0.33632 0.36330 0.00541 6.04515 0.00
Max 1.57878 1.25538 0.00859 2.17519 47.75
FCM 1.30418 1.62948 0.01197 0.97245 5.06
(b)
CVI DB SC S XB Error (%)
(c) Table 1. Clustering results for Kmeans and FCM: (a) Iris; (b) Ruspini; (c) Wine
CVI DB SC S XB Error (%)
Min. 0.58931 0.43933 0.00389 2.50475 10.67
GKA Mean 0.61651 0.45049 0.00416 2.58308 32.39
Max. 0.66188 0.45839 0.00458 2.68649 68.00
Min. 0.45831 0.62725 0.00630 1.35521 10.00
GFCMA Mean 0.62613 0.62947 0.00638 1.63055 15.42
Max. 0.64908 0.63466 0.00644 1.85673 55.33
Min. 0.29046 0.36330 0.00540 0.77498 0.00
GFCMA Mean 0.32035 0.36332 0.00541 1.28165 7.19
Max. 0.32237 0.36339 0.00542 1.95288 76.00
Min. 0.84352 1.62937 0.01197 0.60118 5.06
GFCMA Mean 1.11319 2.41962 0.01913 0.60694 17.10
Max. 1.30337 5.31974 0.04760 5.31974 53.37
(a)
CVI DB SC S XB Error (%)
Min. 0.29046 0.27625 0.00407 2.81341 0.00
GKA Mean 0.29046 0.27625 0.00407 2.90996 11.56
Max. 0.29046 0.27625 0.00407 3.04789 100.00 (b)
CVI DB SC S XB Error (%)
Min. 1.10055 0.96569 0.00670 1.52923 3.37
GKA Mean 1.10605 0.96961 0.00674 1.58911 9.11
Max. 1.29697 0.97382 0.00680 1.63309 23.03 (c)
Table 2. Clustering results for GKA and GFCMA: (a) Iris; (b) Ruspini; (c) Wine
Image Segmentation through Clustering Based on Natural Computing Techniques
69
Fig. 1 shows clustering results for Wine dataset using GFCMA and GKA methods (PCA is used to reduce dimensions) obtaining error rate of 5.05% and 4.5%, respectively.
(a)
(b) Fig. 1. GA clustering results for Wine dataset: (a) GFCMA; (b) GKA
70
Image Segmentation
CVI DB SC S XB Error (%)
Min. 0.27045 0.39613 0.00365 1.19575 6.00
PSOKA Mean 0.36780 0.47915 0.00421 1.52560 65.97
Max. 0.52796 0.54635 0.00516 2.06278 100.00
Min. 0.57841 0.62484 0.00637 1.21399 10.67
PSOFCMA Mean 0.62335 0.62749 0.00637 1.37726 15.75
Max. 0.62613 0.62796 0.00638 1.68562 35.33
Min. 0.00538 0.29046 0.36314 0.80723 0.00
PSOFCMA Mean 0.00540 0.30042 0.36328 1.44156 9.81
Max. 0.00541 0.32035 0.36330 3.27777 92.00
Min. 0.86679 1.62937 0.01197 0.59022 5.06
PSOFCMA Mean 1.10988 1.62937 0.01197 0.60331 13.80
Max. 1.30337 1.62937 0.01197 0.61588 49.44
(a)
CVI DB SC S XB Error (%)
Min. 0.00471 0.29046 0.00832 0.18273 0.00
PSOKA Mean 0.00489 0.30640 0.71131 1.01053 22.25
Max. 0.29046 0.33533 1.68594 1.75728 100.00 (b)
CVI DB SC S XB Error (%)
Min. 0.23707 0.79707 0.00479 1.19477 5.06
PSOKA Mean 0.69798 0.95203 0.00672 1.36998 31.25
Max. 1.02811 1.22435 0.00870 1.54367 72.47 (c)
Table 3. Clustering results for PSOKA and PSOFCMA: (a) Iris; (b) Ruspini; (c) Wine Table 3 summarizes clustering results for PSOKA and PSOFCMA. It can be seen that PSOKA performs better than PSOFCMA considering CVI and PSOFCMA is better than PSOKA considering Error (error of classification). Fig. 2 presents PSOFCMA and PSOKA clustering for Wine. The dataset used in image segmentation experiments was obtained from the BrainWeb system (BrainWeb, 2010; Cocosco et al., 1997; Kwan et al., 1996; Kwan et al., 1999; Collins et al., 1998), it corresponds to simulated MR images of T1 modality, 0% noise, and 0% intensity. BrainWeb dataset contains 10 classes that range from background to connective material. For ground truth and classification error evaluation is used the “crisp” dataset. Fig. 3 presents a slice from the MRI Volume in BrainWeb that is used as dataset for experiments. Fig. 3a represents the input to algorithms. Fig. 3b represents the ground truth. Image segmentation approaches of current work are unsupervised, so the ground truth is used only as a final evaluation step, to quantify image segmentation results.
Image Segmentation through Clustering Based on Natural Computing Techniques
(a)
(b) Fig. 2. PSO clustering results for Wine dataset: (a) PSOFCMA; (b) PSOKA
71
72
Image Segmentation
(a)
(b)
Fig. 3. Slice from volume in BrainWeb dataset: a) fuzzy dataset; b) crisp dataset Final objective is to find the correct classes that represent brain regions. Fig. 4 shows crisp dataset in detail and with every class individually.
Fig. 4. Crisp dataset in detail: every class corresponding to one brain region in BrainWeb Cluster Validation Indexes are commonly used to evaluate image segmentation, but results did show that classification error was not acceptable – experiments did show it was around 90%. From this assumption, another study has begun in the direction of finding a better way to evaluate image segmentation. (Chabrier et al., 2006) work on unsupervised image segmentation evaluation methods present several approaches. Amongst them we use the Rosenberger’s evaluation criterion, which is defined by following equation (Chabrier et al., 2006): 1+ 1 ROS( I R ) = where: a. I R
C N2 R ∑ i , j = 1,i ≠ j NR
( g ( R ) − g ( R ) / 512 − 4 / 255 N ) ∑ I
i
I
j
2
2
R
NR i =1
σ 2 ( Ri ) (12)
corresponds to the segmentation result of image I in a set of regions
{
}
R = R1 ,..., RN R } having N R regions
Image Segmentation through Clustering Based on Natural Computing Techniques
b.
73
gI ( Ri ) can be generalized to a feature vector computed on the pixels values of the
( )
region Ri . The same occurs for g I R j c.
C N2 R
is the number of combinations of 2 regions among N R
According to (Chabrier et al., 2006) this criterion combines intra and interregions disparities: intraregion is computed by the normalized standard deviation of gray levels in each region; interregions disparity computes the dissimilarity of the average gray level of two regions in the segmentation result. For comparison purposes experiments were taken for classical Kmeans and Fuzzy Cmeans (FCM) algorithms, considering 100 rounds – with maximum 100 iterations each. Table 4 presents best results considering lower classification error.
Measure DB MSE SC S XB ROS Error (%)
Min. 0.33098 39.47764 0.15269 1.00000 141.13651 0.50030 50.21514
Kmeans Mean. 0.39152 181.26347 0.20480 4.32406 997.30277 0.50036 65.40306
Max. 0.47994 749.88781 0.27183 10.00000 26302.67634 0.50042 84.72134
FCM 0.38630 86.35377 0.29905 0.00001 145.14488 0.50039 68.78071
Table 4. Image Segmentation results for Kmeans and FCM Important to note is that there were no heuristics for experiments with Kmeans and FCM: values from Table 4 are obtained may be different every time the experiment runs, unless for FCM, for it has the same results have always been found. Fig. 5 and Fig. 6 show qualitative results for Kmeans and FCM, respectively. Both image segmentations using Kmeans and FCM shows that all classes have many classification errors and many of them are indistinguishable from each other. In other words, most classes are very similar. Current work’s objective is that approaches under investigation (GKA, GFCMA, PSOKA and PSOFCMA) achieve better values for all measures and classification error. Each method runs in a set of experiments, which evaluate the effect of some parameters:
Fig. 5. Crisp dataset in detail: every class corresponding to one brain region in BrainWeb
74
Image Segmentation
Fig. 6. Crisp dataset in detail: every class corresponding to one brain region in BrainWeb a. Parameters for GKA and GFCMA • Crossover rate • Mutation rate • Number of generations • Population size b. Parameters for PSOKA and PSOFCMA • Maximum velocity • Number of individuals in swarm For each approach the fitness function is based on one of measures: Cluster Validation Indexes, MSE or ROS. This can be considered a parameter to the algorithm as well. Some measures need to be minimized (DB, MSE, XB, ROS) and others need to be maximized (SC, S). It is important to note that current approaches are unsupervised. This means that obtaining classification error has no influence on approaches’ behavior and is used only as a way to evaluate its performance in a controlled scenario. Based on observations from experiments, GKA and GFCMA experiments evaluate best when they use crossover rate of 70%, mutation rate of 0.5% and number of generations around 100. Higher numbers of generation values have no influence. Population size is of 10 individuals. Numerical results for GKA and GFCMA are shown by Table 5.
Measure\Algorithm
GKA Value
GFCMA Error
Value
Error
DB
0.30636
63.53082
0.34955
MSE
12.40774
66.08193
74.99295
72.4037
SC
0.42729
68.42427
0.90113
48.82756
S XB ROS
66.07175
0.00002
72.61756
0.00007
51.29974
124.14228
66.22705
84.06929
72.45716
0.50026
63.51045
0.50025
40.63447
Table 5. Image Segmentation results for GKA and GFCMA According to results from Table 5 it is noted that GFCMA experiment with ROS measure outperforms other experiment’s configurations – considering classification error. Fig. 7 shows classes for GFCMA’s experiment that achieved best results.
75
Image Segmentation through Clustering Based on Natural Computing Techniques
Fig. 7. Crisp dataset in detail: every class corresponding to one brain region in BrainWeb PSOKA and PFOFCMA experiments use a maximum velocity parameter equals to 0.2 and stops when stabilization is found (value of objective function does not change across 10 iterations). Table 6 shows numerical results for PSOKA and PSOFCMA.
Measure\Algorithm
PSOKA Value
Error
PSOFCMA Value
Error
DB
0.34345
71.69336
0.33365
65.24938
MSE
14.26504
72.21529
74.46232
69.18044
SC
0.77279
66.34926
0.99548
71.43621
S
0.00007
68.00163
0.00004
71.37001
260.60458
60.57489
94.1416
68.5312
0.50018
66.03356
0.50030
68.31479
XB ROS
Table 6. Image Segmentation results for PSOKA and PSOFCMA Table 6 shows that PSOKA experiment with XB measure got lower classification error. Fig. 8 shows brain regions for PSOKA’s experiment that achieved best results.
Fig. 8. Crisp dataset in detail: every class corresponding to one brain region in BrainWeb
76
Image Segmentation
Experiments with BrainWeb dataset had the ground truth to evaluate the approaches and used gray scale images. To show the performance of approaches with general purpose images, we will segment color images: Lena, Peppers and Duck.
(a)
(b)
(c)
Fig. 9. Color images: a) Lena; b) Peppers; and c) Duck Results will show segmentation results considering each approach and all quality measures. The number of clusters proceeds as follows: Lena – 6 classes; Peppers – 5 classes; and Duck – 3 classes. Image has been resized to 96 x 96 pixels, RGB color. Tables 7 to 9 and Figures 10 to 12 present quantitative and qualitative image segmentation results, respectively. For these datasets there is no ground truth (no true labels). Thus, the evaluation about how measure/approach has the best result need to be made through quantitative and qualitative results. Best quantitative results are bolded in tables. Best qualitative results are harder to analyze, so the methodology is to consider: a. For Peppers image: well defined frontiers and region homogeneity b. For Lena image: well defined frontiers between skin, hat and hair and region homogeneity c. For Duck image: well defined frontiers between duck body, mouth and glasses/ background This criterion is used to qualitatively evaluate image segmentation results. Considerations about the results are presented in next section.
Measure\Algorithm DB
GKA
GFCMA
PSOKA
PSOFCMA
0.56110
0.56504
0.54677
0.6147
211.10835
651.25003
239.515
640.409
SC
1.47962
8.50041
1.4389
8.09223
S
0.00017
0.00140
0.00032
0.00087
XB
8.67126
3.70102
5.69572
4.11349
ROS
0.49947
0.51484
0.48019
0.51205
MSE
Table 7. Image Segmentation results for Peppers image
77
Image Segmentation through Clustering Based on Natural Computing Techniques
Fig. 10. Qualitative image segmentation results for Peppers image. Rows: 1 – GFCMA, 2 – GKA, 3 – PSOFCMA, 4 – PSOKA. Columns: 1 – DB, 2 – MSE, 3 – SC, 4 – S, 5 – XB, 6 – ROS.
Measure\Algorithm DB
GKA
GFCMA
PSOKA
PSOFCMA
0,63599
0,67499
0,44408
0,62798
105,67107
373,08084
114,80000
369,99500
SC
0,89346
10,49051
1,21201
4,71058
S
0,00012
0,00112
0,00023
0,00052
11,58644
6,47175
7,62521
5,82961
0,54312
0,54141
0,53541
0,54345
MSE
XB ROS
Table 8. Image Segmentation results for Lena image
78
Image Segmentation
Fig. 11. Qualitative image segmentation results for Lena image. Rows: 1 – GFCMA, 2 – GKA, 3 – PSOFCMA, 4 – PSOKA. Columns: 1 – DB, 2 – MSE, 3 – SC, 4 – S, 5 – XB, 6 – ROS.
Measure\Algorithm DB
GKA
GFCMA
PSOKA
PSOFCMA
0,43669
0,44730
0,30422
0,44495
260,16469
542,49030
347,33520
536,90780
SC
1,05008
4,85890
1,08519
29,61981
S
0,00021
0,00243
0,00019
0,00081
XB
9,32596
6,74430
1,83358
7,99377
ROS
0,46663
0,56625
0,50669
0,58084
MSE
Table 9. Image Segmentation results for Duck image
Image Segmentation through Clustering Based on Natural Computing Techniques
79
Fig. 12. Qualitative image segmentation results for Duck image. Rows: 1 – GFCMA, 2 – GKA, 3 – PSOFCMA, 4 – PSOKA. Columns: 1 – DB, 2 – MSE, 3 – SC, 4 – S, 5 – XB, 6 – ROS.
7. Conclusion and future research The present work presents two natural computing methods for data clustering and image segmentation, their implementation and some results, one based on Genetic Algorithms and the other based on Particle Swarm Optimization. The task of image segmentation is not a trivial process. Considering the medical imaging context it is highly important the specialist’s opinion about the results found. As the MRI dataset is simulated the experiments were guided by this situation. Thus, it is necessary to make experiments with real MRI imagery. Color images were used as well to analyze the performance of approaches on general purpose image segmentation. The methodology used in this work was based on the following: 1. To implement the algorithms 2. To evaluate clustering results on known databases 3. To use the obtained results to guide tests with image segmentation. Image segmentation tests must consider image characteristics. As the present methods are based on Evolutionary Computation and all have a performance (fitness) function, there must be some way to guide this evolution, so tests were made considering several Clustering Validation Indexes (DB, SC, S and XB), a commonly used
80
Image Segmentation
error measure (MSE) and an image segmentation specific measure (ROS). Also, when available, a measure of classification error was used to identify the method’s final and overall performance. CVI, MSE and ROS can be used as a function of quality of a solution (population/generation for GKA or particle for PSO). Considering classical clustering, Kmeans outperforms FCM considering classification error. Qualitative analysis shows that both algorithms did not identify correctly any of the classes and it is dificult to evaluate the quality of solution because, according to ground truth, most classes are merged or part of one class is in other class. Class 1, which may be background is the most correctly identified, even having some elements from other class in its interior. Classes 2 and 4 are almost correct also. Considering GA, the lower classification error was obtained by GFCMA (around 40%), with ROS index. GFCMA also got best results considering SC, S, XB and ROS measures. Qualitative result shows that the same considerations for Kmeans and FCM apply to GFCMA, but most classes are almost identical, which results in weak qualitative evaluation. The quantitative measures were also enhanced. Only index S was better with Kmeans. Considering PSO, the lower classification error was obtained by PSOKA (around 60%), with XB index. PSOKA was better considering MSE, S and ROS, while PSOFCMA was better considering DB, SC and XB. Curiously, better value of XB was not the one that obtained lower classification error. PSO also enhanced quantitative measures. MRI dataset evaluation has considered the ground truth, so it was possible to evaluate experiment's results considering classification error. Experiments were made to evaluate the performance of GA and PSO considering general purpose color images. For Peppers image, GFCMA got best quantitative results (indexes SC, S and XB), followed by PSOKA (indexes DB and ROS). Qualitative analysis shows that GFCMA with index DB got best results, considering that red and green peppers where correctly separated and GFCMA also identify some background (between peppers). For Lena image, GFCMA (indexes SC and S) and PSOKA (indexes DB and ROS) got best results. Qualitative analysis shows that all approaches had problems with regions of hat and skin. Considering skin and hair, GFCMA with ROS index and PSOKA with ROS index got best results. For Duck image, GKA (indexes MSE and ROS), GFCMA (indexes SC and S) and PSOKA (indexes DB and XB) got best quantitative results. Qualitative analysis shows that GKA with index SC and S, PSOFCM with index SC and S got best results. Most experiments using classical Kmeans and FCM run to 100 iterations – and more iteration could lead to lower error values. It’s necessary to remember that GA and PSO both use only one iteration of Kmeans and FCM, and the convergence is fast (about 5 to 10 iterations). The problem of possible premature convergence of PSO is investigated by (Yonggang, et al., 2005), which proposed the Improved PSO (IPSO) algorithm. This is a problem to take into account as a try to improve image segmentation results for PSO and GA also. In summary, considering the results obtained from the experiments, it can be said that methods based on FCM performed better. As the present work does not evolves to image registration and classification more evaluation is necessary to argue about Fuzzy Cmeans superiority over Kmeans, in terms of the implemented algorithms. The use of image segmentation benchmarks to compare to obtained results is also a task for future research, together with studies about newer approaches and definitions for GA and PSO, mainly considering image characteristics, like texture, region and borders (frontiers).
Image Segmentation through Clustering Based on Natural Computing Techniques
81
One problem with these evolutionary algorithms is that the only concern is the quality of the solution, with little attention given to computational efficiency (Hruschka et al., 2009). The authors also analyze that the literature on clustering and image segmentation techniques based on evolutionary or natural computing does not provide detailed theoretical analyses in terms of time complexity. As we agree with this argumentation, one future work is the correct understanding of these algorithms in terms of computational efficiency and complexity.
8. References Angus, D. (2007). Populationbased ant colony optimization for multiobjective function optimization, Proceedings of the 3rd Australian Conference on Progress in Artificial Life 2007, pp. 232244, Eds. Lecture Notes In Computer Science. SpringerVerlag, Berlin, Heidelberg Balasko, B.; Abonyi, J; and Feil, B. (2005). Fuzzy Clustering and Data Analysis Toolbox. Online: http://www.mathworks.com/matlabcentral/fileexchange/7486 Bandyopadhyay, S. & Maulik, U. (2002). Genetic clustering for automatic evolution of clusters and application to image classification. Pat. Recog. Vol. 35. pp. 11971208. Elsevier BarCohen, Y. (2006). Biomimetics: biologically inspired technologies. Taylor & Francis, California Brabazon, A. & O’Neill, M. (2006). Biologically inspired algorithms for financial modeling. Springer Natural Computing Series. SpringerVerlag, Berlin BrainWeb (2010). BrainWeb: Simulated Brain Database. Online: http://mouldy.bic.mni.mcgill.ca/brainweb/ Castro, L. N. (2007), Fundamentals of natural computing: an overview, Physics of Life Reviews, Vol. 4, Issue 1, pp. 136, March 2007, ISSN 15710645 Chabrier, S.; Emile, B.; Rosenberger, C.; and Laurent, H. (2006). Unsupervised Performance Evaluation of Image Segmentation. EURASIP Journal on Applied Signal Processing, Vol. 2006, pp 112 Cocosco, C.A.; Kollokian, V.; Kwan, R.K.S. & Evans, A.C. (1997). BrainWeb: Online Interface to a 3D MRI Simulated Brain Database. NeuroImage  Proceedings of 3rd International Conference on Functional Mapping of the Human Brain. Vol. 5. No. 4. part 2/4, S425, Copenhagen, May 1997 Collins, D.L.; Zijdenbos, A.P.; Kollokian, V.; Sled, J.G.; Kabani, N.J.; Holmes, C.J. & Evans, A.C. (1998). Design and Construction of a Realistic Digital Brain Phantom. IEEE Transactions on Medical Imaging. Vol. 17. No. 3. pp. 463468, June 1998 Davies, E. (1997). Machine Vision: Theory, Algorithms, Practialities. Academic Press, 2nd ed. Doi, K. (2007). Computeraided diagnosis in medical imaging: Historical review, current status and future potential. Comp. Medical Imaging and Graphics. Vol. 31, pp. 198211. Elsevier ElMelegy, M.; Zanaty, E.A.; AbdElhafiez, W.M. & Farag, A. (2007). On Cluster Validity Indexes in Fuzzy and Hard Clustering Algorithms for Image Segmentation. IEEE International Conference on Image Processing, 2007 (ICIP 2007). Vol. 6, pp.VI5VI8, ISBN 9781424414376, Sept. 16 2007Oct. 19 2007, San Antonio, TX Forgy, E. (1965). Cluster Analysis of Multivariate Data: Efficiency versus Interoperability of Classification. Biometrics (21), pp. 768769
82
Image Segmentation
Gonzalez, R. C. & Woods, R. E. (2003). Digital Image Processing. 3rd ed., Prentice Hall, Inc., New Jersey Halkidi, M.; Batistakis, Y. & Vazirgiannis, M. (2001). Clustering algorithms and validity measures, Proceedings Thirteenth International Conference on Scientific and Statistical Database Management, pp. 322, 0769512186, Jul. 2001, Fairfax, VA Haykin, S. (1998). Neural Networks: a comprehensive foundation, 2nd ed. Prentice Hall Hruschka, E.R.; Campello, R.J.G.B.; Freitas, A.A. & de Carvalho, A.C.P.L.F. (2009). A Survey of Evolutionary Algorithms for Clustering. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews. Vol. 39. No. 2, pp.133155, March 2009, ISSN 10946977 Jain, A. K.; Murty, M. N. & Flynn, P. J. (1999). Data clustering: a review. ACM Computer Surveys. Vol. 31, pp. 264323 Kennedy, J. & Eberhart, R. C. (2001). Swarm intelligence: collective, adaptive. The Morgan Kaufmann Series in Evolutionary Computation. Morgan Kaufmann, New York Krishna, K. & Murty, N. (1999). Genetic KMeans Algorithm. IEEE Trans. Systems, Man and Cybernetics – Part B: Cybernetics. Vol. 29. No. 3. pp. 433439 Kwan, R.K.S.; Evans, A.C. & Pike, G.B. (1996). An Extensible MRI Simulator for PostProcessing Evaluation. Visualization in Biomedical Computing (VBC'96). Lecture Notes in Computer Science. Vol. 1131. pp. 135140. SpringerVerlag Kwan, R.K.S.; Evans, A.C. & Pike, G.B. (1999). MRI simulationbased evaluation of imageprocessing and classification methods. IEEE Transactions on Medical Imaging. 18(11):108597, Nov 1999. Mathworks (2010). MATLAB – The Language Of Technical Computing. Online: http://www.mathworks.com/products/matlab/. Last access: 11/11/2010. Omram, M. G. H. (2004) Particle Swarm Optimization Methods for Pattern Recognition and Image Processing, Ph.D. Thesis, University of Pretoria, Pretoria Omram, M. G.; Salman, A. & Engelbrecht, A. P. (2006). Dynamic clustering using particle swarm optimization with application in image segmentation. Pattern Anal. Applic. (8), pp. 332344, SpringerVerlag, London Raut, S.; Raghuvanshi, M.; Dharaskar, R. & Raut, A. (2009). Image Segmentation – A StateOfArt Survey for Prediction, Proceedings of International Conference on Advanced Computer Control 2009, pp. 420424, ISBN 9781424433308, Singapore, Jan. 2009, IEEE Computer Society Turi, R. H. (2001). Clusteringbased Colour Image Segmentation. Ph.D. Thesis. Monash University, Australia Xu, R. & Wunsch II, D. (2005). Survey of clustering algorithms, IEEE Transactions on Neural Networks. Vol. 16. No. 3, pp. 645678, May 2005, ISSN 10459227 Yonggang, L.; Weihua, G.; Chunhua, Y. & Jie, L. (2005). Improved PSO algorithm and its application. Journal of Central South University of Technology. Vol. 12. No. 1. pp. 222226 Zadeh, L. A. (1994). Fuzzy logic, neural networks, and soft computing. Communications of the ACM (37), pp. 7784
4 Segmentation with Learning Automata Erik Cuevas, Daniel Zaldivar and Marco PérezCisneros
Departamento de Electrónica, Universidad de Guadalajara México
1. Introduction Several image processing applications aim to detect and mark remarkable features which in turn might be used to perform highlevel tasks. In particular, image segmentation seeks to group pixels within meaningful regions. Commonly, gray levels belonging to the object are substantially different from the gray levels featuring the background. Thresholding is thus a simple but effective tool to isolate objects of interest from the background. Its applications include several classics such as document image analysis, whose goal is to extract printed characters (Abak et al., 1997; Kamel & Zhao, 1993) logos, graphical content, or musical scores; also it is used for map processing which aims to locate lines, legends, and characters (Trier & Jain, 1995). It is also used for scene processing, aiming for object detection and marking (Bhanu, 1986); Similarly, it has been employed to quality inspection for materials (Sezgin & Sankur, 2001; Sezgin & Tasaltin, 2000), discarding defective parts. Thresholding selection techniques can be classified into two categories: bilevel and multilevel. In bilevel thresholding, one limit value is chosen to segment an image into two classes: one represents the object and the other represents the background. When an image is composed of several distinct objects, multiple threshold values have to be selected for proper segmentation. This is called multilevel thresholding. A variety of thresholding approaches have been proposed for image segmentation, including conventional methods (Guo & Pandit, 1998; Pal & Pal, 1993; Shaoo et al., 1988; Snyder et al., 1990) and intelligent techniques such as in (Chen & Wang, 2005; ChihChih, 2006). Extending the algorithm to a multilevel approach may arise some inconveniences: (i) they may have no systematic and analytic solution when the number of classes to be detected increases and (ii) the number of classes is either difficult to be predicted or must be predefined. However, this parameter is unknown for many real applications. In order to solve these problems, an alternative approach using an optimization algorithm based on learning automata for multilevel thresholding is proposed in this paper. In the traditional multilevel optimal thresholding, the intensity distributions belonging to the object or to the background pixels are assumed to follow some Gaussian probability function; therefore a combination of probability density functions is usually adopted to model these functions. The parameters in the combination function are unknown and the parameter estimation is typically assumed to be a nonlinear optimization problem (Gonzalez & Woods, 1990). The unknown parameters that give the best fit to the processed histogram are determined by using a LA algorithm (Thathachar & Sastry, 2002). The main motivation behind the use of LA as an optimization algorithm for parameter adaptation is to use its capabilities of global optimization when dealing to multimodal
84
Image Segmentation
surfaces. Using LA, the search for the optimum is done within a probability space rather than seeking within a parameter space as done by other optimization algorithms (Najim &Poznyak, 1994). Learning automata is referred to as an automaton, acting embedded into an unknown random environment. Such automaton improves its performance to obtain an optimal action. On the other hand, an action is applied to a random environment and gives a fitness value to the selected action of the automata. The response of the environment is used by automata to select its next action. This procedure is continued to reach the optimal action. LA has been used for solve different sorts of engineering problems. For instance, pattern recognition (SeyedHamid, 2008), adaptive control (Zeng et al., 2000) signal processing (Howell & Gordon, 2000) and power systems (Wu, 1995). Recently, some effective algorithms have been proposed for multimodal complex function optimization based on the LA (see (Howell & Gordon, 2000; Thathachar & Sastry, 2002; Zeng & Liu, 2005; Beygi & Meybodi, 2006)). Furthermore, it was shown experimentally that the performance of these optimization algorithms is comparable to or better than the genetic algorithm (GA) in [22]. This work employs the algorithm proposed in (Zeng & Liu, 2005), which is called continuous action reinforcement learning automata (CARLA). In this chapter, an automatic image multithreshold approach based on Learning Automata is presented. Hereby the segmentation process is considered to be similar to an optimization problem. First, the algorithm approximates the 1D histogram of the image using a mix of Gaussian functions whose parameters are calculated using the Learning automata method. Each Gaussian function approximating the histogram represents a pixel class and therefore the threshold points. This chapter is organized as follows. Section 2 presents the Gaussian approximation to the histogram. Section 3 presents the LA algorithm, while Section 4 shows the determination of the threshold points. In section 5 the implementation details are shown. Experimental results for the proposed approach are presented in Section 6, finally the conclusion are presented in Section 7.normal)
2. Gaussian approximation Assuming an image has L gray levels [0,… , L − 1] , following a gray level distribution which can be displayed in the form of the histogram h( g ) . In order to simplify the description, the histogram is normalized and is considered as a probability distribution function:
h( g ) =
ng N
∑
, h( g ) ≥ 0,
L−1
N=
∑
(1)
L −1
ng , and
g =0
h( g ) = 1,
g =0
Assuming that ng denotes the number of pixels with gray level g while N is the total number of pixels in the image. The histogram function can be contained into a mix of Gaussian probability functions, yielding: K
p( x) =
∑ i =1
K
Pi ⋅ pi ( x) =
∑ i =1
⎡ −( x − μi ) 2 ⎤ exp ⎢ ⎥ 2 2πσ i ⎣ 2σ i ⎦ Pi
(2)
85
Segmentation with Learning Automata
considering that Pi is the a priori probability of class i, pi ( x ) is the probability distribution function of graylevel random variable x in class i, μi and σ i are the mean and standard deviation of the ith probability distribution function, and K is the number of classes within the image. In addition, the constraint
∑
K i =1
Pi = 1 must be satisfied.
The typical mean square error consideration is used to estimate the 3K parameters Pi , μi and σ i , i = 1, . . ,K. For example, the mean square error between the composite Gaussian function p( xi ) and the experimental histogram function h( xi ) is defined as follows: J=
1 n
n
∑ ⎣⎡ p(x ) − h(x )⎦⎤ j
j =1
j
2
⎛ +ω ⋅ ⎜ ⎝
⎞
K
∑ P ⎟⎠ − 1 i
(3)
i =1
Assuming an npoint histogram as in [13] and ω being the penalty associated with the constrain
∑
K i =1
Pi = 1 . In general, the determination of parameters that minimize the square
error is not a simple problem. A straightforward method to decrease the partial derivatives of the error function to zero considers a set of simultaneous transcendental equations (Gonzalez & Woods, 1992). An analytical solution is not available due to the nonlinear nature of the equations. The algorithm therefore makes use of a numerical procedure following an iterative approach based on the gradient information. However, considering that the gradient descent method may easily get stuck within local minima. In the standard gradient method, the new operation point lies within a neighbourhood distance of the previous point. This is not the case for adaptation algorithm based on stochastic principles such as LA, as the new operating point is determined by probability function and is therefore not considered to be near the previous operating point. This gives the algorithm a higher ability to locate the global minima. Some previous experiences have shown that the intelligent approaches actually provide a satisfactory performance in case of image processing problems (Chen & Wang, 2005; ChihChih, 2006; Baştürk & Günay, 2009; Lai & Tseng, 2001; Tseng & Lai, 1999). The LA algorithm is therefore adopted in order to find the parameters and their corresponding threshold values.
3. Learning automata LA operates by selecting actions via a stochastic process. Such actions operate within an environment while being assessed according to a measure of the system performance. Figure 1a shows the typical learning system architecture. The automaton selects an action (X) probabilistically. Such actions are applied to the environment, and the performance evaluation function provides a reinforcement signal β . This is used to update the automaton’s internal probability distribution whereby actions that achieve desirable performance are reinforced via an increased probability, while those notperforming actions are penalised or left unchanged depending on the particular learning rule which has been employed. Over time, the average performance of the system will improve while a given limit is reached. In terms of optimization problems, the action with the highest probability would correspond to the global minimum as demonstrated by rigorous proofs of convergence available in (Narendra & Thathachar, 1989) and (Najim & Poznyak, 1994).
86
Image Segmentation Random Environment
Cost J
Learning Automaton
x1 Set of Actions X
Performance Evaluation
Learning Automaton
X = (x1 , x 2 , x 3 )
x2
x3
β
(a)
Learning Automaton
β
Learning Automaton
(b)
Fig. 1. (a) Reinforcement learning system and (b) Interconnected automata. A wide variety of learning rules have been reported in the literature. One of the most widely used algorithms is the linear reward/inaction ( LRI ) scheme, which has been shown to guaranteed convergence properties (see [1008]). In response to action x i , being selected at time step k, the probabilities are updated as follows: pi ( n + 1) = pi ( n) + θ ⋅ β (n) ⋅ (1 − pi (n)) p j (n + 1) = p j (n) − θ ⋅ β (n) ⋅ p j (n) if i ≠ j
(4)
being θ a learning rate parameter and 0 < θ < 1 and β ∈ [0,1] the reinforcement signal; β = 1 indicates the maximum reward and β = 0 is a null reward. Eventually, the probability of successful actions will increase to become close to unity. In case that a single and foremost successful action prevails, the automaton is deemed to have converged. With a large number of discrete actions, the probability of selecting any particular action becomes low and the convergence time can become excessive. To avoid this, learning automata can be connected in a parallel setup as shown by Figure 1b. Each automaton operates a smaller number of actions and the ‘team’ works together in a cooperative manner. This scheme can also be used where multiple actions are required. Discrete stochastic learning automata can be used to determine global optimal states for control applications with multimodal mean square error surfaces. However, the discrete nature of the automata requires the discretization of a continuous parameter space, and the level of quantization tends to reduce the convergence rate. A sequential approach may be adopted (Howell & Gordon, 2000) to overcome such problem by means of an initial coarse quantization. It may be later refined using a requantization around the most successful action. In this paper, an inherently continuous form of the learning automaton is used to speed the learning process and to avoid this additional complexity. 3.1 CARLA algorithm The continuous action reinforcement learning automata (CARLA) was developed as an extension of the discrete stochastic learning automata for applications involving searching of continuous action space in a random environment (Howell & Gordon, 2000). Several CARLA can be connected in parallel, in a similar manner to discrete automata (Figure 1b), to search multidimensional action spaces. The interconnection of the automata is through the
87
Segmentation with Learning Automata
environment however, no direct interautomata communication exist. The automaton’s discrete probability distribution is replaced by a continuous probability density function which is used as the basis for action selection. It operates a reward/inaction learning rule similar to the discrete learning automata. Successful actions receive and increase on the probability of future selection via a Gaussian neighborhood function which increases the probability density in the vicinity of such successful action. Table 1 shows the generic pseudocode for the CARLA algorithm. The initial probability distribution can be chosen as being uniform over a desired range. After a considerable number of iterations, it converges to a probability distribution with a global maximum around the best action value. If action x is defined over the range ( xmin , xmax ) , the probability density function f ( x , n) at iteration n is updated according to the following rule: ⎧α ⋅ [ f ( x , n) + β (n) ⋅ H ( x , r )] if x ∈ ( xmin , xmin ) f ( x , n + 1) = ⎨ 0 otherwise ⎩
(5)
With α being chosen to renormalize the distribution according to the following condition
∫
xmax
f ( x , n + 1)dx = 1
(6)
xmin
with β (n) being again the reinforcement signal from the performance evaluation and H ( x , r ) a symmetric Gaussian neighbourhood function centered on r = x(n) . It yields ⎛ ( x − r )2 ⎞ H ( x , r ) = λ ⋅ exp ⎜ − ⎟ 2σ 2 ⎠ ⎝
(7)
CARLA Algorithm Initialize the probability density function to a uniform distribution Repeat Select an action using its probability density function Execute action on the environment Receive cost/reward for previous action
Update performance evaluation function β Update probability density function Until stopping condition Table 1. Generic pseudocode for the CARLA algorithm with λ and σ being parameters that determine the height and width of the neighborhood function. They are defined in terms of the range of actions as follows:
σ = g w ⋅ ( xmax − xmin )
λ=
gh ( xmax − xmin )
(8) (9)
88
Image Segmentation
The speed and resolution of learning are thus controlled by free parameters g w and gh . Let action x(n) be applied to the environment at iteration n, returning a cost or performance index J(n). Current and previous costs are stored as a reference set R(n). The median and minimum values J med and J min may thus be calculated, by means of β (n) being defined as: ⎪⎧ ⎪⎩
β (n) = max ⎨0,
J med − J ( n) ⎪⎫ ⎬ J med − J min ⎭⎪
(10)
To avoid problems with infinite storage, and to allow the system to adapt to changing environments, only the last m values of the cost functions are stored in R(n). Equation (10) limits β (n) to values between 0 and 1 and only returns nonzero values for costs that are below the median value. It is easy to understand how β (n) affects the learning process informally as follows: during the learning, the performance and the number of selecting actions can be wildly variable, generating extremely high computing costs. However, β (n) is insensitive to these extremes and to the very high values of J(n) resulting from a poor choice of actions. As learning continues, the automaton converges towards more worthy regions of the parameter space and these actions within such regions are chosen for evaluation increasingly often. While more of such responses are being received, J med gets reduced. Decreasing J med in the performance index effectively enables the automaton to refine its reference around the better responses previously received, and hence resulting in a better discrimination between the competing selected actions. To define an action value x(n) which has been associated to this probability density function, an uniformly distributed pseudorandom number z(n) is generated within the range of [0,1]. Simple interpolation is then employed to equate this value to the cumulative distribution function:
∫
x( n )
f ( x , n)dx = z(n)
(11)
xmin
For implementation purposes, the distribution is stored at discrete points with an equal intersample probability. Linear interpolation is used to determine values at intermediate positions (see full details in [19]).
4. Determination of threshold values The next step is to determine the optimal threshold values. Considering that the data classes are organized such that μ1 < μ 2 < … < μK , the threshold values are obtained by computing the overall probability error for two adjacent Gaussian functions, following: E(Ti ) = Pi + 1 ⋅ E1 (Ti ) + Pi ⋅ E2 (Ti ), i = 1, 2,… , K − 1
(12)
considering E1 (Ti ) =
∫
Ti
−∞
pi + 1 ( x )dx ,
(13)
89
Segmentation with Learning Automata
and E2 (Ti ) =
∫
∞
(14)
pi ( x )dx ,
Ti
E1 (Ti ) is the probability of mistakenly classifying the pixels in the (i + 1)th class to the ith class, while E2 (Ti ) is the probability of erroneously classifying the pixels in the ith class to the (i + 1)th class. Pj′s are the a priori probabilities within the combined probability density function, and Ti is the threshold value between the ith and the (i + 1)th classes. One Ti value is chosen such as the error E(Ti ) is minimized. By differentiating E(Ti ) with respect to Ti and equating the result to zero, it is possible to use the following equation to define the optimum threshold value Ti :
ATi2 + BTi + C = 0
(15)
considering A = σ i2 − σ i2+ 1 B = 2 ⋅ ( μiσ i2+ 1 − μi + 1σ i2 )
(16)
⎛σ P ⎞ C = (σ i μi + 1 )2 − (σ i + 1 μi )2 + 2 ⋅ (σ iσ i + 1 )2 ⋅ ln ⎜ i + 1 i ⎟ ⎝ σ i Pi + 1 ⎠ Although the above quadratic equation has two possible solutions, only one of them is feasible (positive and inside the interval). The figure 2 shows the determination process of the threshold points.
μi
pi + 1 ( x )
μi
Ti
μi + 1
Fig. 2. Determination process of the threshold points
90
Image Segmentation
5. Implementation In the implementation four different pixel classes are used to segment the images. The idea is to show the effectiveness of the algorithm and the little information required for the localization of the threshold points, the implementation could be effortlessly carried out for a bigger number of pixel classes. To approach the histogram of an image by 4 Gaussian functions (one for each pixel class), it is necessary first, calculate the optimum values of the 3 parameters (Pi, μi and σ i ) for each Gaussian function (in this case, 12 values according to equation 2). This problem can be resolved optimizing equation 3, considering that function p( x ) is formed by 4 Gaussian functions. The parameters to be optimized are summarized in table 2. Where kPi is the parameter that represents the a priori probability (P), kσi represents the variance ( σ ) and kμi represents the expected value ( μ ) of the Gaussian function i. Parameters
Gaussian
kP1
kσ1
kμ1
1
kP2
kσ2
kμ2
2
k P3
kσ3
kμ3
3
kP4
kσ4
kμ4
4
Table 2. Parameters to be optimized In LA optimization, each parameter is considered like an Automaton which chooses actions. The actions correspond to values assigned to the parameter, by a probability distribution inside of an interval. The intervals used in this work for the parameters are defined as kPi ∈ [0,0.5], kσi ∈ [0,60] , and kμi ∈ [0,255]. For this 12dimensional problem will be 12 different Automatons which represent the parameters to approach the corresponding histogram. One of the main advantages of the LA is that in a multidimensional problem the Automatons are coupled only through the enviorement, thus each Automaton operated independently, during the optimization. Thus, in each instant n each Automaton chooses an action according to their probability distribution, which can be represented in a vector A(n)={ kP1 , kσ1 , kμ1 …, kP4 , kσ4 , kμ4 }. This vector represents a certain approach of the histogram. Then, the quality of the approach is evaluated (according to equation 3) and converted to a reinforcement signal β (n) (through equation 10). Having obtained the reinforcement value β (n) as product of the elected approach A(n), the distribution of probability is update for n+1 of each Automaton (according to the equation 5). To simplify parameters of equation 8 and 9 are the same for the 12 Automatons, such that g w = 0.02 and gh =0.3. In this work is considers to limit to 2000 the iterations on the optimization process. Next, the optimization algorithm is described:
91
Segmentation with Learning Automata
i ii iii iv v
Set iteration n=0. Define the action set A(n)={ kP1 , kσ1 , kμ1 …, kP4 , kσ4 , kμ4 } such that kPi ∈ [0,0.5], kσi ∈ [0,60] and kμi ∈ [0,255]. Define probability density functions at iteration n: f ( kPi , n) , f ( kσi , n) and f ( kμi , n) Initialize f ( kPi , n) , f ( kσi , n) and f ( kμi , n) as an uniform distribution between the
defined limits. Repeat while n ≤ 2000 (a)
Using a pseudorandom number generator for each Automaton, select zPi (n) , zσi ( n) and zμi (n) uniformly between 0 and 1. Select kPi ∈ [0,0.5], kσi ∈ [0,60] and kμi ∈ [0,255] where the area under the
(b)
probability density function is
∫
k μi ( n )
0
∫
k Pi ( n )
0
f ( kPi , n) = zPi ( n) ,
∫
Evaluate the performance using Eq. (3). Obtain the minimum, J min , and median, J med of J (n).
(e)
Evaluate β (n) via Eq. (10).
(g)
0
f ( kσi , n) = zσi (n) and
f ( kμi , n) = zμi (n) .
(c) (d)
(f)
kσi ( n )
Update the probability density functions f ( kPi , n) , f ( kσi , n) and f ( kμi , n) using Eq. (5). Increment iteration number n.
The learning system search in the 12dimensional parameter space with the aim of reducing the values for J in Eq. (3).
6. Experimental results In this section the performance of the algorithm is tested by two experiments. In both experiments a 4 pixel class segmentation is consider and an approaching of the original histogram of the image by LA. To test the consistency of the algorithm, 10 independent repetitions were made for each experiment. In the first experiment the image represented in figure 3a was used, whose original histogram is shown in figure 3b. Considering the proposed LA algorithm (detailed in the previous section) a global minimum was obtained (equation 3), the point defined as kP1 =0.0210, kσ1 =6, kμ1 =15, kP2 =0.0404, kσ2 =29, kμ2 =63, kP3 =0.0608, kσ3 =10, kμ3 =93, kP4 =0.1002, kσ4 =30, and kμ4 =163. The values of these parameters define 4 different Gaussian functions, which are represented in figure 4. From the mix of these 4 Gaussian functions, an approach to the original histogram is obtained as shown in figure 5. The evolution of the probability densities parameters, whose represent the expected values f ( kμ1 , n) , f ( kμ2 , n) , f ( kμ3 , n) and f ( kμ4 , n) of the Gaussian functions are shown in figure 6. It can be seen that most of the convergence is achieved at the first 1500 iterations, after that a gradual sharpening of the distribution occurs. The final probability densities (n=2000) can be taken as the final parameter value.
92
Image Segmentation
(a)
(b)
Fig. 3. (a) Original image used on the first experiment, (b) and its histogram
Fig. 4. Gaussian functions obtained by LA
Fig. 5. Comparison between the original histogram and its approach
93
Segmentation with Learning Automata
(a)
(b)
(c)
94
Image Segmentation
(d) Fig. 6. Evolution of the probability densities parameters, whose represent the expected values (a) f ( kμ1 , n) , (b) f ( kμ2 , n) , (c) f ( kμ3 , n) and (d) f ( kμ4 , n) , of the Gaussian functions From the Gaussian functions obtained by LA (figure 4), the threshold values Ti are calculated considering equations 1516. From these values the image segmented in 4 classes shown in figure 7 is obtained.
Fig. 7. Image segmented in 4 classes by LA In the second experiment, the image shown in figure 8 was used. The idea is again, to segment it in 4 different pixel classes using the LA approach proposed in this work. After execute the algorithm with the parameters detailed in the previous sections the Gaussian functions obtained are shown in figure 9a. The mix of Gaussian functions obtained by the LA algorithm approach to the original histogram, as can be seen in figure 9b. From figure 9b is clear that the algorithm approaches each one of the pixel concentrations, distributed in the histogram, except to the first one (presented approximately around the intensity value 7). This effect shows that the algorithm discards the smallest accumulation of pixels and prefer to cover those classes that contribute to generate a smaller error during optimization of the equation 3. The results can be improved if 5 pixel classes were used (instead of segmenting the image by 4 pixel classes).
95
Segmentation with Learning Automata
Fig. 8. Image used on the second experiment
(a)
(b) Fig. 9. (a) Gaussian functions obtained by LA, and (b) its comparison to the original histogram From the Gaussian functions obtained by LA (figure 9a), the threshold values Ti are calculated considering equations 1516. From these values the image segmented in 4 classes shown in figure 10 is obtained. Figure 11 shows the separation of each class obtained by the algorithm.
96
Image Segmentation
Fig. 10. Segmentation obtained by LA
(a)
(b)
(c)
(d)
Fig. 11. Separation of each class obtained by the LA algorithm. (a) Pixel class 1, (b) Pixel class 2, (c) Pixel class 3, and (d) Pixel class 3.
6. Conclusions This works presents a novel segmentation algorithm which includes an automatic threshold determination approach. The overall method can be considered as a Learning automata optimization algorithm. Following the intensity distributions for each object. The intensity
Segmentation with Learning Automata
97
distributions of objects and background in an image are assumed to obey Gaussian distributions with distinct variances and means. The histogram of a given image is approach by a mix of Gaussian probability functions. The LA algorithm is used to estimate the parameters in the mix density function obtaining a minimum square error between the density function and the original histogram. The experimental results reveal that the proposed approach can produce satisfactory results. Further work of extending the proposed approach with other techniques and comparing the results with state of the art image segmentation techniques are in progress.
7. Acknowledgments The first author would like to thank SEP (Mexico) and DAAD (Germany) for its economical support.
8. References Abak, T.; Baris, U. & Sankur, B. (1997). The performance of thresholding algorithms for optical character recognition, Proceedings of International Conference on Document Analytical Recognition, 697–700. Baştürk, A. & Günay, E. (2009). Efficient edge detection in digital images using a cellular neural network optimized by differential evolution algorithm, Expert System with Applications, 36(8), 26452650. Beygi, H. & Meybodi, M. (2006). A new actionset learning automaton for function optimization. Int. J. Franklin Inst. 343, 27–47. Bhanu, B. (1986). Automatic target recognition: state of the art survey, IEEE Trans. Aerosp. Electron. Syst. 22, 364–379. Chen, S. & Wang, M. (2005). Seeking multithresholds directly from support vectors for image segmentation, Neurocomputing, 67(4), 335344. ChihChih, L. (2006). A Novel Image Segmentation Approach Based on Particle Swarm Optimization, IEICE Trans. Fundamentals, 89(1), 324327. Gonzalez, R. & Woods, R. (1992). Digital Image Processing, Addison Wesley, Reading, MA. Guo, R. & Pandit, S. (1998). Automatic threshold selection based on histogram modes and discriminant criterion, Mach. Vis. Appl., vol.10, pp.331–338. Howell, M. & Gordon T. (2000). Continuous action reinforcement learning automata and their application to adaptive digital filter design. Engineering Applications of Artificial Intelligence. 14, 549–561. Kamel, M. & Zhao, A. (1993). Extraction of binary character/graphics images from grayscale document images, Graph. Models Image Process. 55 (3) 203–217. Lai, C. & Tseng, D. (2001). An optimal Lfilter for reducing blocking artifacts using genetic algorithms, Signal Process., vol.81, no.7, pp.1525–1535. Najim, K. & Poznyak, A. (1994). Learning Automata  Theory and Applications. Pergamon Press, Oxford. Narendra, K. & Thathachar, M. (1989). Learning Automata: an Introduction. PrenticeHall, London. Pal, N. & Pal, S. (1993). A review on image segmentation techniques, Pattern Recognit., vol.26, pp.1277–1294.
98
Image Segmentation
SeyedHamid, Z. (2008). Learning automata based classifier, Pattern Recognition Letters. 29, 4048. Sezgin, M. & Sankur, B. (2001). Comparison of thresholding methods for nondestructive testing applications, IEEE International Conference on Image Processing, 764–767. Sezgin, M. & Tasaltin, R. (2000). A new dichotomization technique to multilevel thresholding devoted to inspection applications, Pattern Recognition Lett. 21 (2) 151– 161. Shaoo, P. ; Soltani, S. ; Wong, A. & Chen, Y. (1988). Survey: A survey of thresholding techniques, Comput. Vis. Graph. Image Process., vol.41, pp.233–260. Snyder, W. ; Bilbro, G. ; Logenthiran, A. & Rajala, S. (1990). Optimal thresholding: A new approach, Pattern Recognit. Lett., vol.11, pp.803–810. Thathachar, M. & Sastry, P. (2002). Varieties of learning automata: An overview. IEEE Trans. Systems. Man Cybernet. Part B: Cybernet. 32, 711–722. Trier, O. & Jain, A. (1995). Goaldirected evaluation of binarization methods, IEEE Trans. Pattern Anal. Mach. Intel. 17 (12) 1191–1201. Tseng, D. & Lai, C. (1999). A genetic algorithm for MRFbased segmentation of multispectral textured images, Pattern Recognit. Lett., vol.20, no.14, pp.1499–1510. Wu, Q. (1995). Learning coordinated control of power systems using interconnected learning automata. Int. J. Electr. Power Energy Syst. 17, 91–99. Zeng, X. & Liu, Z. (2005). A learning automaton based algorithm for optimization of continuous complex function. Information Sciences. 174, 165–175. Zeng, X. ; Zhou, J. & Vasseur, C. (2000). A strategy for controlling nonlinear systems using a learning automaton. Automatica. 36, 1517–1524.
5 Surround Suppression and Recurrent Interactions V1V2 for Natural Scene Boundary Detection Francisco J. DíazPernas, Míriam AntónRodríguez, Isabel de la TorreDíez, Mario MartínezZarzuela, David GonzálezOrtega, Daniel BotoGiralda, and J. Fernando DíezHiguera
Department of Signal Theory, Communications and Telematics Engineering Telecommunications Engineering School, Valladolid University, Valladolid Spain
1. Introduction Understanding the brain functioning in visual process is one of the more active areas within neuroscience, modeling and computation. This great interest is due to the fact that vision provides us with the more important information about the surroundings. Oriented early filtering processes, perceptual clustering through emerging features, form and depth perception, figureground separation, and object recognition are all involved in the brain visual activity. Important researches have been undertaken to develop models simulating this brain behavior. So, neural models of visual perception biologically motivated in early vision tasks have been arisen versus others not biologically motivated computer vision algorithms. The knowledge about the early vision is notable and, accordingly, the neural modeling of the early vision tasks has been very scientifically productive. There are numerous visual models of the early visual perception (Kokkinos et al., 2007) (Neumann et al., 2007). One of the more significant is the BCS/FCS model of Grossberg and Mingolla (Grossberg & Mingolla, 1985). This model is composed of two systems, boundary contour system (BCS) and feature contour system (FCS), modeling processes given in the interactions among V1, V2 and V4 visual areas. Features so important in the human vision like illusory perception, emergent segmentations, diffusive fillingin are integrated in a coherent way in the BCS/FCS network. This system has experiment an important evolution in its modeling, integrating spatial and orientational competitive processes in a same stage (Mingolla et al., 1999). This model has been the framework of many other researchers in the development of their approaches. Kokkinos et al. (Kokkinos et al., 2008) have recently developed a computational simplification of the BCS/FCS model. They propound a threestage structure incorporating feature extraction with contrast normalization, boundary formation through a competitive stage of information of feature extraction, cooperative signal, anisotropic image smoothing and large scale signal. In their proposal, they include a comparison to Canny’s detector, a classical computer vision algorithm. Kokkinos’ architecture shows better results than Canny’s algorithm. Main contribution of Kokkinos’ is its simplification of the BCS/FCS model more than its evolution as it is somewhat complex.
100
Image Segmentation
Several neurophysiological studies show that the response modulation by stimuli beyond the receptive field is a common phenomenon in the V1 area (Petkov & Westenberg, 2003). There are evidences that 80% of the orientation selective cells perform a response suppression over a bar texture surround beyond its receptive field. Based on these studies Petkov and Westenberg (Petkov & Westenberg, 2003) proposed a surround suppression model for the complex cells and they also analyzed this effect by psychophysical experiments. This interesting inhibitory effect has been considered in the development of our proposal of boundary detection neural network. This surround suppression mechanism is not considered in the BCS. Nevertheless, Petrov and Westenberg’s network is limited to the simple and complex cells for evaluating the surround suppression effect. Competitive and cooperative processes generated by the V2 hypercomplex cells, are not considered in their proposal. Another important biological evidence is the existence of feedback interactions in the visual cortex (Hubel, 1995). BCS includes a competitivecooperative loop where the completion boundary and the illusory contour extraction are performed. Neumann and Sepp (Neumann & Sepp, 1999) presented a V1V2 recurrent process through the integration of the V1 activations from V2 receptive field elongated lobes. In the work presented in this paper, trying to obtain a model as simple as possible, we propose an architecture considering the more significant mechanisms of the human visual system early processing for color scene stimuli: chromatic and achromatic opponent channels, orientational filtering, surround suppression, V1V2 recurrent interactions through competitive and cooperative fields. Additionally, we propose an interscale information fusion stage in order to obtain the boundary output with all the information gathered from the scene. In (AntónRodríguez et al., 2009) we proposed a visual feature extraction architecture for colortexture identification corresponding to the color extension of the BCS/FCS system. In this work we presented a BCS system for processing signals from three channels, two opponent chromatic channels and a luminance one, and two FCS for diffusing the two chromatic channels. In the present work, we propose a new model with six opponent channels, four chromatic and two achromatic, emerged from the transformation of the RGB image. It also includes new mechanisms, like surround suppression and interscale fusion, to achieve the natural scene boundaries.
2. Perceptual boundary recurrent detection neural architecture The Perceptual boundaRy rEcurrent dEtection Neural (PREEN) proposed model (see Fig. 1) comprises five main components, respectively designated as Colour Opponent stage (CO), Chromatic Contour stage (CC), Competitive Fusion stage (CF), Contour Saliency stage (CS), and Interscale Competition (IC). The neural model processing is achieved through multiple spatial scales. The CO stage transforms the chromatic components of the input signals (RGB) into a bioinspired codification system, made up of various opponent chromatic channels and an achromatic channel. In order to do this, the CO stage firstly calculates the activations of the long (L), middle (M), and short (S) wavelength retinal cones, and then, it generates the opponent processes, corresponding to the ONOFF achromatic channel and the L+M, M+L, S+(L+M) and (L+M)+S chromatic opponencies. Studies of the human visual system have found that visual stimuli take part in the color opponent and enhancement processes located in retina and Lateral Geniculate Nucleus (LGN) cells of the mammalian visual system (Hubel, 1995; Wilson et al., 1990).
Surround Suppression and Recurrent Interactions V1V2 for Natural Scene Boundary Detection
101
Fig. 1. Proposed model architecture CC stage is composed of simple and complex cells which filter and perform a surround suppression through multiple scales and orientations in order to extract the real boundaries of the scene. Simple cell multiscale filtering extracts textural features from both the color opponent and the achromatic signals. Complex cells use a competition network with surround suppression, which attenuates the contour activities from inner texture areas where it appears a high concentration of false boundaries. Complex cells coherently fuse the color and texture activities, generated by the simple cells, with a behavior inspired in cells from the V1 visual area of the human visual system (Petrov & Westenberg, 2003). Biologically inspired V1V2 recurrent interactions take place in the PREEN architecture. CF stage competitively fuses topdown and bottomup signals, complexCFCS, defining and shaping the natural boundaries. CF and CS stages model competitive and cooperative behaviors exhibited in the V2 area of the human visual system (Hubel, 1995). Final output is constituted by a contour map image corresponding to the natural boundaries with perceptual significance. The PREEN model architecture includes competitive networks. We model these networks using the membrane potential network (Hodgkin & Huxley, 1990), whose simplified behavior can be expressed according to equation (1).
CM
(
)
(
)
dV = − AV + V + − V g + + V − − V g − dt
(1)
where V is the membrane potential, A is a decay constant, V+ and V are the excitatory and inhibitory reversal potentials, and g+ and g are the excitatory and inhibitory total input, respectively. In a stationary situation, the potential, V, would be defined by equation (2). This is the situation we consider in our model.
102
Image Segmentation
V=
V + g+ + V − g−
(2)
A + g+ + g−
where V + ≥ 0 and V − ≤ 0 . Equation (2) expresses the membrane potential as the normalization between the net input (difference between excitation and inhibition) and the total input (excitation plus inhibition). So, this normalization computes the ratio contrast and solves the noisesaturation dilemma. 2.1 Colour opponent stage The CO stage performs chromatic and achromatic opponent competitive processes based on opponent mechanisms and luminance channel competitive enhancement to generate four chromatic signals shaping opponent pairs, L+M, M+L, S+(L+M) and (L+M)+S, and two achromatic signals, ON and OFF. This is observed as the generation of three channels of opponent pairs. In hierarchical levels of the human system there are evidences of manifold cases of opponent pairs, both in the motor and visual systems (Zrenner, 1990). The CO processing contains a previous stage in which luminance (I signal), activations of the long (L signal), middle (M signal), short (S signal) wavelength cones and (L+M) channel activation (Y signal) are generated from R, G and B input signals. Equations (3), (4) and (5) define the calculations of those activities (AntónRodríguez et al., 2009). The luminance signal (I) is computed as a weighted sum; the L, M and S signals are obtained as the transformation matrix of the three chromatic components R, G and B. I = 0.299 R + 0.587 G + 0.114 B
(3)
⎡ L ⎤ ⎡ 0.293 0.603 0.104 ⎤ ⎡ R ⎤ ⎢ M ⎥ = ⎢0.134 0.704 0.162 ⎥ ⎢G ⎥ ⎢ ⎥ ⎢ ⎥⎢ ⎥ ⎢⎣ S ⎥⎦ ⎢⎣0.046 0.099 0.854 ⎥⎦ ⎢⎣ B ⎥⎦
(4)
Y =L+M
(5)
The CO stage models the behavior of the opponent cells whose main characteristics are their high sensibility and precision to contrasts contours. These attributes recommend their use as a previous step for image contour, shapes, and texture detection (Hubel, 1995). The opponent model for L+M chromatic channels, following membrane competitive network is defined by equation (6). +
cijlm( s )
+
∑ ⎡⎣D(pqs ) ⎤⎦ Lij − ∑ ⎡⎣−D(pqs ) ⎤⎦ Mij = + + A + ∑ ⎣⎡D(pqs ) ⎦⎤ Lij + ∑ ⎣⎡ −D(pqs ) ⎦⎤ Mij
(6) +
where cijlm( s ) is the L+M channel cell activity for position (i,j), [c ] = max ( 0, c ) , A is a decay constant, and D(pqs ) is a difference of Gaussians (see Fig. 2) following equation (7).
(
⎡ ⎛ − p2 + q 2 D(pqs ) = DA ⎢ exp ⎜ ⎢ ⎜ 2σ e( s )2 ⎝ ⎣⎢
) ⎞⎟ − exp ⎛⎜ − ( p ⎟ ⎠
⎜ ⎝
)
+ q 2 ⎞⎤ ⎟⎥ 2σ i( s )2 ⎟ ⎥ ⎠ ⎦⎥ 2
(7)
Surround Suppression and Recurrent Interactions V1V2 for Natural Scene Boundary Detection
103
where σ e and σ i are the excitation and inhibition deviations and DA is the amplitude constant.
Fig. 2. Difference of Gaussians (dog) The remainder opponencies, M+L, S+Y and Y+S, are calculated using equation (6) substituting Lij and Mij signals for the corresponding opponency signals. Analogously to the achromatic process accomplished in the LGN of the human visual system (Hubel, 1995), the neural architecture enhances the luminance signal, Iij, in a competitive network generating two opponent channels ONcenter and OFFcenter under equation (6), similarly to the chromatic channels, replacing the chromatic signals by the luminance signal. ON channel processing is in charge of enhancing the information of regions with higher luminance intensity in relation to their surroundings, while OFF channel enhances regions with lower luminance intensity than their surroundings. These competitive processes, derived from equation (6), establish a gain control network over the inputs from chromatic and luminance channels, maintaining the sensibility of cells to contrasts, compensating variable illumination, and normalizing image intensity. These competitive networks are parallelly applied over three different spatial scales, with suitable values of σ e and σ i for the small, medium and large scales (s=0,1,2). Correlation studies about color codification systems based on trichromatic theory have proven the suitability of using the redgreen and blueyellow opponent signals (Ohta et al., 1980), in all, opponencies of longmiddle and shortlongmiddle wavelength. In like manner, the ganglionar opponent cells with higher presence rate in the retina correspond to LM opponency, next higher rate is S(L+M) (Zrenner et al., 1990), so we consider appropriate using the opponencies chosen. Biologically motivated models for boundary detection, as BCS/FCS (Grossberg et al., 1995) or Kokkinos’ model (Kokkinos et al., 2008) do not use chromatic channels for detecting boundaries. They only work with the luminance component. Nevertheless, they use images of natural scenes for validation. It is clearly that the human visual system operating has an essential chromatic component, based on the color opponencies.
104
Image Segmentation
2.2 Chromatic Contour (CC) The Chromatic Contour stage models V1 simple and complex cells. It undertakes the contour extraction and the merging with surround suppression of the simple cell signals, as it can be seen in Fig. 1. CC stage models behaviors of cells located in V1 visual area, orientation and spatial frequency selective cells and information fusion cells (Hubel, 1995). Simple cells extract contours from chromatic and achromatic CO channels through a Gabor (s) (s) ) and odd ( Oijk ) components for position (i,j), deviation for filter bank, using even ( Eijk
multiple scales (s=0, 1, 2) and orientation (k=0, 1, 2, 3, 4, 5 corresponding to θ= 30º, 60º, 90º, 120º, 150º). These cells respond to variations on the textural and color features. Complex Gabor filters have sensibility to orientation, spatial frequency and position (Daugman, 1980). Grossberg, Mingolla et al. (Grossberg et al., 1995) (Mingolla et al., 1999) modeled simple cells with an oddtype filtering in their BCS model. The inclusion of even fields is justified due to the importance of these profiles in texture detection (Landy & Bergen, 1991). Following the behavior indicated by the membrane potential shown in equation (2), simple cell activities are given by equations (8), (9), (10) and (11).
(s) = aijk
(s) = bijk
∑ E(pqks ) ⎛⎜⎝ ⎡⎣cijon(s ) ⎤⎦
+
+⎞ − ⎡cijoff ( s ) ⎤ ⎟ ⎣ ⎦ ⎠
off ( s ) + ⎞ s ) ⎛ ⎡ on( s ) ⎤ + A + ∑ E(pqk ⎜ ⎣cij ⎦ + ⎡⎣cij ⎤⎦ ⎟ ⎝ ⎠
∑ O(pqks ) ⎛⎜⎝ ⎡⎣cijon(s ) ⎤⎦
+
+⎞ − ⎡cijoff ( s ) ⎤ ⎟ ⎣ ⎦ ⎠
off ( s ) + ⎞ s ) ⎛ ⎡ on( s ) ⎤ + A + ∑ O(pqk ⎜ ⎣cij ⎦ + ⎡⎣cij ⎤⎦ ⎟ ⎝ ⎠
(8)
(9)
lm( s ) eijk =
∑ E(pqks ) cijlm( s ) s) cijlm( s ) A + ∑E(pqk
(10)
lm( s ) f ijk =
∑ O(pqks ) cijlm(s ) s) cijlm( s ) A + ∑O(pqk
(11)
(s) (s) where aijk , bijk are the even and odd simple cell activities for the luminance channel, and lm( s ) lm( s ) eijk , f ijk are the simple cell activities for chromatic channel L+M, position (i,j) and
orientation k for even and odd filters respectively, . represents the absolute value,
[c ]+ = max ( 0, c ) , and A is a decay constant. For the remainder chromatic channels, the activity equations to use are equations (10) and (11) replacing L+M channel signal by the appropriate channel signal (M+L, S+Y, Y+S). For each perceptual position (i,j) a hyper column of simple cells varying in filter type (filter profile) and orientation k is applied to each channel. The complex cell stage, using two cellular layers, fuses information from simple cells giving rise to a map which contains real contours for each of the three scales used (see Fig. 1).
Surround Suppression and Recurrent Interactions V1V2 for Natural Scene Boundary Detection
105
The first layer of cells is in charge of combining responses from different Gabor filters at each opponent and luminance channel at their three scales (s=0,1,2), as shown in equation (12). +
+
+
+
(s) (s) ⎤ (s) ⎤ lm( s ) ⎤ lm( s ) ⎤ hijk = ⎡ aijk + ⎡bijk + ⎡ eijk + ⎡ f ijk + ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ +
+
+
+
+
slm( s ) ⎤ slm( s ) ⎤ ml( s ) ⎤ ml( s ) ⎤ lms( s ) ⎤ lms( s ) ⎤ + ⎡ eijk + ⎡ f ijk + ⎡ eijk + ⎡ f ijk + ⎡ eijk + ⎡ f ijk ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ ⎣ ⎦ ⎣ ⎦
+
(12)
where [c ] = max ( 0, c ) is a halfwave rectifier. +
The second complex cell layer models a competition network with surround suppression (see Fig. 3).
(s) ). Fig. 3. Surround suppression receptive field of CC complex cells ( Sijk
The behavior of layer 2 complex cells is modeled by equation (13): (s) H ijk
(s) s) (s) ⎡ hijk ⎤ hij − λ ∑ S(pqk ⎥ =⎢ (s) s ) lm( s ) ⎢ A + hijk hij ⎥⎦ + ∑ S(pqk ⎣
+
(13)
(s) (s) (s) the CC complex cells output; hijk is the first layer activity, Sijk being H ijk = [ −Dij( s ) ]+ is the
surround suppression receptive field with Dij( s ) a dog following equation (7), as it is shown in Fig. 3, for each of the scales defined by s=0,1,2; A is a decay constant, λ > 0 is the suppression constant and
[c ]+ = max ( 0, c ) .
Equation (13) determines the anisotropic
surround suppression, as the removal is performed according to the orientations.
106
Image Segmentation
In their work, Petrov and Westenberg (Petrov & Westenberg, 2003) observed various manifestations of perceptual modulation by the context. They used image sets of letters and bars, and objects icons (of different sizes and orientations) with superimposed bandspectrum noise. They observed the effects of the surround in contrast for different noise frequencies. The inhibitory effect was modeled designing a nonclassical receptive field (nonCRF) defined by a weighting function with a normalized difference of Gaussians profile. This idea is taken in the present paper but with an important difference. Model proposed of CC complex cells exhibits a positional competition between a position (i,j) and its surroundings. These competitive processes establish a gain control network over the fusion channel input, maintaining the sensibility of cells to contrasts, compensating variable illumination, and normalizing image intensity. In order to obtain a complex nonCRF field external to the CRF field of CC simple cells, we (s) choose same values for σ e deviations of Sijk and for σ ( s ) deviations of CC simple cells and (s) as twice the value of σ ( s ) . we fix σ i deviations of Sijk
In Fig. 4 it can be observed the effect produced by the surround suppression depending on the inhibition strength (λ=1.0 and λ=2.0). Using a high suppression constant, it is possible to remove areas with a high concentration of weak contour caused by inner texture features (see left column). It can be noticed that the surround suppression model makes the boundaries to have less noisy activities near them, thus achieving a better definition and so a better precision. However, as the surround suppression model cleans up the area around the boundaries, sometimes the recall value diminishes. In a later work, Grigorescu, Petrov and Westenberg (Grigorescu et al., 2004) proposed a computational step, called surround suppression for detecting boundaries in natural scenes. This step was incorporated to the Canny edge detector. A comparison was performed using 40 natural images, achieving better results with the step included. To strengthen the analysis about the positive effect of the surround suppression modeling, we took 20 images from the Berkeley Segmentation DataSet (BSDS) and compared the results obtained with and without suppression. In Fig. 4, we can see some processing examples. Fig. 4 (e) includes the Fvalue curve of the processing with (right) and without suppression (left). We achieved a mean Fvalue of 0.64 (0.62, 0.66) when processing with suppression versus a mean value of 0.63 (0.69, 0.57) when processing without suppression. It can be observed a slightly better result counting the suppression. A remarkable feature is the significant difference in the precision value, in favor of the model with suppression, and in the recall value, in favor of the model without suppression, which corroborates the previous point. Using cooperation processes, it is more interesting obtaining high precision than high recall, since these processes generate more boundaries, but also more noise. Accordingly, the model of complex cells showing surround suppression advantages the cooperative processes from V2, which PREEN model includes in the CS stage. The explained complex cell stage modeling has significant differences with other processing models. The complex cell stage of the BCS model (Grossberg et al., 1995) is in charge of summing simple cell signals. This sum produces an independence of the contrast direction, so as the entire real boundary map is obtained in this stage.
Surround Suppression and Recurrent Interactions V1V2 for Natural Scene Boundary Detection
107
a)
b)
c)
d)
e)
Fig. 4. Effect of complex cell surround suppression. Image on the left: image 86016. Image on the right: image 42049. Row a): original image. Row b): output from first stage (fusion) of complex stage (without surround suppression). Row c): output from CC complex stage for λ=1.0. Row d): output from CC complex for λ=2.0. Row e): Fmeasure curves (precision and recall) for 20 test images of the Berkeley database, (e)left without surround suppression and (e)right with surround suppression.
108
Image Segmentation
In PREEN model, the complex cell stage fuses simple cell signals and also removes the surroundings making boundaries to be contrastindependent and more accurate. Kokkinos et al.’s model uses a shunting inhibition for surround suppression which implies a recurrent stage with obliged stabilization (Kokkinos et al., 2008). PREEN model uses a nonrecurrent competition, as the recurrent process is located in upper stages, where V1V2 interaction occurs. 2.3 Competitive Fusion stage (CF) CF stage competitively fuses information from CC complex bottomup and CS cooperation topdown. This stage models the V1V2 recurrent interactions produced in the PREEN model. This recurrent interaction detects, regulates, and completes boundaries into globally consistent contrast positions and orientations, while it suppresses activations from redundant and less important contours, thus eliminating image noise. There are numerous evidences about the existence of hypercomplex cells in the human visual system, where V1V2 recurrent interactions occurs (Hubel, 1995). We use a shunting network to model this behavior, where lateral inhibitions occur within cells of the same competitive stage. Equation (14) describes these CF hypercomplex cells functioning. It includes two inhibitions, orientationally and spatially.
(s) U ijk
(s) s) ⎤ s) (s) ⎡ uijk U pqr − Cc ∑ G(pqs )U (pqk − C i ∑∑ S(pqr ⎢ ⎥ r ≠ k pq =⎢ ⎥ (s) (s) (s) A + uijk + ∑ Gpq U pqk ⎢ ⎥ ⎢⎣ ⎥⎦
+
(14)
(s) (s) (s) (s) being U ijk the CF hypercomplex cells output, uijk = K h H ijk + K f Fijk the input to the (s) (s) competition stage: H ijk is the CC complex cells output, Fijk is the CS topdown signal, K h
s) and K f are gain constants; the orientational inhibitory receptive field, S(pqr , is a Gaussian
kernel with τ (τ>1) aspect ratio, rotated r grades, and with σ i( s ) deviation for each of the scales defined by s=0,1,2; G(pqs ) is a Gaussian kernel with σ u( s ) deviation for each of the scales, A is a decay constant, Cc and C i are inhibition constants and [c ] = max ( 0, c ) . +
(s) Recurrent interaction is solved by an iterative process, where actual state of the U ijk activity (s) . depends on the previous state activity and the CS stage feedback activity, Fijk s) The inhibitory receptive field in the competition among orientations, S(pqr , corresponds to a
Gaussian rotated r grades, with a very high aspect ratio so that all the points in the line of orientation r passing through (i,j) have a maximum inhibition (see Fig. 5). The point of difference between competitive stages of PREEN model and BCS system is the competitive model with shunting inhibition used in PREEN model versus a contrast normalization network from BCS system. Shunting inhibition allows higher boundary enhancement and thinning yielding higher precision and deletion of noise activities, which work against cooperative mechanisms, advantaging the recurrent process.
Surround Suppression and Recurrent Interactions V1V2 for Natural Scene Boundary Detection
109
Kokkinos’ architecture also uses a shunting inhibition model, including in the competition a gradient signal from the smooth stage through a diffusion process, similar to the FCS stage of the BCS/FCS system. Kokkinos’ model does not perform a competition among orientations within this stage. PREEN architecture, similarly to BCS system, proposes a competition among orientations to better define the boundary orientation in each position. In this competition, PREEN uses the inhibition signal from competitive stage neighbor instead of the input signal to the competitive stage taken in the BCS model. That is to say PREEN model heightens the shunting mechanism in the inhibition.
s) Fig. 5. Orientational inhibition receptive field, S(pqr .
110
Image Segmentation
2.4 Contour saliency stage (CS) CS stage models the cooperative process for the saliency detection of the contour activity interpolation. In visual cortex, it has been detected cooperative completion or grouping processes by realizing a “bipole property” (Mingolla et al., 1999). This property refers to the disposition of certain cells in visual cortical areas V1 and V2 to fire in the presence of approximately aligned, but spatially separated, image gradients. Salient contours are detected determining existent contour activity in both sides of each position (i,j) through a bipole receptive field. The behavior of CS cells is modeled by equation (15). (s) Fijk =
(∑ P U ) z(∑ N A + ( ∑ P U )( ∑ N (s) pqk
z
(s) pqk
(s) pqk
(s) pqk
(s) (s) pqkU pqk
) )
(s) (s) pqkU pqk
(15)
(s) (s) (s) is the CF hypercomplex activity, Pijk and N ijk are the lobes of the bipole where U ijk
receptive field of CS cells (see Fig. 6 ), A is a decay constant, and z(s) is a lineal function with a positive cooperative threshold, α: ⎧0 si s ≤ α z(s ) = ⎨ ⎩s si s > α
Each lobe is generated as a Gaussian with an aspect ratio of τ and a profile relation of K l( s ) for each scale s, with a coordinate translation Dc and a rotation k, following equation (16). Fig. 6 shows a dipole example.
Fig. 6. Oriented bipole receptive field. 2 ⎛ ⎞ 2 ⎛ x' ⎞ (s) = exp ⎜ −K l( s ) ⎜ ⎟ + ( y ') ⎟ BIPijk ⎜ ⎟ ⎝τ ⎠ ⎝ ⎠ 1 (s) (s) = Pijk BIPijk (s) BIPijk
(16)
Surround Suppression and Recurrent Interactions V1V2 for Natural Scene Boundary Detection
111
where . is the L1 norm, the coordinates of the spatial transformation: x ' = ( x − xc )cos( k ) + ( y − yc )sin( k ) y ' = −( x − xc )sin( k ) + ( y − yc )cos( k ) and the translation origin coordinates are: xc = Dc cos( k ) yc = Dc sin( k ) (s) Equations for computing N ijk are similar but with a coordinate translation of −Dc . Each
lobe measures one unit of area, and it is essential to be no activity in the dipole center. (s) is normalized, bounded between 0 and 1. Hence, maximum Competitive stage output U ijk
activity in each lobule will be 1. Thereby, we can appropriately choose the value of α. Thanks to this contour cooperative saliency, illusory boundaries of perceptual figures are extracted as it can be observed in Fig. 7, where it appears the PREEN model processing result of the Kanizsa’s square image.
Fig. 7. PREEN processing of Kanizsa’s square image. In BCS system, Kokkinos’ model, and PREEN architecture cooperation is performed in a similar way. All uses bipole receptive field cells to generate the cooperative activity produced when both lobes of the receptive field are active. Difference between both models lie in the way of determining when such excitatory situation is produced. As it is expressed in equation (15), PREEN uses the product of the lobe activities to obtain the cooperation activity. Each lobe contributes to excite the dipole cell is its activity exceeds the threshold α. Hence, the product demands the contribution of both lobes and, so, a normalized bipole cell output near 1 will be guaranteed. In remainder cases, the output will be 0. Kokkinos utilizes the product of the lobule activities but without activation threshold. And BCS uses a pseudologic function of decision AND, with a fast saturation function to a value near to 1, and with a threshold of 2 (Mingolla et al., 1999). By doing so, it is guaranteed that the bipole cell exceed 2 only if there are inputs to both lobules.
112
Image Segmentation
2.5 Interscale Competition stage (IC) PREEN model proposes an output stage where signals from the three scales are merged in order to obtain the best possible output combining all the information gathered in each scale. While scale increases, noise is better filtered and so boundary signals improve, though precision and thinning become reduced. Noise does not appear in larger scales, but only contours with higher gradient are detected; those are high quality contours but not all existing. Boundaries from the small scale are accurate and thin; the boundary map includes all perceptual boundaries of the natural scene images, therefore achieving a high recall value. It appears evident that better boundary qualities will be found in the small scale signal. However, its small receptive fields cause extracting a significant level of noisy contours. So, precision, recall and thinning are provided by the small scale, and noise shortage is given by larger scales, mainly by the large scale. This analysis leads us to propose next assumptions to model the output stage of the PREEN architecture: Small scale signal provides all interesting boundaries for segmenting natural scenes. Medium and large scale signals enclose a reduced level of contour noise. An accurate boundary of a natural scene generates activity in all the scales. Output will be composed of thin and accurate boundaries, with a correct level of recall and generating activity in all the scales. Basing upon these assumptions, PREEN model proposes a single boundary output signal resulting from the interscales competition for detecting boundaries with higher quality. Equation (17) shows the integration of the three scales through an interscale competition with major scale context dependent inhibition, ρ ijk .
Oijk
small small small ⎤ ⎡ U ijk U pqk − ρijkCo∑ Gpq ⎥ =⎢ small small small ⎢⎣ A + Uijk + ρ ijk ∑ Gpq U pqk ⎥⎦
+
(17)
small small is the small scale CF hypercomplex activity, G pq is a Gaussian kernel with a where U ijk
small scale deviation of σ small , A is a decay constant, C o is an inhibitory constant, and
[c ]+
= max ( 0, c )
and ρ ij is the major scale context dependent inhibition gain, defined by
equation (18). ⎛ ⎜ ⎝
⎛ ⎜ ⎝
ρ ijk = exp ⎜ −K o ⎜ 1 − ν ijk =
∑
8 − neightbor
⎞⎞ ⎟⎟ 1 + ην ijk ⎟⎠ ⎟⎠
medium
U pqk
1
+
∑
(18) large
U pqk
8 − neightbor
with K 0 and η positive constants; vijk is the sum of the activities of scales medium and large in the 8connected neighborhood of the consider position. Equation (18) shows that ρijk has a Gaussian profile, comprised between 0 and 1, with a shape dependent of K 0 , and close to 0 when higher contour activity from larger scales. Equation (17) displays strong inhibition when the 8connected neighborhood from larger scales has a high boundary activity and the boundary activity from the small scale will not
Surround Suppression and Recurrent Interactions V1V2 for Natural Scene Boundary Detection
113
be affected if the neighborhood does not include contours. Image areas without contours in larger scales will sharply attenuate the boundary activity from the small scale.
3. Experimental results We quantitatively compare our method against the Kokkinos et al.’s model (Kokkinos et al., 2008), that has been made available publicly. This model proposes a significant simplification of the BCS/FCS model proposed by Grossberg, Mingolla et al. (Grossberg et al., 1995) (Mingolla et al., 1999). The comparison to the Kokkinos et al.’s proposal (and to the BCS/FCS model by extension) is performed according to a common philosophy about propounding biologically inspired models. Differences among these models lie in the modeling of each stage and substantially in the color contribution of the PREEN model in the boundary detection processes in natural scenes. We will try to demonstrate that natural scenes have a chromatic component essential for defining and processing them. The comparison is accomplished with the boundarybased error, Fmeasure, which are the measure used in Kokkinos et al.’s paper. The measure computation has been made using the Matlab code supplied in next web page: http://www.eecs.berkeley.edu/Research/ Projects/CS/vision/grouping/segbench/. We have used all the 100 test images from the Berkeley Segmentation Dataset (Martin et al., 2001) and their human segmented images were taken as the ground truth to accomplish the Fmeasure values. Fig. 8bottom shows a precisionrecall curve (PR curve) with the average Fmeasure value of the boundaries detected in the all 100 test images. In Fig. 9 it can be observed processing examples for the PREEN architecture, including the Fmeasure values with their precision and recall as well as the position that would achieve in the ranking published in the Berkeley Segmentation Benchmark web page (BSDS, 2001). Kokkinos et al. (Kokkinos et al., 2008) compared its model to the Canny edge detection algorithm using the images from the Berkeley dataset achieving better results in different scenarios. In their paper, they show the PR curves and the Fmeasure of the comparisons. Their better setting match with their BCS learned fine scale proposal, achieving an Fmeasure of 0.573 versus the 0.568 obtained by the Canny’s algorithm. PREEN model achieves an Fmeasure with PR coordinates of 0.59 (0.68, 0.52), as it is shown in Fig. 8. So, it obtains better average result when detecting boundaries. Hence, PREEN boundary detection process achieves higher precision and recall values. There are three notable differences between Kokkinos et al.’ model and PREEN model. As previously said, the more significant one is the color contribution in the PREEN model. Another modeling difference is that PREEN model includes orientational competition, so PREEN obtains more accurate contours. A third significant difference is the output of the model. PREEN includes an interscale competitive stage as architecture output. This fusion provides us with an output enclosing all the better features of each scale. The substantial effect of color in the boundary detection process can be analyzed in the boundary detection algorithm ranking published in the Berkeley Segmentation Benchmark web page (http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/bench /html/algorithms.html). The summary table shows different algorithms for grayscale and color processing. There are versions of algorithms for processing both grayscale and color images and it can be observed that color version obtains better results. Considering the average Fvalue achieved, PREEN architecture would gain the eighth position in the ranking
114
Image Segmentation
Fig. 8. Comparative results when processing all the 100 images from the test set of the Berkeley database. Top: Color gradient algorithm. Bottom: PREEN architecture.
Surround Suppression and Recurrent Interactions V1V2 for Natural Scene Boundary Detection
115
145086, F=0.85 (0.85, 0.84), Fhuman=0.85, ranking=1
147091, F=0.8 (0.79, 0.82), Fhuman=0.87, ranking=1
167062, F=0.92 (0.98, 0.87), Fhuman=0.95, ranking=1
210088, F=0.56 (0.69,0.47), Fhuman=0.54, ranking=34
62096, F=0.78 (0.7, 0.87), Fhuman=0.90, ranking=6 Fig. 9. Processing examples of PREEN architecture. Each example includes the image name in the Berkeley Segmentation Dataset, the Fmeasure value, the recall and precision coordinates, and the position that would achieve in the ranking published in the Berkeley Segmentation Benchmark website.
116
Image Segmentation
published in the BSDS web page, outperforming the Color Gradient algorithm. The PR curve of this method can be seen in Fig. 8left (extracted from the aforementioned web page) comparing it to that of the PREEN model. Fig. 8 shows PREEN model achieves outstanding results in some benchmark images, overcoming in some cases the first algorithm in ranking. However, global result is not so excellent, locating the eighth position. This contradiction can be accurately analyzed considering the results shown in Fig. 4 c)left. The surround effect determines the suppression of weak contours sited near other stronger. When processing image 86016 by the surround suppression PREEN model, we can observed lineal contours are drastically attenuated while round contour is reactivated. Apparently, this can be observed as a positive effect, since it affords the possibility to segment the scene into two regions with perceptual significance: the furrows area and the vegetation round area. However, segmentation performed by humans in the Berkeley database includes lineal contours as significant perceptual boundaries, which makes Fmeasure achieved by the PREEN architecture under these circumstances will lean to be lower, considerably diminishing the average Fmeasure value. Fig. 4 c)right shows the opposite case, PREEN model causes the elimination of weak contours from inside tree branch, which makes boundaries to be finer considering the human segmentation.
4. Conclusion In this paper we propose a model bioinspired in the recurrent interactions of the early visual areas for detecting boundaries in color natural scenes, called PREEN model. To perform the comparative analysis, we have chosen BCS/FCS model (Grossberg et al., 1995) (Mingolla et al., 1999) and Kokkinos et al.‘s model (Kokkinos et al., 2008) on the grounds of the similar aims. Images used are taken from the Berkeley Segmentation Dataset (BSDS, 2001). Important features of the proposed model are providing a simple and effective modeling, using even and odd oriented filtering, participation of surround suppression mechanisms in the complex cell stage, oriented and positional shunting competition, contour cooperative saliency extraction through bipole interactions, and a new output stage with inhibition among stages. Test simulations over al the 100 test images from the Berkeley dataset and a comparative analysis have been included. Results achieved outperform those obtained by Kokkinos et al.’s model. Furthermore, PREEN model has displayed outstanding results, overcoming the best algorithms published in the BSDS web page when processing some images of the database. In the analysis performed we have observed PREEN model favors boundaries with high perceptual significance attenuating the remainder boundaries. Thereby, PREEN model obtains low values in such images with human segmentations including boundaries of secondary importance, more germane to attentive mechanisms. We think if the database would have human segmentations with higher participation, the proposed model would achieve more favorable results. This higher participation, in our view, would mean remove or strongly attenuate secondary boundaries in human segmentations.
5. References AntónRodríguez, M.; DíazPernas, F. J.; DíezHiguera, J. F.; MartínezZarzuela, M.; GonzálezOrtega, D. & BotoGiralda, D. (2009). Recognition of coloured and
Surround Suppression and Recurrent Interactions V1V2 for Natural Scene Boundary Detection
117
textured images through a multiscale neural architecture with orientational filtering and chromatic diffusion. Neurocomputing, Vol. 72 (3713–3725). ISSN: 09252312. Berkeley segmentation dataset and benchmark (2001). http://www.cs.berkeley.edu/projects/vision/grouping/segbench/. Last visited: September 2010. Daugman, J.G. (1980) Twodimensional spectral analysis of cortical receptive field profiles. Vision Research, Vol. 20 (pp. 847856). ISSN: 00426989. Grigorescu, C.; Petkov, N. & Westenberg, M.A. (2004). Contour and boundary detection improved by surround suppression of texture edges, Image and Vision Computing, Vol. 22, No. 8 (pp. 609622). ISSN: 02628856. Grossberg, S. & Mingolla, E. (1985). Neural dynamics of form perception: Boundary completion, illusory figures and neon color spreading. Psychological Review, Vol. 92 (pp. 173211). ISSN: 19391471. Grossberg, S., Mingolla, E. and Williamson, J. (1995). Synthethic aperture radar processing by a multiple scale neural system for boundary and surface representation. Neural Networks, Vol. 8 (pp. 10051028). ISSN: 08936080. Hodgkin, A.L. & Huxley, A.F. (1990). A quantitative description of membrane current and its application to conduction and excitation in nerve, Bulletin of Mathematical Biology, Vol. 52, No. 12 (pp. 2571). ISSN: 00928240. Hubel, D. H. (1995). Eye, Brain, and Vision. Scientific American Library. W. H. Freeman, New York. ISBN: 9780716760092. Kokkinos, I.; Deriche, R.; Faugeras, O. & Maragos, P. (2008). Computational analysis and learning for a biologically motivated model of boundary detection. Neurocomputing, Vol. 71, No. 1012 (Jun. 2008) (pp. 17981812). ISSN: 09252312. Kokkinos, I.; Deriche, R.; Papadopoulo, T.; Faugeras, O. & Maragos, P. (2007). Towards bridging the gap between biological and computational image segmentation. INRIA Research Report: Thème BIO Systèmes biologiques, Vol. 6317. ISSN: 02496399. Landy, M.S. & Bergen, J.R. (1991). Texture segregation and orientation gradient. Vision Research, Vol. 31, No. 4 (pp. 679693). ISSN: 00426989. Martin, D.; Fowlkes, C.; Tal, D. & Malik, J. (2001). A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics, Proceedings of the 8th International Conference on Computer Vision, vol. 2, pp. 416423, ISBN: 0769511430, Vancouver, Canada, July 2001. Mingolla, E.; Ross, W. & Grossberg, S. (1999). A neural network for enhancing boundaries and surfaces in synthetic aperture radar images. Neural Networks, Vol. 12 (499511). ISSN: 08936080. Neumann, H. & Sepp, W. (1999). Recurrent V1–V2 interaction in early visual boundary processing, Biological Cybernetics, Vol. 91 (pp. 425–444). ISSN: 03401200. Neumann, H.; Yazdanbakhsh, A. & Mingolla, E. (2007). Seeing surfaces: The brain’s vision of the world. Physics of Life Reviews, Vol. 4 (pp. 189222). ISSN: 15710645.
118
Image Segmentation
Ohta, Y.I.; Kanade, T. & Sakai, T. (1980). Color information for region segmentation. Computer graphics and image processing, Vol. 13 (222241). Petkov, N. & Westenberg, M.A. (2003). Suppression of contour perception by bandlimited noise and its relation to nonclassical receptive field inhibition, Biological Cybernetics, Vol. 88 (pp. 236246). ISSN: 14320770. Wilson, H. R.; Levi, D.; Maffei, L.; Rovamo, J. & De Valois, R. (1990). The perception of form: Retina to striate cortex. In: Visual perception: The neurophysiological foundations, L. Spillmann & J. S. Werner (Eds.), (pp. 231–272). San Diego: Academic Press, Inc. ISBN: 0126576750. Zrenner, E.; Abramov, I.; Akita, M.; Cowey, A.; Livingstone, M. & Valberg, A. (1990). Color Perception: Retina to Cortex. In Visual perception: The neurophysiological foundations, L. Spillmann & J. S. Werner (Eds.), (Chap. 8). San Diego: Academic Press, ISBN: 0126576750.
6 Using Emergence Phenomenon in Meaningful Image Segmentation for Contentbased Image Retrieval Sagarmay Deb
Central Queensland University, 400 Kent Street, Sydney 2000, NSW Australia 1. Introduction Research on multimedia systems and contentbased image retrieval has gained momentum during the last decade. Contentbased image retrieval (CBIR) is a very difficult area in the access of multimedia databases simply because there still exist vast differences in the perception capacity between a human and a computer. There are two basic problems that still remain unresolved in the area although some progresses have been made [13]. The first one is the problem of efficient and meaningful image segmentation where we breakup a particular image into meaningful parts based on lowlevel features like color, texture, shape and spatial locations. Developing a segmentation algorithm which will meaningfully segment all images is yet an open problem in image analysis [8]. The second one is the vast gap existing for an image between lowlevel features mentioned earlier and highlevel or semantic expressions contained in the image like the image of a car, a house, a table and so on [11]. To develop efficient indexing techniques for the retrieval of enormous volumes of images being generated these days, we need to achieve reasonable solutions to these abovementioned two problems. But only in very limited and selected cases, some kinds of solutions have been achieved with apparently promising experimental results. In this paper we focus our attention on the first problem. The research identifies few issues causing this gap, for example, failure to capture local image details with low level features, unavailability of semantic representation of images, inadequate human involvement in the retrieval, and ambiguity in query formulation [9]. We offer future directions of research in solving this difficult problem using emergence phenomena. Section one gives an introduction of the area. Section two provides a definition of emergence phenomenon. Section three talks about the use of emergence phenomenon in extracting meanings in image segmentation. Section four suggests future directions of research. We put our concluding remarks in section five.
2. Emergence phenomenon 2.1 Definition of emergence phenomenon A feature of an image which is not explicit would be emergent feature if it can be made explicit. There are three types of emergence: computational emergence, thermodynamic
120
Image Segmentation
emergence and emergence relative to a model [1]. In computational emergence, it is assumed computational interactions can generate different features or behaviors [3],[7]. This is one of the approaches in the field of artificial life. Thermodynamic emergence is of the view that new stable features or behaviors can arise from equilibrium through the use of thermodynamic theory. In emergence relative to a model, deviation of the behavior from the original model gives rise to emergence. We will use this latter view in our work. In computational emergence, new shapes or images develop but within certain limit as programmed by the computer programmers. No new shape can emerge beyond the logic of the program. In thermodynamic emergence, emergence can be defined as emergence of order from noise. Stochastic processes at microlevel form discrete macrolevel structures or behaviors. The example of this type of emergence is gas where stochastic movements of atoms or molecules within the gas create the ordered properties of temperature, pressure and volume at a higher level. Example of emergence relative to a model is where changes in internal structure and consequently in its behavior occur and we as observers will need to change our model to track the device’s behavior in order to successfully continue to predict actions. The example of a square having two triangles hidden in it as given below is of this type. Whenever we shift our focus on an existing shape in other words an image, new shape emerges. The representation of the new shape is based upon view of the original shape. The new shape emerges as we change our view of the original shape. This is the fundamentally most important idea of emergence. Two classes of shape emergence have been identified: embedded shape emergence and illusory shape emergence. In embedded shape emergence all the emergent shapes can be identified by set theory kind of procedures on the original shape under consideration. For example, in a set S = {a, b, c, d, e}, we can find subsets like S1 = {a, b, c}, S2 = {c, d, e}, S3 = {a, c, e} and so on. But in illusory shape emergence, where contours defining a shape are perceived even though no contours are physically present, this kind of set theory procedures are not enough and more effective procedures have to be applied to find these hidden shapes [4],[5]. These procedures could be based on geometrical, topological or dimensional studies of the original shape. 2.2 Structure, behavior and function of emergence Structure of a shape is the physical definition of the shape. For example, a box could be rectangular in shape, its length, width and height as well as color, substance like wood, metal or hard paper would define the structure of the shape. Behavior of the box could be to contain certain stuffs in it and the function could be to carry stuff from one place to another using the box as a container, which is the purpose for which the box is used. Emergence of new structure, behavior or function takes place when these descriptions are interpreted in ways not anticipated in the original description [4]. 2.3 Examples of emergence Shape emergence is associated with emergence of individual or multiple shapes. The following figures are examples of shape emergence.
Using Emergence Phenomenon in Meaningful Image Segmentation for ContentbasedImage Retrieval
121
The Existing Shape
Fig. 1.1. Two emergent shapes derived from the existing one [5] 2.4 Definition of emergence index Image retrieval where the hidden or emergence meanings of the images are studied and based on those hidden meanings as well as explicit meanings, an index of search is defined to retrieve images is called emergence index. When images are retrieved based on textual information, various parameters and descriptions might define the input and the images of the database. Whenever there would be symmetry of parameters and descriptions, the image could be retrieved. In CBIR, color, texture and shape are widely used as index to retrieve images. But in our studies, we can find the hidden meanings of the images and whenever those hidden meanings match with the input given, although the original image may not match at all with the input, we can retrieve that image.
Fig. 1.2. Example of emergence where diagonal generates two triangles
122
Image Segmentation
To describe in detail, as we have mentioned, emergence is a phenomenon where we bring out the shapes, which are not explicit but implicit. The above figure shows a simple example of emergence where an apparently square shape has two triangles hidden in it as we discussed earlier. When an input would come in the form of an image, the image could be studied based on features, constraints, variables, domains and emergence and converted into parametric form. Then the image database would be accessed and each image would be interpreted considering the items mentioned above and converted into parametric form like the input image. Whenever there would be a match between parameters of the input and the images of the database, those records would be selected. In other words, indexing would be decided by the outcome of emergence which means more meaningful images could be found hidden in an image which would otherwise not be understood. As we have mentioned earlier, many images of the database may not have any apparent similarities with the input, but emergence could bring out the hidden meaning of the image and could establish similarities with the input image. So emergence outcomes of the images would form the index structure of the search. 2.5 Structure of emergence index 2.5.1 Parameter definitions To make an effective query, the images in the query or database must be analysed so that we know what we are looking for and where to look for. Features, domains, variables, constraints, similarities, indexing are the parameters, which play very important role in similarity searching. Hence they constitute the structure of the emergence index. Features Input, as we mentioned earlier, could come in the form of text. For example, the text may indicate we have to pick up all the images of a database that contains the image of a particular person or an object. In this case, nothing much could be done on input side in the sense that we cannot study the input's features etc. We have to go through the images and pick up the image records. But if the query comes in the form of an image of a person then we have to analyse it before accessing the database images. Features would tell us about a few important characteristics of the input image. To start up, our query could be an image where a particular person is sitting on a chair. Then obviously there are two important features in the query image  the particular person and the chair. We have to locate these two features in the database image while searching for similarities. We know color plays a very important role in the definition of features. Quite often a query might mention an object with certain specified color to be picked up from the databases. Besides, color is being used extensively in various models as a tool in finding symmetry with the input image. Sometimes the input may be in the form of a sketch. In that case, a similar kind of image should be selected from the image database. The selection should be made on the basis of few dominant characteristics of the input and image and finding similarities in those characteristics. Texture is another part of the feature where the general alignment of the image like the background part of the image is considered where the image of a person or an object could be the dominant part.
Using Emergence Phenomenon in Meaningful Image Segmentation for ContentbasedImage Retrieval
123
There would be some global features of an image like area, perimeters and a set of rectangles or triangles that cover the entire shape and there would be some local features, which can be obtained from the image's local region. When we retrieve images by browsing, then in most cases the user does not have clear idea about what he or she is looking for. In this case, there would not be input image and the search through browsing would be manual. The user would have vague idea about the features of the images to be retrieved like the picture of a particular person with certain specified background. In the objective features based queries, the retrieval is performed on an exact match of attribute values whereas in the subjective features, query is specified by features, which could be interpreted differently by different users. Retrieval by motion facilitates retrieving spatiotemporal image sequence depicting a domain phenomenon that varies in time or geographic space. Again, sometimes images are retrieved, as we mentioned earlier, by text. In other words, text defines the features of the images to be selected. Then the database is searched based on those features defined in the input [6]. Domain We now proceed to discuss the domain. Domain is a sort of classification of the images into certain categories. Domain is a way for a class of objects to present knowledge representing a certain concept held by objects [10]. We can make use of the various properties of the features of an image to define the domain in which the image concerned would lie. For example, from an image we can understand whether the image is that of a geographical location of a certain area of the earth or the image is that of a person or object with certain background like a screen or a landscape behind. This kind of classification at the initial stage of search should enable us to access the image database rapidly and more efficiently. Also in a multimedia database, the database might contain various kinds of records like the images, the data in figures, the documents, audios, videos and so on. The concept of domain would classify them according to their categories. The domain could also be formulated based on the length of the certain features of the image like finding the images of a particular person or object where the length is of certain range value. Color could also define a domain where the images having a particular color or a combination of colors lie in one domain. Domain could be defined on the basis of objects only. For example, we can pick up image where the images of a triangle and square would be present. We can make it more specific by mentioning color and length or size of the triangle and the rectangle to pick up images like where the triangle is red colored and sides of length, say, (3, 3, 3) and rectangle of color white and sides of length, say, (4, 2). Variables Since in our research we are considering a multimedia database where a particular record in the database could be an image or a data record or a document and so on, the definition of variables would vary depending upon the type of records we are considering. In an image database, where we would consider twodimensional pictures, the image of an object or a person have to be measured. If we try to measure it graphically, then the size could be measured in terms of x and y coordinates. Therefore, size would be a very important variable in our research.
124
Image Segmentation
Color could be another very important variable. There are, as we know, many colors available and for specific definition of the image, the color would play a very vital role. It is possible to define colors digitally in the sense that each color could be given a digital number like red =1, blue = 2 and so on. Location would point towards where a particular object of interest is situated. If we were interested in finding the image of a person with a certain background at a particular point of the image, then location would tell us about where it is present. In a graphically defined image, the coordinate of the center of the image would give information about the location of the object. Distance between two particular objects of interest is a variable to be considered. Sometime in an image depicting a geographic picture, the distance between two points is very important. Here also we should be able to measure the distance between two objects by applying graphical methods. Motion of objects, like storm or cloud moving in a satellite picture, is also an important variable. We have to measure the distance travelled by the object in the picture and then convert it into kilometers and notice the time difference. From these we can measure the speed, velocity and so on of the object. Constraints It could be a very good idea to define an image in terms of various constraints in the sense that constraints help define the image more specifically. In our case of multimedia database, where various kinds of data could be there, the concept of constraints is very important. For example, if the image is that of a rectangle, we know one of the constraints would be that the number of sides of the object is 4. Then the second constraint would be opposite sides are parallel. The third one would be opposite sides are equal. If we include the emergence phenomenon, then if there is a diagonal drawn on it, this would give rise to two triangles. These constraints together could define the image successfully. In an image of a geographical map of any part of the world, the concept of constraint would be effective in finding the location. If we are interested in finding a place, for example, an island with triangular shape, then obviously the constraint would be number of sides is 3. If we have more information about sides like whether any two sides are same or all sides are same or all sides are of different size, then this kind of information should help us identify the object more accurately. Similarities In a database containing only data, the input may be a query with certain constraints like to pick up records from a SALARY database where salary is greater than, say, 30000. In relational database, as we know, this can be accomplished by a SQL command with the following kind of statement: SEL * FROM SALARY_DB WHERE SALARY > 30000. This would pick up all the records with salary > 30000. In our multimedia database system, this kind of queries could also be made and we can handle them with this kind of or more complicated SQL statements. But when the input is in the form of image, then we have to find the similarities of the input in the image part of the database. The basic approach to the problem of similarity is to find
Using Emergence Phenomenon in Meaningful Image Segmentation for ContentbasedImage Retrieval
125
certain parametric values as well as some coordinates of the input image. Then we find the same for various image of the image database and pick up records where some matching occurs. Of course, we study the emergence phenomenon in both input and images of an image database while calculating parameters. For example, if we want to find similarities involving a triangular figure as input, then some of the parametric values could be defined like, number of sides which is 3, length of each sides, color of the triangle. Based on these values and constraints, we can find out similarities in the image database. But in the image database, there could be figures like squares or rectangles with a diagonal drawn on them. Then obviously this diagonal gives rise to two triangles according to emergence. So we have to study these cases too, find out the parameters of these triangles to see whether they match our parameters from the input. Indexing In the early stage of data processing, there was no established conception of indexing. Most of the data files were accessed sequentially. This was pretty slow and inefficient particularly when the data file is big enough. To get rid of this problem, the concept of indexing came to the picture. At the initial stage, a number is used to be given against each record by the system in a file created on disk. We could specify these numbers to access any record randomly. Then came the concept of Indexed Sequential Access Method where instead of assigning separate number against each record, a field or a combination of fields were started being used as key. There could be two kinds of indexing, one where the key value in a particular file is unique and the other where the key value could be duplicate. The search method is called Binary Search where to find a particular key value, the whole file is divided into two halves and the part, which contains the particular key value we are searching, is taken and then divided into two halves again. The part here which contains the key value is again taken and divided into two halves. This process continues until it finds the match against the key value. Nowadays an old file system is hardly used in maintaining computer records. Instead, a database system has been developed. The latest development in this field is Relational Database System, which contains tables to store data. In our problem of dealing with multimedia databases, which would contain images, the concept of indexing is very important. We are trying to develop a more sophisticated method of indexing where there won't be any clearcut definition of index against the images, but indexes would be defined based on our study of emergence phenomenon of each of the image. Sometime to locate a particular spot in the geographic map of a part of the world, an input image would point to a particular part and that particular part in one or more than one image could be the outcome of emergence or it could be straight away present in the map without any emergence. In either case, input image refers to an index, which is nothing but that particular spot of the map. 2.5.2 Model of the emergence index Emergence indexes can be defined out of five factors as discussed in section 2.5.1. EI = f(D, F, V, C, E)
(1)
Where EI stands for emergence index, D for domain where the image belongs, F for features, V for variables which can define the feature’s constraints under which the features are defined, C for constraints and E for emergence characteristics of images.
126
Image Segmentation
We believe any image, static or in motion, could be expressed semantically in terms of the above mentioned five parameters. Construction We take the case of a square with a diagonal, as mentioned earlier, to build up an emergence index. If this is an image in the database, then firstly we have to put it under certain domain D for the ease of accessing. Since images are generated in enormous volume, we have to put them in various separate entities or tables according to certain classification rather than in one table which could be extremely time consuming to access. The table that would contain this square image record would define the domain of the image. We can term it as TAB1. To define the second factor F, we find the number of maximum sides present would be 5, where there are 4 regular sides and 1 diagonal. The variables are a, b, c, d, e where first four define the perimeter of the square and e the diagonal. The constraints c are a = b = c = d since it is a square. The emergence E is composed of two triangles with sides a ,b, e and c, d, e. Hence Emergence Index EI = {TAB1; 5; a, b, c, d, e; a = b = c = d; (a, b, e and c, d, e)}
3. Use of emergence phenomenon We provide a very simple example of how emergence phenomenon can give rise to meanings for a segmented image where apparently no meaning exists.
Fig. 1.3. The image of a tiger in a picture before segmentation [12]
Fig. 1.4. The image of the tiger after segmentation [12]
Using Emergence Phenomenon in Meaningful Image Segmentation for ContentbasedImage Retrieval
127
Here we see in Figure 1.3 the image is very clearly of a tiger. We segment the while picture and get the image of the tiger in Figure 1.4. But we note that the figure does not clearly point to the image of the tiger but indicates an animal. So we can interpret the image of the tiger in Figure 1.4 as that of an animal instead of a tiger. This is the outcome of emergence phenomenon where we are able to interpret the segmented image in a different way than original picture shows or in other words we extract different meaning of the segmented image. This would enable us to ascribe some meanings to segmented image which otherwise after segmentation may not deliver any meaning. This way through emergence phenomenon we interpret segmented image in a meaningful way.
4. Suggested future directions of research There are various attempts made to extract semantic meanings from an image to fillin the semantic gap between lowlevel features and highlevel semantic meanings which can arise from image segmentation. These include Latent Semantic Indexing (LSI), contextual search, user feedback, data clustering in the extraction of perceptual concepts, contentbased soft annotation (CBSA), image classifications, ontology, topdown, ontologically driven approaches and bottomup, automaticannotation approaches, using machine learning methods to associate lowlevel features with query concepts, using relevance feedback to learn users' intention, generating semantic template to support highlevel image retrieval, fusing the evidences from HTML text and the visual content of images for WWW image retrieval , use of ontology which represent taskspecific attributes, objects, and relations, and relate these to the processing modules available for their detection and recognition, use of contextawareness for identifying image semantics and relationships to contribute to closing the semantic gap between user information requests and the shortcomings of current contentbased image retrieval techniques , enhanced ICBIR system which allows users to input partial relevance which includes not only relevance extent but also relevance reason for a multiphase retrieval where partial relevance can adapt to the user's searching intention in a fromcoarsetofine manner [2]. Although these are good, constructive progresses in solving the problem of semantic gap in CBIR, they cannot define the semantic meanings of an image specifically. They can contribute to some broad classification of the image in certain groups. To solve this problem we have to develop devices to define the semantic meanings of an image very specifically from lowlevel features and that should be done automatically without users’ interaction. We seem to be still far away from this objectivity. As we have shown in Section 3 emergence phenomenon can bring meanings out of a segmented image where apparently no meaning could be found. We plan to work on the theory of emergence index as described in Section 2 in future to generate softwares to provide the assistance in identifying the images from a segmentation.
5. Conclusion In this paper we studied the problem of extracting meaningful image segmentation using emergence phenomenon. We plan to continue our work in this area to extract meanings from a more complex image segmentation where it is really difficult to find any meaning from the segmented image. Since this is a very important area of research with major
128
Image Segmentation
implications in all spheres of life beginning with medical images, the necessity of this study cannot be overestimated.
6. References [1] Cariani, P. (1992). Emergence and Artificial life, in Langton, C., Taylor, C., Farmer, J.D. and Rasmussen, S. (eds), Artificial Life II, AddisionWesley, Reading, pp. 775797 [2] Deb, S. (2010). Using Relevance Feedback in Bridging Semantic Gaps in Contentbased Image Retrieval, The Second International Conference on Advances in Future Internet (AFIN2010) , Venice/Mestre, Italy, July 18  25, 2010 [3] Forrest, S. (ed.) (1991). Emergent Computation, Elsevier, New York [4] Gero, J. S. and Maher, M. L. (1994). Computational Support for Emergence in Design, Information Technology in Design Conference, Moscow, September 1994 [5] Gero, J. S. (Year Unknown). Visual emergence in design collaboration, Key Center of Design Computing, University of Sydney [6] Gudivada, V. N. and Raghavan, V. V. (1995). ContentBased Image Retrieval Systems. IEEE, September 1995 [7] Langton, G .L. (1989). Artificial Life, AddisionWesley, Reading [8] Mehta, D., Diwakar, E.S.V.N.L.S., Jawahar, C.V. (2003). A Rulebased Approach to Image Retrieval, www.iiit.net/techreports/2003_8.pdf [9] Reducing semantic gap in content based image retrieval using region based relevance feedback techniques, (2006), http://www.gscit.monash.edu.au/gscitweb/seminar.php?id=41 [10] Yoshitaka, A., Kishida, S., Hirakawa, M. (1994). KnowledgeAssisted ContentBased Retrieval for Multimedia Databases, IEEE Multimedi,Winter 1994, pp.1221 [11] Zhau, R., Grosky, W.I. (2002). Bridging the semantic gap in image retrieval, http://citeseer.ist.psu.edu/497446.html [12] Image Segmentation, www.cs.toronto.edu/~jepson/csc2503/segmentation.pdf [13] Bridging the Semantic Gap in Image Retrieval http://citeseer.ist.psu.edu/497446.html
7 Dual Active Contour Models for Medical Image Segmentation Gilson Giraldi1, Paulo Rodrigues2, Jasjit Suri3,4 and Sameer Singh5 1National
Laboratory for Scientific Computing (LNCC) 2FEI University 3Biomedical Technologies, Inc. Denver, 4Idaho State University, 5Loughborough University, 1,2Brazil 3,4USA 5UK
1. Introduction Deformable Models, which includes the popular snake models (Kass et al., 1988) and deformable surfaces (McInerney & Terzopoulos, 1996; Suri & Editors, 2006), are well known techniques for boundary extraction and tracking in 2D/3D images. Basically, these models can be classified into three categories: parametric, geodesic snakes and implicit models. The relationships between these models have been demonstrated in several works in the literature (Sapiro, 1997). Parametric Deformable Models consist of a curve (or surface) which can dynamically conform to object shapes in response to internal (elastic) forces and external forces (image and constraint ones) (Suri & Editors, 2006). Snake models, also called active contour models, are 2D deformable models proposed by Kass at al. (Kass et al., 1988) which have been successfully applied in a variety of problems in computer vision and image analysis. Its mathematical formulation makes easier to integrate image data, an initial estimated, desired contour properties and knowledgebased constraints, in a single extraction process (Suri & Editors, 2006). In fact, despite of the mentioned capabilities, parametric models in general can not deal with topological changes. Among the approaches to deal with the topological limitations of the traditional snake model (Bischoff & Kobbeit, 2004; Oliveira et al., 2004), the TSnakes has the advantage of being a general one (McInerney & Terzopoulos, 1999). Besides, parametric models are too sensitive to their initial conditions due to nonconvexity problems (see (Davatzikos & Prince, 1999) and references therein). To address this limitation some authors have proposed multiscale techniques (Leymarie & Levine, 1993), dynamic program (DP) (Amini et al., 1990)and dual methods, also called dual snakes (Gunn & Nixon, 1997). The noninvariance under affine transformations is another limitation of the traditional snake models. As a consequence, the internal energy is sensitive to distortions due to changes in viewing geometry. From a dynamical point of view, it means that the elastic forces may
130
Image Segmentation
affect the efficiency of the energy minimization process (Ip & Shen, 1998). Some methods have been proposed to address this problem (Giraldi & Oliveira, 2004; Ip & Shen, 1998), even in the context of dual active contour models (Gunn & Nixon, 1997). The basic idea of the dual snakes is to reject local minima by using two contours: one which contracts from outside the target and one which expands from inside. Such proposal makes possible to reduce the sensitivity to initialization through the comparison between the two contours energy and positions. The two contours are interlinked to provide a driving force to carry the contours out of local minima, which makes the solution less sensitive to the initial position (Gunn & Nixon, 1997). In (Giraldi et al., 2000b), it is presented an extension of the dual method through the TSnakes. The dual approach was embedded in the TSnakes framework to propose a generalized dual method: one Tsnake contracts and splits from outside the target(s) and the other one(s) expand(s) from inside in the process of seeking for the objects boundaries in an image. Such generalization, called the DualTSnakes model, allows to address some limitations of the dual approach proposed in (Gunn & Nixon, 1997) and offers an efficient framework for dual surface models. In (Suri & Editors, 2006) some of us present an implicit formulation for dual snakes, based on the level set approach. The key idea of that work is to view the inner/outer contours as a level set of a suitable embedding function. The mathematical background of the method is explained and its utility for segmentation of cell images discussed. Besides, a fast dual front implementation of active contours has been proposed in (Li & Yezzi, 2005) and applied for segmentation of 2D images. The method is motivated by minimal path technique (Cohen, 2001; Cohen & Kimmel, 1996) and uses fast marching methods (Sethian, 1999) to compute the minimal partition curve that represents the object boundary. The same formulation can be used for both 2D and 3D and it was applied for segmentation of the brain cortex (Li & Yezzi, 2005). Dual active contour models have been applied for feature and geometric measures extraction (Gunn, 1996), boundary extraction on ultrasound images (Chen et al., 2001; 2002), brain surface extraction from PET images (Tohka et al., 2004) and cell image segmentation based on a two stage approach (Bamford & Lovell, 1997; Giraldi et al., 2000b) : (1) the region of interest is reduced; (2) a search based technique, like dynamic programming, is used to find the object boundaries. In this chapter we review parametric and implicit dual snake models. Following, in section 2, we offer some background for the material. Section 3 gives a review of parametric dual models. The DualTSnakes algorithm and the cellbased dual snake model are discussed in this section. Following, in section 4 we describe the implicit formulations for dual snakes: the DualLevelSet and DualFront methods. During the presentation of these methods, we show results in medical image segmentation. We offer a discussion on section 5 by comparing the presented methods and by pointing out drawbacks of dual approaches. In section 6 we present the conclusions and future developments in dual approaches.
2. Background review The original snake model (Kass et al., 1988) is formulated as a functional energy minimization process that consists of an initial model which is carried to the desired object boundary by forces described by the EulerLagrange equations. In a different way, the snake evolution can be formulated by local deformations to dynamically reshape the initial model
Dual Active Contour Models for Medical Image Segmentation
131
in a process which do not apply minimization techniques explicitly. This approach is used in the TSnakes model which is revised in section 2.1. On the other hand, implicit formulations based on the level set method can be used, as we shall see in section 2.2. 2.1 TSnakes model The classical snake model proposed in (Kass et al., 1988) do not incorporate topological changes. Among the proposals for incorporating topological capabilities to parametric snake models (Bischoff & Kobbeit, 2004; Oliveira et al., 2004), the TSnakes approach (McInerney & Terzopoulos, 1999) has the advantage of been a general one, in the sense that the same formulation can be used for 2D and 3D, for both splits and merges. The TSnakes approach is composed basically by four components (McInerney & Terzopoulos, 1999): (1) a simple CFtriangulation of the image domain; (2) projection of each snake over the grid; (3) a binary function called characteristic function χ , defined on the grid nodes, which distinguishes the interior from the exterior of a snake; (4) a discrete snake model. To clear the ideas, consider the characteristic functions ( χ 1 and χ 2 ) relative to the two contours pictured in Figure 1. The vertices marked are those where max {χ 1 , χ 2 } = 1 . Observe that the merge of the curves belongs to the triangles in which the characteristic function changes value.
Fig. 1. Two snakes colliding with the inside grid nodes and snake points (snaxels) marked. Thus, from the points obtained in the step (2), we can choose a set of N points vi = ( xi , yi ) , i = 0,..., N − 1 to be connected to form a closed contour (TSnake). In (Giraldi et al., 2003)we evolve a TSnake based on a tensile force ( Bi ), an external (image) force ( f i ), and a normal (balloonlike) force ( Fi ), defined as follows:
{
}
⎛ vt − 2 vt + vt ⎞ i i +1 ⎟ Bi = w1 ⎜ i − 1 , 2 ⎜ ⎟ Δ s ( i) ⎝ ⎠
(1)
f i = γ∇P ( vi ) ,
(2)
G Fi = k ( signi ) ni ,
(3)
G where ni is the normal at the snaxel vi and w1 , γ ,k are force scale factors, signi = 1 if 2 I ( vi ) ≥ T and signi = 0 otherwise ( T is a threshold for the image I ) and P = − ∇I . The
132
Image Segmentation
utility of the balloonlike force given by expression (3) is twofold. Firstly, it avoids the unwanted contraction effect of the tensile force due to the inflation (see (Cohen, 1991) for details). Secondly, this force pushes the model towards the object(s) of interest, characterized by the threshold T . The external force f i given by expression (2) attracts the snake to object boundaries in the image. We update the TSnake position according to the evolution equation: vi(
t +Δt )
(
)
= vit + Δt Bit + Fit + f it ,
(4)
where Δt is a time step. The TSnakes model incorporates also an entropy condition: “once a node is burnt (passed over by the snake) it stays burnt “ (McInerney & Terzopoulos, 1999). A specific termination condition is defined based on the number of deformations steps (temperature) that a triangle was cut by a TSnake. A TSnake is considered to have reached its equilibrium state when the temperature of all the snaxels fall bellow a preset value (called ''freezing point'' in the TSnakes literature (McInerney & Terzopoulos, 1999)). 2.2 Level set In this section we review some details of the level set formulation (Malladi et al., 1995). The main idea of this method is to represent the deformable surface (or curve) as a level set
{x ∈ ℜ
3
}
G ( x ) = 0 of an embedding function:
G : ℜ3 × ℜ+ → ℜ,
(5)
such that the deformable surface (also called front in this formulation), at t=0, is given by a surface S :
{
}
S ( t = 0 ) = x ∈ ℜ3 G ( x , t = 0 ) = 0 ,
(6)
The next step is to find an Eulerian formulation for the front evolution. Following Sethian (Malladi Get al., 1995), let us suppose that the front evolves in the normal direction with velocity F that may be a function of the curvature, normal direction, etc. We need an equation for the evolution of G ( x , t ) , considering that the surface S is the level set given by:
{
}
S ( t ) = x ∈ ℜ3 G ( x , t ) = 0 ,
(7)
Let us take a point x ( t ) , t ∈ ℜ+ of the propagating front S . From its implicit definition given above we have:
G ( x ( t ) , t ) = 0,
(8)
Now, we can use the Chain Rule to compute the time derivative of this expression:
Gt + F ∇G = 0,
(9)
133
Dual Active Contour Models for Medical Image Segmentation
where F = dx / dt is called the speed function and ∇ is the gradient operator, with respect to x . An initial condition G ( x , t = 0 ) is required. A straightforward technique to define this function is to compute a signeddistance function as follows:
G ( x, t = 0 ) = ±d,
(10)
where d is the distance from x to the surface S ( t = 0 ) and the sign indicates if the point is interior () or exterior (+) to the initial front. The “fast marching method” (FMM) can be used to efficiently compute this function (Sethian, 1996). Finite difference schemes, based on an uniform grid, can be used to solve equation (9). The same entropy condition of TSurfaces (once a grid node is burnt it stays burnt) is incorporated in order to drive the model to the desired solution (in fact, TSurfaces was inspired on the level set model (McInerney & Terzopoulos, 1999)). In this higher dimensional formulation, topological changes can be efficiently implemented. Numerical schemes are stable, and the model is general in the sense that the same formulation holds for 2D and 3D, as well as for merge and splits. Besides, the surface G geometry is easily computed. For example, the front normal ( n ) and curvature (K) are given, respectively, by: G n = ∇G ( x , t ) ,
⎛ ∇G ( x , t ) K = ∇ ⋅⎜ ⎜ ∇G ( x , t ) ⎝
⎞ ⎟, ⎟ ⎠
(11)
where the gradient ( ∇ ) and the divergent ( ∇. ) are computed with respect to x. The update of the embedding function through expression (9) can be made cheaper if the narrowband technique is applied. The key idea of this method comes from the observation that the front can be moved by updating the level set function at a small set of points in the neighbourhood of the zero set instead of updating it at all the points on the domain (see (Malladi et al., 1995; Sethian, 1996) for details).
3. Parametric dual models Parametric dual active contour models have been applied for cell image segmentation (Chen et al., 2002; 2001; Giraldi et al., 2003), features and geometric measures extraction (Gunn, 1996; Gunn & Nixon, 1997). The main advantage of these methods against usual snake models is their capability to reject local minima by using two contours: one which contracts from outside the target and one which expands from inside. Such proposal makes possible to reduce the sensitivity to initialization, by enabling a comparison between the two contours energy, which is used to reject local minima. In what follows, we firstly present the original dual snake model, proposed by Gunn and Nixon (Gunn, 1996; Gunn & Nixon, 1997). Next, we review more recent approaches that incorporate topological abilities (the DualTSnakes) and improve the efficiency for ultrasound images (DualCell) 3.1 Original dual model The dual snake methodology was firstly proposed in (Gunn & Nixon, 1997). To obtain the conventional continuity and smoothness constraints, but removes the unwanted contraction force, a scale invariant internal energy function (shape model) is developed. In (Gunn &
134
Image Segmentation
Nixon, 1997) a snake is considered as a particle system
{vi = ( xi , yi ) , i = 0,..., N − 1}
whose
particles are linked by internal constraints. The shape model is accomplished by the following internal energy:
ER =
1 N
N −1
∑ Eint ( vi ) ;
Eint =
i =0
1 ⎛⎜ ei 2⎜ h ⎝
2
⎞ ⎟, ⎟ ⎠
(12)
where: ei =
1 1 ( vi −1 + vi + 1 ) − vi + θi R ( vi −1 + vi + 1 ) , 2 2
(13)
h is the average space step, R is a 90 o rotation matrix and θ i is related to the internal angle
φi in the vertex vi by: ⎛ ϕi ⎞ ⎟. ⎝ 2⎠
θ i = cot ⎜
(14)
The Figure 2 helps to understand the geometric meaning of these elements. In this figure, the vector ei is such that the triangle with vertices vi + 1 , vi + ei , vi − 1 is isosceles.
Fig. 2. Geometric elements of the local shape model. Reprinted from (Gunn & Nixon, 1997) It is clear that ER has a global minimum when ei = 0, i = 0,1, 2,..., N − 1 . From (13)(14) it can be shown that this happens when:
ϕi = π ( N − 2 ) / ( 2 N ) , i = 0,1,..., N − 1, which are the internal angles of a regular polygon with vertices given by the points vi (Gunn & Nixon, 1997). The energy (12) can be also shown to be rotation, translation and scale invariant (Gunn & Nixon, 1997). Therefore there is no tendency to contraction as already pointed out for the original snake model (see (Cohen, 1991) for details). With the internal energy given by expression (12), the curve is biased towards a regular polygon (Gunn & Nixon, 1997). As before, the external energy is defined by:
135
Dual Active Contour Models for Medical Image Segmentation 2
Eext ( vi ) = − ∇I ( vi ) .
(16)
The total energy of the model is given by: E=
1 N
N −1
∑ ( λEint ( vi ) + ( 1 − λ ) Eext ( vi ) ).
(17)
i =0
where λ is a smoothing parameter which lies between 0 and 1 (Gunn & Nixon, 1997). This expression guarantees that during the optimization process, the snake will seek for strong object boundaries, due to Eext , whose shape resembles a regular polygon, due to shape model given by expression (12). This fact makes easier to establish the correspondence (matching) between the points of the two contours because the form of the snake during the evolution is limited by the energy (12). The methodology takes advantage of this correspondence by proposing the driving force:
Fdriving = g ( t )
ui − vit ui − vit
,
(18)
where vit is the contour being processed at time t, uit is the contour remaining at rest and g ( t ) is the strength of the force. The termination condition adopted in (Gunn & Nixon, 1997) is the following one, based on low velocity criterion: t +1
vi max i
− vit < δ ,
(19)
where δ is a termination parameter. The dual approach consists in making the inner and outer contours evolve according the following algorithm: The contour with the highest energy is selected. If its motion remains below some termination condition then the driving force (18) is increased until it moves at a rate greater than the chosen threshold δ . When the energy begins to decrease, the added driving force is removed and the contour is allowed to come into equilibrium. The procedure is then repeated until both contours have found the same equilibrium position. 3.2 DualTSnakes algorithm The key idea behind this method is to explore the TSnakes framework to propose a generalized dual active contour model: one TSnake contracts and splits from outside the targets and another one expand from inside the targets (Giraldi et al., 2003; 2000b). To make the outer snake to contract and the inner ones to expand we assign an inward normal force to the first and an outward normal force to the others according to expressions (3). Also, to turn the TSnakes evolution interdependent we use the image energy and an affinity restriction. We use two different definitions for image energy: one for the outer contour ( Eouter ) and another one for the set of inner contours enclosed by it ( Einner ): Eouter =
N −1
∑ ⎛⎜⎝ − ∇I ( vi ) i =0
2
⎞ / N. ⎟ ⎠
(20)
136
Image Segmentation
Einner =
1 ⎛ m−1 ⎛ ⎜∑⎜ m ⎜⎝ k = 0 ⎜⎝
∑ ( − ∇I ( vi )
N k −1 i =0
2
) / N ⎞⎟⎟⎠ ⎟⎟⎠ . ⎞
k
(21)
where m is the number of inner curves, N , N k are the number of snaxels of the outer snake and of the inner snake k, respectively. The normalization is necessary in order to be compared. Otherwise, the snake energy would be a decreasing function of the number of snaxels and comparisons would not make sense. Following the dual approach methodology (Gunn & Nixon, 1997), if Einner > Eouter an inner curve must be chosen. To accomplish this, we use an affinity operator which estimates the pixels of the image most likely to lie on the boundaries of the objects. Based on this operator, we can assign to a snaxel the likelihood that it is close to a boundary. That likelihood is thresholded to obtain an affinity function that assigns to the snaxel a 01 value: ''0'' for the snaxels most likely to lie away from the target boundaries and ''1'' otherwise. Then, the inner curve with highest number of snaxels with affinity function value null is chosen. If Einner < Eouter the outer snake is evolved if the corresponding affinity function has null entries. Also, the balance between the energy/affinity of the outer and inner snakes allows to avoid local minima. For instance, if a TSnake has been frozen, we can increase the normal force at the snaxels where the affinity function is zero, that is, we add a driving force only to the snaxels most like to lie far from the boundary. The selfintersections that may happen during the evolution of a snake when increasing the normal force are naturally resolved by the TSnakes model. This is way we can use that added normal force to play the role of the driving force used by Gunn and Nixon (avoiding the matching problem required in (Gunn & Nixon, 1997)). To evaluate similarity between two contours, we use the difference between the characteristic function of the outer snake and the characteristic functions of the inner ones ( Characteristic _ Diff ). For example, in the case of the CF triangulation of the Figure 1 we can stop the motion of all snaxels of an inner snake inside a triangle σ if any of its vertex v ∈ σ has the two following properties: (a). All the six triangles adjacent to v have a vertex where Characteristic _ Diff = 0 ; (b). One of these triangles is crossed by the outer contour The freezing point (section 2.1) is used to indicate that a TSnake has found an equilibrium position. In what follows, we call Dual Snake a list of TSnakes where the first one is an outer contour and the other ones are inner contours. The algorithm can be summarized as follows: Algorithm 1: DualTSnakes Put all the dual snakes into a queue. repeat Pop out a dual snake from the queue; Use the energies (equations (20) and (21)) and the affinity function to decide the snake to be processed; if all snaxels of that snake are frozen repeat increase the normal force at those with affinity zero until the snake energy starts decreasing
137
Dual Active Contour Models for Medical Image Segmentation
Remove that added normal force; repeat Evolve the snake until the temperature of all snaxels falls bellow the freezing point; } Analyze the Characteristic\_Diff of the current snake; if the snake being processed is close to a snake of the other type (inner/outer) then remove the dual snake from the queue.} else mount the resulting dual snake(s) and go to the beginning. until the queue is empty The experience with this method shows that it is very useful to reduce the search space. So, we proposed in (Giraldi et al., 2000b) a two stage segmentation approach: (1) the region of interest is reduced by the DualTSnakes; (2) a global minimization technique is used to find the object boundaries. In (Giraldi et al., 2000b) we apply the Viterbi algorithm, which is a dynamic program technique. The search space is constructed by discretizing each curve in N points and establishing a matching between them. Each pair of points is then connected by a segment which is subdivided in M points. This process provides a discrete search space, with NM points, that is pictured in the Figure 3:
Fig. 3. Search space obtained through a matching between inner and outer snakes. The target boundary is then determined by minimizing the following energy functional (Gunn, 1996): Esnake =
N −1
∑ Ei .
(22)
i =0
Ei = α Eint ( vi − 1 , vi , vi + 1 ) + β Eext ( vi ) + λ Eline ( vi ) .
(23)
with Eint , Eext and Eline defined as follows: 2
⎛ v − 2 vi + vi + 1 ⎞ Eint ( vi − 1 , vi , vi + 1 ) = ⎜ i − 1 ⎟⎟ . ⎜ v −v i −1 i+1 ⎝ ⎠
(24)
Eext ( vi ) = − ∇I ( vi ) ,
(25)
Eline = ± I ( vi ) ,
138
Image Segmentation
where the real parameters α , β and λ must be chosen in advance. The Viterbi algorithm was also used in (Bamford & Lovell, 1997) and sometimes it is called nonevolutionary dual model, in the sense that it is not based on a curve evolution. The following example shows the application of the segmentation framework that combines the DualTSnakes and the Viterbi algorithm (Giraldi et al., 2000b; 2003) for a cell image. The Figure 4.a shows a blood cell obtained by an electronic microscope technique. When passband filter is applied, we get an edge map resembling a ribbon whose thickness depends on the kernel size of the used filter ( Figure 4.b). That is an ideal situation for applying DualTSnakes plus Viterbi because, firstly, the former extracts the ribbon (Figure 4.c). Then, the later is applied to the original image to give the final result (Figure 4.d).
(a)
(c)
(b)
(d)
Fig. 4. Search space obtained through a matching between inner and outer snakes. (a)Image to be processed. (b)BandPass filtered image. (c)DualTSnakes solution. (d)Viterbi solution. 3.3 CellBased dual snake model In (Chen et al., 2002) it is proposed a cellbased dual snake model for ultrasound image. Boundary extraction and segmentation for this kind of image are a much harder problem than for other image modalities, due to the speckle, the tissuerelated textures, and the artifacts resulting from the ultrasonic imaging process. To address such difficulties it is proposed in (Chen et al., 2002) a model which is devised into three main stages, namely, cell generation, cellbased deformation and contour smoothing. In the cellgeneration stage, the immersion watershed algorithm (Vincent & Soille, 1991) is used to generate the nonoverlapped cells. To alleviate the interference of speckle in cell generation, the speckle is reduced by using the multiscale Gaussian filters before computing the gradient map. The cell boundaries are defined as the watersheds formed in the gradient map of the specklereduced ultrasound images.
139
Dual Active Contour Models for Medical Image Segmentation
Once the cell decomposition of the image domain is performed, we must define the dual snakes and the evolution model. Thus, given an initial contour enclosing the region of interest (ROI), a set of minimum covering cells, C 1 , C 2 ,..., C N containing the ROI is found, based on the cells generated in the previous stage (Figure 5). Let Γ 0 and Γ i be the initial outer and inner snakes, respectively, which are pictured on Figure 5. The outer snake is defined as the outermost boundary of
N
C i . Suppose that the center of the ROI is the cell ∪ i =1
C 1 . Then, its boundary is the initial inner snake Γ i , also represented on Figure 5.
Let Γ1t and Γ 2 t be the inner and outer snakes, respectively, at the deformation step t. Then, the model energy E Γ 1t ∪ Γ 2 t = E Γ 1t + E Γ 2 t is defined by (Chen et al., 2002):
(
) ( ) ( )
Fig. 5. Region of interest (ROI) and initial snakes. Reprinted from (Chen et al., 2002)
( )
( )
( )
( )
( )
E Γti = α i Elen Γti + β i Eθ Γti + γ i Eext Γti + δ i EArea Γti .
(26)
where: N −1
( ) ∑
Elen Γti =
j =0
N −1
( ) ∑ cos
Eθ Γti =
vit, j − vit+ 1, j , ⎛ ut − ut i, j i, j+3 ⎜⎜ ut ut ⎝ i, j i, j+3
−1 ⎜
j =0
N −1
( ) ∑ ∇I ( v ) ,
Eext Γti =
( )
j =0
t i, j
( )
EArea Γti = Area Siot ,
(27) ⎞ ⎟, ⎟⎟ ⎠
(28)
(29)
(30)
where vit, j means the jth snaxel of the ith snake at time t, uit, j = vit, j − vit, j − 3 , Siot denote the set of cells enclosed by the inner and outer snakes at the deformation step t, and α i , β i , γ i , δ i , i = 0,1,..., N − 1 are real parameters that controls the influence of each term in the expression (26).
140
Image Segmentation
The first energy term Elen gives the perimeter of the contour Γtj . If we take α 1 < 0 and α 2 > 0 , then, minimizing this energy would force the inner snake into expanding outward and the outer snake into contracting. The term Eθ approximates the curvature along the snake and aims to control the smoothness of the contour. The third energy Eext is the external energy defined based on edge features. The last energy, EArea , is the area covered by the cells in Siot . When minimizing energy (26) this term will provide the attraction force to pull the inner and the outer snakes to each other. The energies (27)(30) may have different ranges. Therefore, the target function to be
( )
( ) itself, that is:
minimized is the normalized energy variation ΔE Γti rather than the E Γti
(
)
ΔE Γt1 , Γt2 ; Γt1+ 1 , Γt2+ 1 =
( ) ( ) ( Γ ) , ΔE ( Γ ) , ΔE ( Γ ) , ΔE ( Γ )
(31)
( )
( )
α i ΔElen Γti + β i ΔEθ Γti + γ i ΔEext Γti + δ i ΔEArea Γti .
where ΔElen
t i
θ
t i
ext
t i
Area
t i
(32)
have the general form:
(
)
ΔE Γt1 , Γt2 ; Γt1+ 1 , Γt2+ 1 =
(
)
(
) ( )
( ) ( )
( )
⎡ E Γ t + 1 + E Γt + 1 ⎤ − ⎡ E Γ t + E Γt ⎤ 2 2 ⎥ ρ ρ ⎣⎢ ρ 1 ⎦⎥ ⎢⎣ ρ 1 ⎦ t t ⎡E Γ + E Γ ⎤ 2 ⎥ ρ ⎢⎣ ρ 1 ⎦
with ρ ∈ {len ,θ , ext , Area} and α i , β i , γ i , δ i are the same of expression (26). Before to proceed, we need other definitions. Let S1t be the set of cells inside the ROI that intersect the curve Γt1 (similarly for S2t ). Besides, it is defined in (Chen et al., 2002) the operators Φ and Ψ such that:
( )
Φ Siot = Γt2
and
( )
Ψ Siot = Γt1 .
(35)
The deformation of the model is based on the two operators called cellerosion and the celldilation. These operators are described on Figure 6. The former, denoted by CE, is defined as:
(
)
{ }
(
)
(36)
(
)
(37)
CE Siot , C p = Siot − C p , C p ∈ S1t ∪ S2t ∩ Siot .
The celldilation, denoted by CD, is defined by:
(
)
{ }
CD Siot , C p = Siot ∪ C p , C p ∈ S1t ∪ S2t − Siot .
The cellbased dual snakes evolution is the greedy procedure that aims to minimize the normalized energy variation defined in expression (32). It is based on the following algorithm to find the minimum of the energy variation given by expression (32).
141
Dual Active Contour Models for Medical Image Segmentation
(a)
(b)
Fig. 6. (a)Erosion operator. (b) Dilation operation. Reprinted from (Chen et al., 2002) 1. 2.
( )
( )
Initialization: define Sio0 , Φ Sio0 = Γ 02 , Ψ Sio0 = Γ 01 and and set t = 0 . While
Sio0
≠ ∅,
Ce =
arg min (
Cp ∈
S1t ∪S2t
)
∩Siot
{
(
( (
)) ( (
( (
)) ∪ Ψ (CD (S
ΔEerode = ΔE Γt1 , Γt2 ; Φ CE Siot , C p , Ψ CE Siot , C p
{
(
C d = arg min ΔEdilate = ΔE Γt1 , Γt2 ; Φ CD Siot , C p
(
Cp ∈
S1t ∪S2t
)
− Siot
( ) = Ψ (S ) .
(
)))},
t io , C p
)))},
(38)
(39)
)
2.1. If ΔEerode < ΔEdilate , Siot + 1 = CE Siot , C e . Else, Siot + 1 = CD Siot , C d . 2.2.
Γt2+ 1
=Φ
( ) and Siot + 1
Γt1+ 1
t +1 io
Once the boundary is extracted based on this algorithm, some kind of smoothing process may be applied in order to improve it. This stage can be performed by an usual snake model, that is, the dual result is used to initialize a parametric (single) snake model. Other possibility, also discussed in (Chen et al., 2002), would be spline interpolation. The Figure 7 pictures the stages of the method for breast ultrasound segmentation. Figure 7.a pictures the original image with a benign lesion. This input image is then filtered by a gaussian kernel followed by a Sobel edge detector. The cells generated by the watershed algorithm are pictured on Figure 7.b. The initialization of the method and the obtained result (white contour) are shown on Figures 7.c and 7.d, respectively.
4. Implicit models In this section we review the implicit formulation for dual models: the DualLevelSet and the DualFront methods. The former maintains the philosophy of DualTSnakes: one snake contracts and splits from outside the targets and another ones expand from inside the targets (section 3.2). However, the snake model will be the level set described on section 2.2. The later, the DualFront method, uses fast marching methods (Sethian, 1999) to propagate two action maps until they met each other generating an interface that represents the boundary. These action maps are derived from potentials that take lower values near the
142
Image Segmentation
(a)
(b)
(c)
(d)
Fig. 7. (a)Original breast ultrasound image. (b) Cells generated by the watershed algorithm. (c)Initial snakes (white contours). (d) Final result. Reprinted from (Chen et al., 2002) desired boundaries. So, both the DualLevelSet and the DualFront methods keep the idea of using two linked processes to seek for the global minimum. 4.1 DualLevelSet approach In this section we maintain the philosophy of DualTSnakes: one snake contracts and splits from outside the targets and another ones expand from inside the targets (section 3.2). However, the snake model will be an implicit one. To set ideas, let us consider the Figure 8.a, which shows two contours bounding the search space and Figure 8.b that pictures a surface which zero level set is the union of the two contours just presented.
(a)
(b)
Fig. 8. (a)Dual snakes bounding the search space. (b) Initial function which zero level set is the two contours presented.
Dual Active Contour Models for Medical Image Segmentation
143
If the surface evolves such that the two contours get closer, we can obtain the same behavior of DualTSnakes. That is the key idea of the method proposed by some of us in (Suri & Editors, 2006). In order to accomplish this goal we must define a suitable speed function and an efficient numerical approach. For simplicity, we consider the one dimensional version of the problem pictured on Figure 9. In this case, the level set equation given by expression (9) can be written as: Gt +
∂G F = 0. ∂x
(40)
The main point is to design the speed function F such that Gt > 0 . Therefore, if we set the sign of F opposite to the one of Gx we get this goal, once: Gt = −
∂G F = 0. ∂x
(41)
Hence, the desired behavior can be obtained by the sign distribution of F , shown in Figure 9.
Fig. 9. Sign of speed function. However, we should notice that Gx = 0 for singular points. So, the values of G remain constant over these points because Gt becomes null. Thus, we should be careful about the surface evolution nearby the singular points because anomalies may happen. One possibility to avoid this problem is to stop front evolution before getting close to this point. Another possibility could be to change the evolution equation in order to allow the Gt ≠ 0 over singular points. Such proposal implies that the isolines may be not preserved, that is, they become a function of time also. Thus:
(
)
G x (t ) , t = y (t ) ,
(42)
consequently, by applying Chain rule: Gt +
∂G dx dy , = ∂x dt dt
(43)
Therefore, we should provide an speed function in y direction. ⎛ ∂G ⎞ ⎛ dx dy ⎞ , −1 ⎟ ⋅ ⎜ , ⎟ = 0, Gt + ⎜ ⎝ ∂x ⎠ ⎝ dt dt ⎠
(43)
144
Image Segmentation
We can write this expression as: Gt + F ∇G , −1 = 0,
(45)
where F is the speed function. For fronts in 3Dwe get: Gt +
∂G dx ∂G dy dz + = , ∂x dt ∂y dt dt
(46)
therefore: ⎛ ∂G ∂G ⎞ ⎛ dx dy dz ⎞ Gt + ⎜⎜ , , −1 ⎟⎟ ⋅ ⎜ , , ⎟ = 0, ∂ ∂ x y ⎝ ⎠ ⎝ dt dt dt ⎠
(47)
One way to deal with these models is through viscous conservation laws (Sethian, 1996). For example, expression (43) becomes: Gt +
∂G dx ∂ 2G =ε 2 , ∂x dt ∂x
(48)
If dy / dt is replaced by ε Gxx where ε is a new parameter. For 2D we will have: ⎛ ∂G ∂G ⎞ ⎛ dx dy ⎞ 2 , Gt + ⎜⎜ ⎟⎟ ⋅ ⎜⎝ dt , dt ⎟⎠ = ε∇ G , ∂ ∂ x y ⎝ ⎠
(49)
where ∇ 2 means the Laplace operator defined by: ∇ 2G =
∂ 2G ∂ 2G + ∂x 2 ∂y 2 .
(50)
In our model we will maintain the idea that the front evolves in the normal direction. Thus, expression (49) can be rewritten as: Gt + F ∇G = ε∇ 2G ,
(51)
following the same development to derive expression (9). Such model has been studied in the context of front propagation in (Malladi et al., 1995; Sethian, 1996). Once our application focus is shape recovery in a image I , we must choose a suitable speed function F as well as a convenient stopping term S to be added to the righthand side of equation (51). Among the possibilities (Suri et al., 2002), the following ones have been suitable for our DualLevelSet: 1 +αk
,
(52)
S = β ∇P ⋅ ∇G ,
(53)
F=
1 + ∇I
2
where k is the curvature, defined by expression (11), α , β are scale parameters and 2 P = − ∇I . Therefore, we are going to deal with the following level set model:
145
Dual Active Contour Models for Medical Image Segmentation
⎛ 1 +αk Gt = ⎜ ⎜ 1 + ∇I 2 ⎝
⎞ ⎟ ∇ G + ε∇ 2 G + β ∇ P ⋅ ∇ G , ⎟ ⎠
(54)
The evolution of the fronts follows this governing equation and are interdependent due to the embedding function. However, once the evolution stops, we must evaluate the similarity between the two contours and apply a driving velocity instead of the driving force of section 3.1. The numerical method is a first order one already known in the level set literature (Sethian, 1996). We have also simplified the initialization of the method through smoothed versions of step functions (see (Suri & Editors, 2006), for details). As usual in level set approaches, we use the narrow band method; that is, only the values of G within a tube placed around the front are updated. When the front moves near to the edge of the tube boundary, the computation is stopped and a new tube is built with the zero level set at the center. For our dual approach, the narrow band is attractive not only for computational aspects but also because it allows an efficient way to evaluate similarity between two contours. In fact, instead of using the criterion of section 3.2, we take the procedure pictured on Figure 10: Firstly, the intersection point is computed (Figure 10.a); then, we take a neighborhood of this point (Figure 10.b) and stop to update the function G in all the grid points inside it or we can set to zero the speed function for these points. We say that those grid points are frozen ones.
(a)
(b)
Fig. 10. (a)Narrow bands touching each other. (b) Neighborhood to define similarity between fronts. Once the fronts stop moving, we must decide in which grid points we add a driving velocity. It is an extra velocity term which goal is the same of the driving force in section 3.2; that is, to keep fronts moving again. Therefore, we get a less sensitive model to the initial position of the fronts. To accomplish this task we can add an extra velocity term to equation (54), called Vdrive . We must be careful when choosing the grid points to apply this term. As in the case of DualTSnakes, the fronts may be nearby the boundary somewhere, but far away from the target in another place. We should automatically realize this fact when the fronts stop moving. To accomplish this, we can use the affinity operator explained on section 3.2. Based on this operator, we can define an affinity function that assigns to a grid point inside the narrow band a 0 − 1 value: 0 for the grid points most likely to lie away from the target boundaries and 1 otherwise. Like in the DualTSnakes, such affinity operator can be defined through fuzzy segmentation methods (Giraldi et al., 2003), image transforms (Falcão et al., 2001),
146
Image Segmentation
region statistics, etc. The whole DualLevelSet algorithm can be summarized as follows: (1) Initialization through Step Functions; (2) Evolution until fronts stop. (3) Evaluate similarity. If frozen, stop. (4) Add Vdrive for some time steps. (5) After that, turnoff Vdrive . Go to step 2. Likewise in the DualTSnakes model, when all the grid points inside the narrow bands are frozen, we stop the DualLevelSet evolution and apply a search based algorithm to get the final result. Figure 11 shows the application of the DualLevelSet to segment the cell image of Figure 11.a. However, instead of taking a bandpass version of it, like in Figure 4.b, we just apply a low pass filter to smooth the image. The DualLevelSet parameters are: α = 0.1 , ε = 2.0 , β = 0.1 , T = 150 , Δt = 0.05 .
(a)
(b)
(c) Fig. 11. (a) Original image. (b) DualLevelSet result. (c) Final result. Once the two snakes in Figure 3 are very close to each other, a search method based on a greedy algorithm can be efficient and less expensive then the Viterbi (section 3.2). Firstly, we compute a curve located inbetween the two fronts by taking the midpoint of each segment of the search space in Figure 3. The obtained curve can be used to initialize a snake model based on a greedy algorithm that works as follows (Suri & Editors, 2006). For each snaxel vi we take a 3 × 3 neighborhood V , and for each pixel p ∈ V we compute: E ( p ) = α Eint ( vi − 1 , p , vi + 1 ) + β Eext ( p ) + λ Eline ( p ) .
(55)
147
Dual Active Contour Models for Medical Image Segmentation
where Eint , Eext and Eline are defined in equations (24) and (25), respectively. Then, we solve the problem:
{
}
p0 = arg min E ( p ) ; p ∈ V .
(56)
If E ( p0 ) < E ( vi ) then vi ← p0 . The Figure 11.c pictures the obtained result. 4.2 DualFront approach In (Li & Yezzi, 2005), a fast and flexible dualfront implementation of active contours is proposed by iteratively dilating an initial curve to form a narrow region and then finding the new closest potential weighted minimal partition curve inside. The method is motivated by minimal path technique (Cohen & Kimmel, 1996; Cohen, 2001). In this method, given a potential P > 0 and a point p in the domain Ω , the minimal action map U 0 ( p ) is defined as:
(
)
U 0 ( p ) = min ∫ P c ( s ) ds , Ap0 , p Ω
(57)
where P = P + w , with w been a constant, and Ap0 , p is the set of paths connecting p0 and p . Expression (57) gives the minimal energy integrated along the paths between the starting point p0 and any point p inside the domain Ω . Because the action map U0 has only one minimum value at the starting point p0 and is a convex function in Ω , it can be easily determined by solving the equation (Cohen & Kimmel, 1996): ∇U 0 = P , and U0 = 0.
(58)
If we set P = 1 / V then this expression becomes the so called Eikonal equation which can be efficiently solved by using fast marching methods (Sethian, 1999). Equations (57)(58) are the starting point for the dualfront technique (Li & Yezzi, 2005). So, given two points p0 , p1 ∈ Ω , the method computes the action maps U 0 ( p ) and U 1 ( p ) , respectively, through the solution of expression (58), seeking for the points p ∈ Ω such that: U0 ( p ) = U1 ( p ) .
(59)
At these points, the level set of the minimal action map U0 meets the level set of the minimal action map U1 generating the Voronoi diagram that decomposes the whole image into two regions containing the points p0 and p1 . We can generalize definition (57) for a set X ⊂ Ω through the expression:
()
U X ( q ) = min U p q ,
(60)
p∈X
that mens, U X ( q ) is the minimal energy along the paths of the set Ap ,q where p ∈ X . Therefore, given two curves cin and c out bounding the search space called Rn in the Figure 12, and two potentials Pin and Pout that takes lower values near desired boundaries, the dualfront algorithm firstly computes the minimal action maps U in and U out until these two
148
Image Segmentation
action maps meet each other. Then, the evolutions of the level sets of both the action maps stops and a minimal partition boundary is formed in the region Rn of the Figure 12. Mathematically, this boundary is the solution of the following equations: ∇U in = Pin , with U in ( cin ) = 0,
(61)
∇U out = Pout , with U out ( c out ) = 0,
(62)
U in ( p ) = U out ( p ) .
(63)
Fig. 12. One interaction of the DualFront algorithm. (a) Initial curve C = cn of the iteration n. (b) Search space defined through dilation of the initial curve cn with bounds cin and c out . (c) Obtained solution C new , the minimal partition curve. The curve C is replaced by the curve C new to initialize the next iteration. Reprinted from (Li & Yezzi, 2005) The dualfront approach is an iterative method which is picture on Figure 12. Firstly, the curves cin and c out are placed by user interaction or obtained by dilation of an initial curve, named by C in Figure 12. Then, the minimal partition boundary is computed by solving equations (61)(63). To perform this task, the actions U in and U out are computed inside the region Rn , through expressions (61) and (62), respectively, until condition (63) is achieved. The obtained result, named by cn + 1 will replace C for processing the next iteration. The method proceed until the distance between consecutive minimal partition curves is less than a predefined threshold δ , that means d ( cn , c n + 1 ) < δ (like in expressions (19)). The potentials Pin and Pout are defined in (Li & Yezzi, 2005) using the following general expressions which integrates region based and the edgebased information.
(
)
r b Pin = win ⋅ f I ( x , y ) − μin ,σ in2 + win ⋅ g ( ∇I ) + win ,
(
)
r 2 b Pout = wout ⋅ f I ( x , y ) − μout ,σ out + wout ⋅ g ( ∇I ) + wout ,
(64) (65)
Dual Active Contour Models for Medical Image Segmentation
where
149
μin ,σ in2 are the mean and variance of the image intensity inside region
2 are the mean and variance of the image intensity inside region ( Rin − Rin ∩ Rn ) , μout ,σ out r ( Rout − Rout ∩ Rin ) , and win , winb , win are parameters to be set in advance (the same for r b wout , wout , wout ).
The Figure 13 shows an example of the application of the dualfront method for 2D human brain MRI image where the segmentation objective is to find the interface between the gray matter and the white matter. This example is interesting to observe the sensitivity of the method against the width of the search space, called active region in (Li & Yezzi, 2005). We observe that the obtained result was much better for the narrower search space than for the other ones. In this test, the potentials in expressions (64)(65) are defined by setting r r b b win = wout = 1 , win = wout = 0.1 , win = wout = 0.1 , and f , g given by: f ( x , y ) = I ( x , y ) − μin ,
(66)
g ( x , y ) = 1 + ( ∇I ) ,
(67)
2
Fig. 13. Sensitivity of the of the DualFront against different sizes of the active region (search space): (a) The original 2D human brain MRI image and the initial curve; (b) The corresponding edge map obtained through the gradient information; (c)(f) Segmentation results obtained for a search space defined through morphological dilation of the initial curve with 5 × 5 , 7 × 7 , 11 × 11 , 15 × 15 pixels circle structuring elements, after 15 iterations, respectively. Reprinted from (Li & Yezzi, 2005)
5. Discussion An important point for dual snakes is how to proceed the evolution after both snakes comes at rest but are far from each other. The affinity operator in the DualTSnakes and DualLevelSet models is used in this step, in order to avoid that some snaxels pass over the
150
Image Segmentation
desired boundary, due to the driving term. Such procedure is bypassed in the original dual model (section 3.1) due to the shape model used. The corresponding energy term may prevents that a snake pass over a global minimum. However, the shape model given by expression (12) limits the application of the method for general shapes and topologies. On the other hand, in the cellbased dual model, the cell decomposition of the image domain provided by the watershed strongly reduces the search space. Therefore, the energy minimization method may be less sensitive to local minima. That is way authors do not take care about the possibility of snakes reach equilibrium position far from the target. However, it does not seems to be possible to prove that such situation never happens. In the DualFront, actually the level sets of the action map U give the evolution of the front. The velocity of the evolving front is decided by the potential. Therefore, both the potentials in expressions (64)(65) must be defined such that the velocity is much lower when the evolving fronts arrive the boundary. If the constant w > 0 , the front velocity will be never null, and so, fronts only stop when the two action maps meet each other. Such policy is interesting to pass over local minima but once there is no an energy balance between fronts the global minimum may be also lost. DualLevelSet as well as DualTSnakes are topologically adaptable deformable models which increases their range of applications. However, such generality has also a price: the care with local minima should be higher than in the original model, for example, because there is no a shape model to bias the solution to the desired shape. Also, the characteristic function in the DualTSnakes plays a similar role of the embedding function for the DualLevelSet in the sense that they provide the framework for topological changes. However, the update of the former is based on exhaustive tests (McInerney & Terzopoulos, 1999) while the later is just a consequence of the model evolution based on the equation (54). As a practical consequence, we observed that it is easier to implement the DualLevelSet than the DualTSnakes. Also, we observe in our experiments that sometimes it is more efficient to apply topologically adaptable dual models just to reduce the search space, and, then to apply a search based technique to get the final result. This procedure was exemplified in sections 3.2 and 4.1. It is attractive because it simplifies the choice of parameters for the dual method and makes the computational cost of the application of a global optimization technique smaller. Despite of the capabilities to reject local minima, dual models have also some disadvantages. Firstly, the method is at least two times more expensive than single approaches. Secondly, the initialization may be a tedious task because the user should set two curves at the beginning of the process. The choice of parameter values is also another point to be careful because in this case there are two snakes to be set. A fundamental and more difficult point for dual snakes is how to proceed the evolution after both snakes comes at rest but they are far from each other. The snake with higher energy must be evolved, but the method should automatically realize the snaxels most likely to lie away from the boundary. The affinity operator (section 3.2) was proposals to address this problem without imposing restrictions to the snake evolution.
6. Conclusions and the future of dual methods The original dual approach is an interesting technique to address the sensitivity to local minima of usual snake models. The idea of using two snakes to seek for the global minimum, originally proposed in (Gunn & Nixon, 1997), have been used and extended in
Dual Active Contour Models for Medical Image Segmentation
151
recent works. Topological capabilities were incorporated, implicity formulations were developed and a cellbased approach were designed in order to improve the efficiency. This chapter offered a review of these stateoftheart techniques as well as the background behind dual approaches. Dual methods can avoid local minima through the comparison between the two contours energy and positions but they are at least two times more expensive than single approaches. Therefore, the ultrasound images seems to be the main target for the application of dual snakes because the difficulties inherent to the segmentation of these images force the application of a more robust approach against local minima. The application of GPU techniques (Lefohn et al., 2003) must be considered in further works in order to improve the performance of the dual methods. The development of 3D dual approaches is another point. For example, DualLevelSet as well as DualTSnakes can be easily extended to 3D without any extra machinery. Both the DualTSnakes and the DualLevelSet methods are suitable for a parallel implementation in shared memory machines because, in the implicit formulation we must distribute the narrow band evolution while in the parametric case we should focus in the distribution of the TSnakes processing. Multigrid techniques can be also implemented in both methods through Adaptive Mesh Refinement schemes (Giraldi et al., 2000a; Sethian, 1996).
7. Acknowledgment Authors would like to thank the support provided by CNPq, CAPES (grant 094/2007) and FAPERJ (grant E26/170.030/2008)
8. References Amini, A. A., Weymouth, T. E. & Jain, R. C. (1990). Using dynamic programming for solving variational problems in vision, IEEE Trans. on Pattern Analysis and Machine Intel. 12(9): 855–867. Bamford, P. & Lovell, B. (1997). A twostage scene segmentation scheme for the automatic collection of cervical cell images, Proceedings of TENCON ’97, Brisbane, Australia. Bischoff, S. & Kobbeit, L. (2004). Snakes with topology control, The Visual Computer 20: 217–228. Chen, C.M., LU, H. H.S. & HSIAO, A.T. (2001). A dualsnake model of high penetrability for ultrasound image boundary extraction, Ultrasound in Med. Biol. 27(12): 16511665. Chen, C.M., LU, H. H.S. & HUANG, Y.S. (2002). Cellbased dual snake model: A new approach to extracting highly winding boundaries in the ultrasound images, Ultrasound in Med. Biol. 28(8): 10611073. Cohen, L. (2001). Multiple contour finding and perceptual grouping using minimal paths, Journal of Mathematical Imaging and Vision 14: 225–236. Cohen, L. D. (1991). On active contour models and balloons, CVGIP:Image Understanding 53(2): 211–218. Cohen, L. & Kimmel, R. (1996). Global minimum for active contour models: A minimal path approach, IEEE International Conference on CVPR (CVPR’96), pp. 666–673. Davatzikos, C. & Prince, J. (1999). Convexity analysis of active contour algorithms, Image and Vision Computing 17(1): 27–36.
152
Image Segmentation
Falco, A., da Cunha, B. & Lotufo, R. (2001). Design of connected operators using the image foresting transform, SPIE on Medical Imaging, Vol. 4322, pp. 468–479. Giraldi, G. A. & Oliveira, A. A. F. (2004). Invariant snakes and initialization of deformable models, Int. J. Image Graphics 4(3): 363–384. Giraldi, G. A., Strauss, E. & Oliveira, A. F. (2000a). A boundary extraction approach based on multiresolution methods and the Tsnakes framework, International Symposium on Computer Graphics, Image Processing and Vision (SIBGRAPI’2000). Giraldi, G. A., Strauss, E. & Oliveira, A. F. (2000b). A boundary extraction method based on DualTSnakes and dynamic programming, IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’2000). Giraldi, G. A., Strauss, E. & Oliveira, A. F. (2003). DualTSnakes model for medical imaging segmentation, Pattern Recognition Letters 24(7): 993–1003. Gunn, S. R. (1996). Dual Active Contour Models for Image Feature Extraction, PhD thesis, Faculty of Engineering and Applied Science, Department of Eletronics and Computer Science. Gunn, S. R. & Nixon, M. S. (1997). A robust snake implementation; a dual active contour, IEEE Trans. Pattern Anal. Mach. Intell 19(1): 63–68. Ip, H. H. S. & Shen, D. (1998). An affineinvariant active contour model (aisnake) for modelbased segmentation, Image and Vision Computing 16(2): 135–146. Kass, M., Witkin, A. & Terzopoulos, D. (1988). Snakes: Active contour models, International Journal of Computer Vision 1(4): 321–331. Lefohn, A., Cates, J. E. & Whitaker, R. T. (2003). Interactive, gpubased level sets for 3d segmentation, In: Medical Image Computing and Computer Assisted Intervention (MICCAI), pp. 564–572. Leymarie, F. & Levine, M. D. (1993). Tracking deformable objects in the plane using and active contour model, IEEE Trans. Pattern Anal. Mach. Intell. 15(6): 617–634. Li, H. & Yezzi, A. (2005). Local or global minima: flexible dual front active contours, Proc. Of WorkshopComp. Vis. for Biom. Image App. (CVBIA 05, pp. 356–366. Malladi, R., Sethian, J. A. & Vemuri, B. C. (1995). Shape modeling with front propagation: A level set approach, IEEE Trans. Pattern Anal. Mach. Intell. 17(2): 158–175. McInerney, T. & Terzopoulos, D. (1996). Deformable models in medical image analysis: A survey, Medical Image Analysis 1(2). McInerney, T. & Terzopoulos, D. (1999). Topology adaptive deformable surfaces for medical image volume segmentation, IEEE Trans. on Medical Imaging 18(10): 840–850. Oliveira, A., Ribeiro, S., Farias, R., Esperanca, C. & Giraldi, G. (2004). Loop snakes: Snakes with enhanced topology control, SIBGRAPI ’04, pp. 364–371. Sapiro, G. (1997). Color snakes, Computer Vision and Image Understanding 68(2): 247–253. Sethian, J. A. (1996). Level Set Methods: Evolving Interfaces in Geometry, Fluid Mechanics, Computer Vision and Materials Sciences, Cambridge University Press. Sethian, J. A. (1999). Fast marching methods, SIAM Review 41: 199–235. Suri, J. & (Editors), A. F. (eds) (2006). deformable models: clinical and biological applications, Springer, NY.
8 Image Segmentation Using Maximum Spanning Tree on Affinity Matrix 1Department
Qiang He1 and CheeHung Henry Chu2
of Mathematics, Computer and Information Sciences Mississippi Valley State University Itta Bena, MS 38941 2The Center for Advanced Computer Studies University of Louisiana at Lafayette Lafayette, LA 705044330 USA
1. Introduction The objective of image segmentation and clustering is to extract meaningful regions out of an image. A graph theoretic approach, as an alternative to the gradientbased methods, is usually based on the eigenvectors of an affinity matrix [3,4,5,6,7]. The theoretical foundation of this development is the Spectral Graph Theory [1], through which the combinatorial features of a graph can be revealed by its spectra. This characteristic can be applied into graph partitioning and preconditioning. The typical eigendecompostion based segmentation work is called the normalized cuts [6]. The normalized cut measure incorporates the local similarity within cluster and total dissimilarity between clusters. The minimization of this measure is to solve the Rayleigh quotient, a generalized eigenvalue solving problem. However, solving a standard eigenvalue problem for all eigenvectors has exponential complexity and is very time consuming. Shi [6] made use of the sparsity of the affinity matrix and introduced the Lanczos method to simplify the computation of eigenvalues. In this paper, we give a new image segmentation algorithm using the maximum spanning tree [2]. Our method works on affinity matrix and addresses the physical meanings of an affinity matrix. Instead of computations of eigenvalues and eigenvectors, we proved that the image segmentation could be transformed into an optimization problem: finding the maximum spanning tree of the graph with image pixels as vertices and pairwise similarities as weights. Section 2 describes the related theory and Section 3 gives the experimental results on synthetic and real data to illustrate the performance of this algorithm. Finally, we draw a conclusion and discuss future work in Section 4.
2. Method descriptions In this section, we first discuss the characteristics of affinity matrix and then define an optimization measure based on the weighted graph associated with an image. The solution to the optimization problem satisfies the clustering standard with maximal withinclass similarity and minimum betweenclass similarity.
154
Image Segmentation
2.1 Affinity matrix The affinity matrix is a symmetric matrix and describes the pairwise pixel similarity. Every element Wi , j of an affinity matrix W represents the similarity between pixels i and j . There are various definitions for the similarity measures. In general, Wi , j can be defined as Wi , j = e
−xi − x j2 2σ 2
(1)
where  .  is Euclidean distance and σ is a free parameter. This is somewhat similar to the definition to Gaussian distribution. The characteristics of an affinity matrix (or similarity measures) are listed as follows. • Symmetric property The affinity matrix is symmetric, that is, Wi , j = W j , i . So it can be diagonalized. •
Unit normalization That is, 0 ≤ Wi , j ≤ 1 . The similarity Wi , j between pixels i and j increases as Wi , j goes
•
from 0 to 1 while the dissimilarity decreases. Transitive property If pixels i and j are similar and pixels j and k are similar, then pixels i and k are
•
similar. Coherence property That is, Wi , j ≥ Wl , k holds for ∀i , j , l , k if pixels i and j are in the same cluster while
•
pixels l and k are in different clusters. If similarity Wi , j is greater than some threshold, then we say that pixels i and j are similar.
2.2 Similarity measure for cluster and whole image Now, we define a similarity measure for one cluster and the whole image. If we consider the affinity matrix represents a weight matrix of a complete graph with all pixels as vertices. Then, there is a maximum spanning tree for this complete graph. Obviously, there is a subaffinity matrix for every cluster. We define the cluster similarity measure as the product of those weights in its maximum spanning tree. That is,
Sh =
Nh −1
∏ Pih
(2)
i =1
where h represents cluster number, N h is the number of entities (pixels) in cluster h , and Pih are weights in the maximum spanning tree of cluster h . Because of the symmetric, transitive, and coherence properties of affinity matrix, we can understand this as follows. Given a pixel p in cluster h , in order to find all pixels in cluster h , we first find the pixel q with maximum similarity to pixel p . Then we find another pixel not in set { p, q} , but with maximum similarity either with p or with q . Repeatedly, until all pixels of cluster h are found. We can see that this measure is reasonable to represent the maximum withincluster similarity for cluster h .
Image Segmentation Using Maximum Spanning Tree on Affinity Matrix
155
After we define cluster similarity measure, we further define a similarity S for the whole image, as follows c Nh −1
c
S = ∏ Sh = ∏ h =1
∏ Pih
(3)
h =1 i =1
where c is the number of clusters of image or number of segmentation components. For convenience, sometimes, we use log on S . We have c Nh −1
c
log S = ∑ logSh = ∑ h =1
∑ logPih
(4)
h =1 i =1
Next, we will show that to maximize the similarity measure S is to maximize the withincluster similarity and minimize the betweencluster similarity, which is preferred by the clustering and image segmentation. Proposition 1. The following optimization problem c
argmax S = argmax h,i
h,i
N h −1
∏∏P
i
h
(5)
h =1 i =1
guarantees that the withincluster similarity is maximum and the betweencluster similarity is minimal. Proof. By contradiction. Assume that there is a pixel p in cluster m is misclassified into cluster n. In the maximum spanning tree of cluster m, pixel p either connects two edges in the middle of the tree or connects one edge as a leaf node. Consider that the pixel p is removed from the cluster m. If p is a leaf node, then one its associated edge (also the weight) is removed from the maximum spanning tree. If p is in the middle of the tree, then two its associated edges (also the weights) are removed from the maximum spanning tree. But a new edge must be added to connect the two separate parts into a new maximum spanning tree. When pixel p is added into cluster n, it is either in the middle of the tree or exists as a leaf node. However, because of the coherence property of affinity matrix, the pairwise similarity between p and any pixel in cluster n is the smallest in the maximum spanning tree of cluster n, then p can not be added in the middle of the maximum spanning tree of cluster n. So p is added as a leaf node and one more edge (also the weight) is added onto the new maximum spanning tree of cluster n. If p is removed as a leaf node from the cluster m, its cluster similarity measure Sm becomes ′ = Sm
Sm wr
(6)
where wr is the removed weight. If p is removed as a node in the middle of the maximum spanning tree of cluster m, its cluster similarity measure Sm becomes
156
Image Segmentation
′ = Sm
waSm wb wc
(7)
where wb and wc are the removed weights from the maximum spanning tree and wa is the added new weight. From Prim’s algorithm [2] (for minimal spanning tree, but inverse weights of maximum spanning tree, we can use it), wa ≤ wb or wa ≤ wc . Or else, wa > wb and wa > wc , wa will be in the maximum spanning tree. After p is added as a leaf node into the cluster n, its cluster similarity measure Sn becomes Sn′ = wdSn
(8)
where wd is the added weight. Because of coherence property, wd < wr , wa , wb , wc . Then either,
Sm w × wdSn = d SmSn < SmSn wr wr
(9)
waSm ww × wdSn = a d SmSn < SmSn wb wc wb wc
(10)
′ × Sn′ = Sm or
′ × Sn′ = Sm
Therefore, the maximum withincluster similarity and the minimal betweencluster similarity are guaranteed under the above similarity measure. 2.3 Maximum spanning tree After we define the above optimization problem, we want to solve it. We show that the above optimization problem can be solved by finding a maximum spanning tree for the complete weighted graph of an image. First, we give a brief introduction to Prim's algorithm, which, in graph theory, is used to find a minimum spanning tree for a connected weighted graph. If we inverse all weights of affinity matrix, to find a maximum spanning tree of the original graph is equavalent to finding the minimum spanning tree of the graph with new weights. So Prim's algorithm can be used to find a maximum spanning tree. Prim's algorithm is an algorithm that finds a minimum spanning tree for a connected weighted graph, where the sum of weights of all the edges in the tree is minimized. If the graph is not connected, then it only finds a minimum spanning tree for one of the connected components. The time complexity of the Prim's algorithm is O(ElogV ) , where E is the number of edges in the graph and V  is the number of nodes. For a complete graph, the number of ⎛V ⎞ V (V −1) edges is E= ⎜ . This is also the maximum number of edges that a graph ⎟= 2 ⎝ 2 ⎠
can have. So the time complexity of the Prim's algorithm is also O(V 2 logV ) . If we only
Image Segmentation Using Maximum Spanning Tree on Affinity Matrix
157
use the transitive property of affinity matrix and compute the local similarity for a pixel, then there are only eight similarities for a pixel between it and its eight neighboring pixels. The number of edges becomes E= 8V  and the complexity becomes O(V logV ) . This will reduce the complexity considerably. On the other hand, the complexity to a standard eigenfunction problems takes O(V 3 ) , where V  is the number of nodes in the graph. Next, we show that the optimization problem can be solved by finding the maximum spanning tree and then removing the c − 1 minimal weights. Proposition 2. The following optimization problem c Nh −1
argmax S = argmax ∏ h,i
h,i
∏ Pih
h =1 i =1
can be solved by finding the maximum spanning tree of the graph associated with the image and then removing the c − 1 minimal weights. Proof. Prim’s algorithm is used to find the maximum spanning tree. Searching the maximum spanning tree of the whole image starts from a pixel x in some cluster m. According to coherence property of affinity matrix, the maximum spanning tree of cluster m must be put into the maximum spanning tree of the image first. Then the maximum spanning tree of cluster m will connect some pixel y in another cluster n through a betweencluster edge. From pixel y, the maximum spanning tree of cluster n is put into the maximum spanning tree of the image next. Repeatedly, all maximum spanning trees of clusters will be put in the maximum spanning tree of the image. The c − 1 minimal weights connect those maximum spanning trees of c clusters into the maximum spanning tree of the whole image. Therefore, we obtain the solution N −1
max S =
T c −1
∏ wl l =1
=
∏ Tk k =1 c −1
∏ wl
(11)
l =1
where T=
N −1
∏ Tk
is the product of all weights in the maximum spanning tree of the whole image.
k =1
wl are the c − 1 minimal weights.
3. Experimental results We test our algorithm using synthetic data and real data. The synthetic data are binary data, generated by drawing white squares on the black background. The real data is a kid picture. The number of clusters (components) is give as prior here. The selection of number of clusters is a model selection problem that depends on the application, and is beyond the scope of our discussion here.
158
Image Segmentation
Test on synthetic data
The synthetic data are a picture with two white squares on the black background. The similarity is computed using Equation 1 based on intensity values. We chose σ = 0.1. Then the withinclass similarity is
e
−xi − x j2 2σ 2
=e
−0 − 02 2 ⋅0.12
=e
−1 − 12 2 ⋅0.12
0
=e =1
(12)
and the betweenclass similarity is
e
−xi − x j2 2σ 2
=e
−1 − 02 2 ⋅0.12
=e
−50
≈0
(13)
For the data with two white squares, the maximum spanning tree of the image has two edges with zero weights. All other edges have weight one. The zero weight edges are two minimal weights and separate the maximum spanning tree of the image into three spanning trees. One of the new spanning trees represents the background and the other two represent two foreground squares. The segmentation results are shown in Figure 1. We can see that we obtain perfect results.
(a)
(b)
(c)
(d)
Fig. 1. Image segmentation on synthetic data. (a). the original image (b). the segmented background (white part) (c). the first segmented square (white part) (d). the second segmented square (white part) Test on real data
The real data is a kid picture. The similarity is computed using Equation 1 based on average values of three channel values. We still chose σ = 0.1. The segmentation results are shown in Figure 2. The results are reasonable. Since boundary contours of kid face and shirt are not consistent with the background and foreground (kid face or shirt), they can be clearly separated as we chose 5 components. In real images, because of noise and outliers in cluster, some withincluster similarities are very small. Correspondingly, some very small weights do not represent the betweencluster separation weights. In practice, we follow the order of edges from the Prim’s algorithm and pick those edge weights with considerable differences from (viz. smaller than) its previous and afterwards edge weights as the betweencluster separation edges. This method works robustly for real images. In practice, the χ 2 statistic [4] for histograms may give a better similarity measure for color and texture.
159
Image Segmentation Using Maximum Spanning Tree on Affinity Matrix
(a)
(b)
(c)
(d)
(e)
(f)
Fig. 2. Image segmentation on kid picture. (a). original image (b). the segmented background (white part) (c). the segmented kid face (white part) (d). the segmented shirt (white part) (e,f). the segmented contours (white part)
4. Conclusions In this paper, we presented a new graphbased image segmentation algorithm. This algorithm finds the maximum spanning tree of the graph associated with the image affinity matrix. Instead of solving eigenvalues and eigenvector, we proved that the image segmentation could be transformed into an optimization problem: finding the maximum spanning tree of a graph with image pixels as vertices and pairwise similarities as weights. In our future work, we will explore different similarity measures and test the segmentation algorithm on more data.
5. References [1] Chung, F.R.K.: Spectral Graph Theory. American Mathematical Society, 1997. [2] Cormen, T.H., Leiserson, C.E., Rivest, R. L., and Stein, C.: Introduction to Algorithms, Second Edition. MIT Press and McGrawHill, 2001. [3] Perona, P. and Freeman, W. T.: A factorization approach to grouping. In Proc ECCV, H. Burkardt and B. Neumann, editors, 1998, 655670. [4] Puzicha, J., Rubner, Y., Tomasi, C., and Buhmann, J. M.: Empirical evaluation of dissimilarity measures for color and texture. In Proceedings of the IEEE International Conference on Computer Vision, 1999, 11651173. [5] Scott, G.L. and LonguetHiggins, H. C.: Feature grouping by relocalisation of eigenvectors of the proximity matrix. In Proc. British Machine Vision Conference, 1990, 103108. [6] Shi, J. and Malik, J.: Normalized cuts and image segmentation. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, 1997, 731737.
160
Image Segmentation
[7] Weiss, Y.: Segmentation using eigenvectors: a unifying view. In Proceedings IEEE International Conference on Computer Vision, 1999, 975982.
9 Image Segmentation by Autoregressive Time Series Model PeiGee Peter Ho
Naval Undersea Warfare Center, Newport Rhode Island USA 1. Introduction The objective of the image segmentation is to simplify the representation of pictures into meaningful information by partitioning into image regions. Image segmentation is a software technique to locate certain objects or boundaries within an image. There are many algorithms and techniques have been developed to solve image segmentation problems for the past 20 years, though, none of the method is a general solution. Among the best, they are neural networks segmentation, onedimensional signal segmentation, multiscale segmentation, model based segmentation, graphic partitioning, region growing and Kmean clustering segmentation methods. In this chapter, the newly developed Autoregressive (AR) time series model will be introduced for image segmentation. Time series statistical models such as Autoregressive Moving Average (ARMA) were considered useful in describing the texture and contextual information of an image. To simplify the computation, a twodimensional (2D) Autoregressive (AR) model was used instead. The 2D AR time series model is particularly suitable to capture the rich image pixel contextual information. This model has been applied for both rough and smooth target surfaces and performed very well for image segmentation. In the typical statistical approach of image segmentation, there are two broad classes of segmentation procedures: The supervised and the unsupervised segmentation methods. The unsupervised segmentation procedure is the means by which pixels in the image are assigned to classes without prior knowledge of the existence or labeling of the classes. Whereas, in the supervised learning process, a teacher provides a label and cost function for each pattern in a training set and tries to minimize the sum of cost function for all patterns. Each method finds its own applications in the areas of the image analysis. The Support Vector Machine, a close cousin of classical multilayer perceptron neural networks and a newer supervised segmentation procedure, was adopted after feature extraction for single AR model image or pixel features vector extraction from multispectral image stack. On the other hand, the unsupervised region growing segmentation method was applied after univariate time series model was built. For the experimental results by applying the proposed AR time series segmentation model, the USC texture data set as well as satellite digital remote sensing image data are used. The algorithms performance comparisons with other existing contextual models such as Markov Random Field model, Kmeans, PCA, ICA …etc can be found in reference (Ho, 2008).
162
Image Segmentation
2. 2D Univariate time series image segmentation model and estimation Time series analysis (Shumway 2000) (Wei, 1990) has been used in many applications, specifically the weather forecast, atmosphereocean model for global climate changes and the Dow Jones economics analysis. Interesting enough, at the University of Massachusetts ,Dartmouth School of Marine Science and Technology (SMAST) professor Joachim Gröger (Gröger 2007) can successfully trace a group of cod fishes’ approximate swimming routes by time series model analysis on tidal data. In this section, we make use of time series concepts for image data mathematical formulation which is very new. To understand how, we introduce time series theory briefly. The primary objective of time series is to develop certain mathematical models that provide descriptions for sample data. We assume any image can be defined as a collection of random variables indexed according to the order they are obtained in timespace. For example, we consider a sequence of random variables x1 , x2 , x3 , ......., in general, {xt} indexed by t is the stochastic process. The adjacent points in time are correlated. Therefore, the value of series xt at time t depends in some fashion on the past values xt − 1 , xt − 2 , ........ Suppose that we let the value of the time series at some point of time t to be denoted by xt . A stationary time series is the one for which the probabilistic behavior
of
xt1 , xt2 , .......xtk is identical to that of the shifted set
xt1 + h , xt2 + h , ......., xtk + h . In an image segmentation application, the 2D image was scanned from left upper corner to right bottom as a sequence of time series pixel values. Further, to simplify the numerical calculations, we model each class of surface textures by 1st order and 2nd order Autoregressive stationary time series models. In another way of thinking, the twodimensional Markov model is a similar mathematical model to describe. By using time series model, when the withinobject interpixel correlation varies significantly from object to object, we can build effective imaging region segementation. The unsupervised Region Growing is a powerful image segmentation method for use in shape classification and analysis. Take one satellite remote sensing example to show, we present the LANDSAT 5 database in the area of Italy’s Lake Mulargias image data acquired in July 1996 to be used for the computing experiments with satisfactory preliminary results (figure 1, figure 2 and figure 3). The advanced statistical techniques, such as Gaussian distributed white noise error confidence interval calculations, sampling statistics based on mean and variance properties are adopted for automatic threshold finding during Region Growing iterations. The linear regression analysis with least mean squares error estimation is implemented as a time series system optimization scheme. 2.1 Image modeling and estimation theory The raw image data are statistically random and can be modeled with some basic ideas of time series analysis and stochastic processes. It is based on a model which treats the pixels (picture elements) of a digitized textural scene as a twoway seasonal time series. A time series is a set of observations, each one is recorded at a specific time t. The target image is well treated as a time series in which the observations are made at the fixed time intervals. Therefore, for a two dimensional image, each pixel gray tone information G(x,y) is equivalent to G(t) in the time series analysis. In the multispectral scanner images, the texture information is one of the important characteristics for machine vision analysis. The texture classification techniques include determination of a number of classes to which the texture region belongs. It estimates
Image Segmentation by Autoregressive Time Series Model
163
texture model parameters for a region and constructs the boundaries between different texture regions. The standard probability and the statistics concepts, such as autocovariance and sample covariance functions are considered. The techniques in parameter estimation theory to extract texture typed trend and seasonality from the observed time series are adapted for image segmentation analysis. Due to the fact that rarely is there an exact priori knowledge about the image data, a mathematical model is often used. The time series stationary Autoregressive (AR) models on both the firstorder and the secondorder with Region Growing algorithm are studied and the results are presented below. 2.2 Firstorder autoregressive image modeling and parameter estimation In the firstorder AR analysis, each gray tone pixel value of the image is estimated through its neighboring pixels. This is based on the assumption of a causal, linear time invariant remote sensing system. The gray level of the (i , j ) th pixel of the remote sensing image is
expressed by the equation: x(i , j ) = a + b * [ x(i − 1, j ) + x(i , j − 1)] + n(i , j )
(1)
where n(i , j ) is modeled as a Gaussian white noise random process. Let xˆ (i , j ) be an estimate of x(i , j ) . Note that for an 100 by 100 image array, i = 1 : 100 is the row index, j = 1 : 100 is the column index of the two dimensional image example. Starting with a set of
region seeds, the weighted coefficients are determined through unsupervised region growing iterative procedure to group pixels of a subregion into larger regions. The ground truth data were provided as “seeds” for initial processing. The criterion is to minimize the mean square error MSE formulated as (Seber 1977), (Papoulis 2002): 100 100
MSE = ∑∑ E{[ x(i , j ) − xˆ (i , j )]2 } i =1 j =1
which will be a minimum if setting ∂( MSE) =0 ∂a
∂( MSE) =0 ∂b
(2)
By ordinary linear regression calculation, we get the following optimal least squares estimator of the first order AR parameters: Let z(i , j ) = x( i − 1, j ) + x(i , j − 1)
Numerator
M N
= ∑∑ z( i , j ) * x(i , j )  M * N * E[ z(i , j )] * E[ x(i , j )]
(3)
i =1 j =1
Denominator
M N
= ∑∑ [ z(i , j )]2 − M * N * [ E( z(i , j ))]2 i =1 j =1
(4)
164
Image Segmentation
M = Total region growing sample row pixels number N = Total region growing sample column pixels number bˆ =
Numerator Denominator
(5)
aˆ = E( x(i , j )) − bˆ * E( z(i , j ))
(6)
Note that E( x( i , j )) denotes the expectation value. 2.3 Secondorder autoregressive image modeling and parameter estimation In the following analysis, each gray tone pixel value of the image is estimated through weighted summation of its neighborhood pixels. This is based on an assumption that a causal, linear time invariant remote sensing system can be modeled mathematically. The gray level of the (i , j ) th pixel of the remote sensing image is expressed by the following
formula: x(i , j ) = a + bx(i − 1, j ) + cx(i , j − 1) + dx(i − 1, j − 1) + n(i , j )
(7)
where n(i , j ) is assumed a Gaussian white noise random process. Let xˆ ( i , j ) be an estimate of x(i , j ) Note that i = 1 to M is the row index, j = 1 to N is the column index of the two dimensional image. Starting with a set of region seeds, the weighted coefficients are determined through unsupervised region growing iterative procedure to group pixels of a subregion into larger regions. The ground truth “seeds” data were used for initialization. Again, the same as the 1st order, the criterion is to minimize the mean square error MSE (Haykin 2003) that is formulated as: M N
MSE = ∑∑ E{[ x(i , j ) − xˆ (i , j )]2 }
(8)
i =1 j =1
which will be a minimum if setting ∂( MSE) = 0; ∂a
∂( MSE) = 0; ∂b
∂( MSE) = 0; ∂c
∂( MSE) =0; ∂d
(9)
By centering and scaling of the linear regression data, we can easily find the following matrix operation which relates estimated parameter matrix with covariance matrix of the sample of one image class region. P * Tˆ = Q Let
M ,N
∑ x(i , j ) be the abbreviation of ∑
(10)
x( i , j ) due to limited mathematical symbol spacing.
i = 1, j = 1
M = Total region growing sample row pixels number N = Total region growing sample column pixels number Then,
165
Image Segmentation by Autoregressive Time Series Model
⎡ aˆ ⎤ ⎢ ˆ⎥ ⎢b ⎥ Tˆ = ⎢ ⎥ cˆ ⎢ ⎥ ⎢⎣ dˆ ⎥⎦ ⎡ M*N ⎢ ⎢ ∑ x(i − 1, j ) P=⎢ ⎢ ∑ x(i , j − 1) ⎢ x(i − 1, j − 1) ⎣∑
⎡ ⎤ ∑ x(i , j ) ⎢ ⎥ − ( 1, ) ( , ) x i j x i j ∑ ⎥ Q=⎢ ⎢ ∑ x( i , j − 1)x( i , j ) ⎥ ⎢ ⎥ ⎢⎣ ∑ x(i − 1, j − 1)x(i , j )⎥⎦
(11)
= The optimized parameter matrix
(12)
∑ x(i − 1, j ) ∑ x(i − 1, j )2
∑ x(i , j − 1)
∑ x(i − 1, j − 1) ⎤⎥ x ( ∑ i − 1, j )x(i − 1, j − 1)⎥ (13) ⎥ 2 ∑ x(i − 1, j )x(i , j − 1) ∑ x(i , j − 1) ∑ x(i , j − 1)x(i − 1, j − 1)⎥ ∑ x(i − 1, j − 1)x(i − 1, j ) ∑ x(i − 1, j − 1)x(i , j − 1) ∑ x(i − 1, j − 1)2 ⎥⎦ ∑ x(i − 1, j )x(i , j − 1)
In which, the estimated parameters aˆ , bˆ , cˆ , dˆ can be found from Tˆ matrix. Also the optimal estimated pixel gray level can be calculated as
ˆ (i − 1, j ) + cx ˆ (i − 1, j − 1) ˆ (i , j − 1) + dx xˆ (i , j ) = aˆ + bx
(14)
The more iterations performed to grow the region, the more accurate the estimated parameters. e(i , j ) = xˆ (i , j ) − x(i , j )
Now, let
(15)
Decision making for region growing:
A two class discrimination decision can be made by: Set class_label = 1 if e(i , j ) < Threshold * Set class_label = 0
MSE( optimal )
(16)
otherwise
Where MSE( optimal ) is the mean square error calculation of the growing region. Based on statistical hypotheses and significance property of the sampling theory plus the Gaussian distribution nature of the pixel error populations, the 95% confidence interval threshold can be found as: Threshold =
2 /π +
1 − 2 /π
(17)
To prove (17) for the threshold: Threshold =
2 /π +
1 − 2 /π ;
let SD = MSE( optimal )
The error Gaussian distribution function can be expressed as
(18)
166
Image Segmentation
1
f(e(I,J))= ∞
2π SD
E(e(I,J))= 2 ∫ e( I , J )( 0
exp ⎡⎣ −1 / 2( e( I , J ) / SD)2 ⎤⎦
(19)
1 exp ⎡⎣ −1 / 2( e( I , J ) / SD)2 ⎤⎦ ) d(e(I,J)) 2π SD
= ( 2 / π ) * SD
(20) (21)
M e( I , J ) = E[ e( I , J ) ]
(22)
(σ e( I , J ) )2 = E{( e( I , J )  M e( I , J ) )2 }
(23)
= ( SD2 )  ( (
= (1
2
π
σ e( I , J ) =
2
π
) * SD)2
(24)
) ( SD2 )
(1 −
2
π
(25)
) *SD
(26)
For 95% confidence of 1sigma interval value e(I,J) < E[ e( I , J ) ] +
(1 −
2
π
) * (SD)=
= ( 2 /π +
2
π
* (SD) +
1 − 2 / π )* (SD)
(1 −
2
π
) *(SD)
(27) (28)
An example of experiment for explanation:
As shown in figure 1, figure 2 and figure 3, after certain iterations of Lake Mulargias (water) region growing (when we are assured that no more water pixel can be found), we stop processing. By unsupervised region growing the optimized parameter matrix Tˆ can be determined to classify the entire image. 2.4 Region growing algorithm Region Growing, as implied by its name, is an algorithm that groups pixel subregions into bigger areas. It starts with a set of known seeds (ground truth information, for example, in figure 2) to grow regions by merging to each seed point those neighborhood pixels that have similar statistical features. In remote sensing image analysis and segmentation application, there are just a few researchers who proposed or attempted to use region growing. As a matter of fact, its practical implementations have proven that in general, it does work well in most circumstances as far as region segmentation is concerned. In the IEEE IGARSS 2000 paper, Dr. Tilton and Lawrence (Tilton 2000) presented the interactive hierarchical segmentation to
Image Segmentation by Autoregressive Time Series Model
167
grow a region. But, this method has some limitations and can only apply to certain scenes only. The Region Growing method we proposed is a different approach and can deal with the shortfall they have. Other useful references on the subject can be found in Reference (Haralick 1992). A basic region growing procedure is Outlined in Table 1.
ITERATION_COUNT = 1; MAX_ITERATION = 100; Border of R = lake seed (water); Start with lake seed, calculate and estimate A, B, C, D parameters DO until ITERATION_COUNT > MAX_ITERATION or NOT LAKE CLASS Increment border of R by one pixel up, down, left, right to R’ for each pixel p at the border of R’R do if ( e(I,J) < Threshold * SD ), set CLASS of p = LAKE; else CLASS of p = OTHER; /* CLASSIFICATION */ Updated calculation of A,B,C,D parameters for R’ region with new CLASS LABEL End ITERATION_COUNT ++ ; End Table 1. AR Model Based Region Growing Algorithm Remote Sensing Original Image
Fig. 1. Original Lake Region Remote Sensing band 5 Image
Fig. 2. Ground Truth Information
168
Image Segmentation
Fig. 3. Segmentation Result After Region Growing Based On AR Model In this chapter, we have introduced a new univariate time series AR model based region growing algorithm, which attempts to extract a special property region such as water at Lake Mulagias. As a matter of fact, this new algorithm can be further used to apply to other remote sensing imagery area, such as corn field, city block, grassland, pasture … etc for scene classifications. The computer experiments show that the Time series Region Growing algorithm is efficient for the remote sensing image region segmentation problems. A Region growing is a procedure that group pixels or subregions into larger regions based on predefined criteria (Ho 2004). By starting with a set of seed points, it forms the growing regions by appending to each seed of the neighboring pixels that have similar properties to the seed points. The process is in the iteration mode as depicted in figure 4, where (a) is the region
(a)
(b)
Region Growing : (a) => (b) => (c)
(c) Fig. 4. Region Growing demonstration in graphics (a) => (b) => (c)
169
Image Segmentation by Autoregressive Time Series Model
before growing, (b) is in the process of growing one pixel layer to the right and one pixel layer toward bottom. After certain decision making, the new region was formed as in (c). There are many different ways of region growing depends on programmer’s preferences. Usually, the first step of image analysis is to partition an input image that is more than one uniform region into several homogeneous subimages (Gonzalez 2004). The region of homogeneity can be determined by many different properties. In region growing method, the image segmentation is implemented by extending some starting pixels or internal small area inside regions to boundaries between texture areas. Pixels in a homogeneous region are expected to be assigned by an identical class. The boundaries will be drawn by the difference in the adjacent class regions. We also need to make sure that regions with the same texture but in different parts of an image are labeled by the same class. In the remote sensing and USC texture type data examples, the texture property for each observed window is extracted from time series 2D AR model. Under each iteration, the unknown regions were segmented into blocks. Each internal region is expanded by comparing the properties of its surrounding blocks. As texture boundaries are reached, the size of blocks is reduced, each block might contains only one pixel at the end. Figure 5 shows an example result on USC data set for the effectiveness of the AR time series image segementation.
(a)
(b)
(c)
Fig. 5. (a) Orginal 3classes real texture image from USC data base (brickwall, grass and pigskin) (b) The segmentation result after 2D time series multiclass SVM in black and white display (c) The Segmentation result after 2D time series multiclass SVM in color display 2.5 Higher order AR model The 3rd order of AR image model is described as: x(i , j ) = a + bx(i − 1, j ) + cx(i , j − 1) + dx( i − 1, j − 1) + ex( i − 2, j ) + fx( i , j − 2) + n( i , j )
(29)
We have done some experiments on the same Lake Mulagias remote sensing data set as well as USC texture data set by 3rd order AR model, but the region segmentation results did not improve further. This told us that the term higher than 2nd order had no impact. The AR model order selection criteria is based on sample estimated error or residuals. 2.6 Experimental results The following figures from Figure 6 to Figure 13 are the texture and remote sensing segmentation results:
170
Image Segmentation
Fig. 6. Original TM remote sensing image of Italy Lake Mulagias
Fig. 7. Original Italy Lake Mulagias displayed in color format
Fig. 8. Segmentation Result by 2nd Order AR in black and white
Fig. 9. Segmentation Result by 2nd Order ARMA in color format
Fig. 10. Original USC brick and grass two classes natural scene
Fig. 11. Original USC brick and grass displayed in color format
Fig. 12. Texture Segmentation Result by 2nd Order AR in black and white
Fig. 13. Texture Segmentation Result by 2nd Order AR in color format
171
Image Segmentation by Autoregressive Time Series Model
The 2D time series model based support vector machine (Ho, 2007) was tested with real remote sensing images from satellite and the results in figure 14 shows that this method can perform well in separating lake region from city building with the accuracy of 96.9%.
(a)
(b)
(b)
(d)
Fig. 14. (a) Original 2 Classes Remote Sensing Image in black and white mode (b) Original 2 Classes Remote Sensing Image in color mode (c) The Segmentation Result 2 classes SVM Method (d) The Segmentation Result after 2 classes SVM Method in color display
3. Contexture image models A problem of particular importance in remote sensing image classification is the use of contextual information. All experiments indicate that classification performance improves with proper use of contextual information. Because of interest with texture analysis, and the fact that remote sensing data is rich in texture, this problem has been studied by pattern recognition researchers for some time with a number of approaches presented. Markov random field provides a rigorous mathematical characterization of contextual information of textures from neighboring pixels on an image. Basically the often rich
172
Image Segmentation
textural information in remote sensing image makes Markov random field a very powerful image model. Another image model is the 2D multivariate autoregressive time series analysis. The Markov random field model is for a single image while the 2D multivariate autoregeessive model is for the entire stack of images under consideration and treated as a vector time series. The two image models will be briefly presented in this section. 3.1 Markov random field Markov random field (MRF) contextual image model assumes that the texture is stochastic and stationary with conditional independence. MRF theory provides a basis for modeling contextual constraints in image processing, analysis, and interpretation (Geman, 1984). Several methods are introduced and published about 2D image restoration or reconstruction with MRF model. Very recently a comprehensive treatment of Markov random field in remote sensing is given by Serpico and Moser. The neighborhood system of a pixel x, in an MxN image, which contains the contextual information under consideration, consists of adjacent pixels with a Euclidean distance of say r. and r takes an integer value that is denoted as the order of neighborhood system. The neighborhood system with a definitive order is shown in Figure 15 (a), where 1, 2, ⋅⋅⋅⋅⋅⋅ denote the order of neighborhood system. The second order neighborhood system has eight neighboring pixels. Therefore, it is also called the 8neighborhood system. In a Markov random field the conditional probability density function of each pixel is dependent only on its neighborhood system So the MRF represents the local characteristics of x. The MRF can be causal or noncausal. Assume that the pixels are scanned sequentially top to down and left to right. The causal MRF depends only on the past while the noncausal or bilateral MRF depends not only on the past but also on the future as shown in Figure 15 (b). In general, the neighborhood system used in image modeling is symmetrical and sequentially ordered. Also, the noncausal Markov chain is usually selected as 2D image model in practice. Each observed data y can be regarded as the noisy version of original data x . In other words, y = x + n , where n is a sample of a white Gaussian noise field with variance σn2. The conditional probability density function of x given y is given by:
p( xy ) =
1 1 exp{ − 2 ( y − x )T ( y − x )} (2πσ n2 )− MN /2 2σ n
(30)
The estimation problem is to obtain the original data x from the observed data y . As proposed by Jeng and Woods (Jeng 1991), the relationship of a pixel in a compound Gaussian Markov random field (CGMRF) and its neighborhood system is presented by the collection of binary variables Co = {ckm,n : k = 1, 2, ⋅⋅⋅, 8; m = 1, 2, ⋅⋅⋅, M; n = 1, 2, ⋅⋅⋅, N }, where k is the same as the Figure 15 (b). If the neighbor pixels are independent, the associated ckm, is set to be one and otherwise ckm,n is equal to zero. Let A(C0) be the covariance matrix of x , which depends on C0. The estimation of original image x is obtained with observed y and the matrix A(C0), which should be estimated from the observed y . In general, we have p ( xC 0 , y ) ∝
(
2πσ n2
)
1 − MN /2
1 ⎪⎧ 1 ⎪⎫ T y − x ) I ( y − x )⎬ exp ⎨− 2 xT A (C 0 ) x − 2( 2σ n ⎪⎩ 2σ x ⎭⎪ ∗ Z (C 0 )
(31)
173
Image Segmentation by Autoregressive Time Series Model
where I is the identity matrix and Z(C 0 ) is a normalizing constant called partition function (Jeng 1991). Maximizing the equation gives us the MAP estimate of x −1
⎛σ 2 ⎞ −1 xˆ MAP = ⎜⎜ n2 A (C 0 ) + I ⎟⎟ y = ( kA (C 0 ) + I ) y σ ⎝ x ⎠
(32)
To apply this estimation of observed image, the matrix C0 and the variance ratio k should be estimated with the observed data.
5
4
3
4
5
4
2
1
2
4
3
1
X
1
3
4
2
1
2
4
5
4
3
4
5
(a)
(b)
Fig. 15. The neighborhood system and its order (a) Ordered neighborhood system for MRF. The order of the model relative to × is indicated by numbers; (b) The second order neighborhood system 3.2 2D Multivariate time series analysis The remote sensing image data from SAR (Synthetic Aperture Radar), TM (Thermal Mapper ) or other sensors can be described by time series image models. To analyze and segment the images, we need to consider the correlation between neighborhood image pixels rather than independent random variables as considered by many pattern recognition researchers. The SAR and TM remote sensing data are in multispectral format which consists of a stack of images. There are correlations between pixels in a single image as well as correlations among image slices. The univariate Autoregressive (AR) model is not enough to describe information extracted from multispectral satellite sensors. The 2D Multivariate Vector AR time series model described in this section is aimed to solve this problem. The multivariate time series data analysis model is a generalized form of univariate models. It is also called ARV (AutoRegressive Vector) model. Each time series observation is a vector containing multiple components.
Let Xi = [ x1i , x2 i , x3 i ,.....xmi ]T
−∞ ≤ i ≤ ∞
Xi = W + φ1Xi − 1 + φ2 Xi − 2 + ..... + φp Xi − p + ε i
(33) (34)
174 Xi :
Image Segmentation
mby1 column vector, time series observation variable
ε i : mby1 column vector, multivariate white noise φk : k = 1, 2, 3,.....p mbym autoregressive parameter matrix W : mby1 Constant Vector ( deterministic DC term ) To estimate the Multivariate AR model Parameter Matrix, we need to adopt the least squares method. Consider an mdimensional time series and let the Parameter Matrix be
B = ⎡⎣ W φ1 φ2 .... φp ⎤⎦
(35)
⎡1 ⎤ ⎢X ⎥ ⎢ v −1 ⎥ Define U v = ⎢ X v − 2 ⎥ ⎢ . ⎥ ⎢ . ⎥ ⎢ ⎥ ⎢Xv− p ⎥ ⎣ ⎦
(36)
An AR(p) time series model can be expressed as the following regression model: X v = BU v + ε v
(37)
where ε v = noise vector with covariance matrix C v = 1, 2,.....n and n is the total number of samples. Let n
T = ∑ U vU Tv
(38)
v =1
n
X = ∑ X v X Tv
(39)
v =1 n
S = ∑ X vU Tv
(40)
C = E(ε vε vT )
(41)
v =1
The Parameter Matrix B = ⎡⎣ W φ1 φ2 .... φp ⎤⎦ can be estimated as Bˆ = ST −1
The error covariance matrix can be calculated as:
(42)
Image Segmentation by Autoregressive Time Series Model
Cˆ =
1 ( X − ST −1ST ) n − nf
175
(43)
The matrix Cˆ can be factorized by Cholesky decomposition. The diagonal terms of this factorized Cholesky matrix are used as feature vector inputs to the pattern classifiers such as Support Vector Machine (SVM) or others. Cˆ = ET E
(44)
where E is the upper triangular matrix. Suppose E is an mbym upper triangular, lower triangular or diagonal matrix , the eigenvalues of E are the entries on the main diagonal of E . The eigenvalues of E matrix are the characteristics that E contains. The E matrix contains the major information of the estimated error covariance matrix Cˆ . The optimal estimated error covariance matrix Cˆ can be adopted to our remote sensing image classifier. When this optimal parameter matrix Bˆ is applied to the class that the data belongs, the total estimation errors are small, whereas when it is applied to any other class, the total estimation error would be much larger. Therefore, the differences in total estimation errors can be used in the classification process to distinguish among different remote sensing image classes.
4. Time series model segmentation adopted by international researchers In this section, we like to present some evidences from international academic researchers to prove the effectiveness and efficiency out of time series models. Research scholars P. Steetal and N. Natarajan of IITM, Chennai36 (Seetal 2010) was looking at rock fracture mapping problems that has applied to many issues related to rock mechanics. The difficult task was on fracture extraction from rock images. They found one interesting time series algorithm mathematically derived by P. Ho (Ho 2004). By using the first order autoregressive time series image segmentation model, they were able to apply and extract features on both rough and smooth fractures successfully as shown in figure 16 and figure 17. In Steetal and Natarajan computer experiments, three examples of rock fracture image segmentation results were processed by traditional edge detection methods such as Canny, Sobel and Prewitt as well as the newer time series models as described in this book chapter. The Seetal’s proposed method is a minor modified version from Ho and Chen (Ho 2004). For the detailed image segmentation result comparisons, please refer to International Journal of Engineering Science and Technology Vol. 2(5), 2010 (Seetal 2010).
5. Conclusion Image segmentation is one of the difficult tasks in image processing problems. The accuracy of the segmentation will determine a success or failure of the analysis. Therefore, decision making in the segmentation algorithm is very important. In this chapter we have established the innovative autoregressive time series model theory and presented many experimental results to support its image segmentation use. This image model indeed the effectively statistical approach to extract different properties for different objects.
176
Image Segmentation
Fig. 16. Two sets of Rock fracture mapping image segmentation by AR time series model, original rock fractures are on the left column, processing results are on the right column ((Seetal 2010)
Fig. 17. Cameraman image segmentation results by AR time series methods, original Image is on the left, AR processing result is on the right (Seetal 2010)
6. References Box, G. and Jenkins, G. Time Series Analysis: Forecast and Control, San Francisco, HoldenDay, 1970. Bruzzone, L. and Prieto, D. A Technique for The Selection Of Kernel Function Parameters in RBF neural Networks for Classification of Remote Sensing Images, IEEE Trans. Geosci. Remote Sensing, Vol. 37, No.2, pp. 11791184, March 1999.
Image Segmentation by Autoregressive Time Series Model
177
Bruzzone, L. Advanced Methods for the Analysis of Multisensor and Multitemporal RemoteSensing Images, PhD dissertation 1998, Electronics and Computer Science, Univeristy of Genova, Italy. Chapra, S. Canale, R. Numerical Methods for Engineers with Software and Programming Applications, 4th edition, McGraw Hill 2002. Chen, C. editor, Information Processing for Remote Sensing, World Scientific 1999, pp. 1213 Chen, C. and Ho, P. On the ARMA model based Region Growing method for extracting lake region in a Remote Sensing image, on Proceedings of SPIE, Vol. 5238, Paper No. 523813, September 812, 2003, Barcelona, Spain. Chen, H. Remote Sensing Data Unsupervised Classification Using GaussMarkov Random Fields and PCA, MS thesis 2002 ECE University of Massachusetts at Dartmouth. Cheney W. and Kincaid, D. Numerical Mathematics and Computing, 4th edition, Publisher: Gary W. Ostedt Book 1999. Geman S., Geman D., Stochastic relaxation, Gibbs distribution, and the Bayesian restoration of images, IEEE Trans. Pattern Anal. Machine Intell., 6 (6) (1984) 721 741. Glantz S. Primer of Biostatistics, 5th edition, McGrawHill 2002. Gonzalez Rafael C., Woods Richard E. and Eddins Steven L. Digital Image Processing using Matlab, Pearson Prentice Hall, 2004. Gröger J. P., R. A. Rountree, U. H. Thygesen, D. Jones, D. Martins, Q. Xu and B. Rothschild. Geolocation of Atlantic cod, Gadus morhua, movements in the Gulf of Maine using tidal information. Fisheries Oceanography, 2007. Haykin Simon and Widrow Bernard, LeastMeanSquare Adaptive Filters, Wiley and Sons, 2003. Haralick R.M. and Shapiro L.G. Computer and Robot Vision Vol.1 1992. Ho, P. Multivariate time series model based support vector machine for multiclass remote sensing image classification and region segmentation, Ph.D dissertation, Univ. of Massachusetts Dartmouth, Jan. 2008. Ho, PeiGee Peter and Chen C. H., Time Series Model Based Region Growing Method for Image Segmentation in Remote Sensing Images, IEEE 2004 International Geoscience and Remote Sensing Symposium, Alaska. Ho, PeiGee Peter, Chen C.H., 2D Time Series Model Based Support Vector Machine for Remote Sensing Image Segmentation. Proceedings of International Joint Conference on Neural Networks (IJCNN) paper No.1095, Aug. 1217, 2007. Jeng F. C., Woods J. W., Compound GaussMarkov random fields for image estimation, IEEE Trans. on Signal Processing, 39 (.3) (1991) 683697. Moler, C. Numerical Computating with Matlab, Chapter 10, SIAM 2004 . Papoulis A. and Pillai S.U., Probability, Random Variables and Stochastic Processes, 4th edition, McGrawHill 2002. Prieto, D. Automatic Analysis of Multisource and Multitemporal RemoteSensing Data for LandCover Monitoring Application, Ph.D Thesis, University of Genova, 2000. Serpico S. and Roli, F. Classification of Multisensor Remote Sensing Images by Structured Neural Netorks, IEEE Transactions on Geoscience and Remote Sensing, Vol 33, No.3, pp. 562578, May 1995.
178
Image Segmentation
Seetal, P. and Natarajan, N. Image Segmentation for rock fractures based on ARMA Model, International Journal of Engineering Science and Technology Vol. 2(5), 2010, pp. 11551159. Seber, G., Linear Regression Analysis, PP. 330334, John Wiley & Sons, 1977. Shumway Robert H., Stoffer David S., Time Series Analysis and Its Application, SpringerVerlag, 2000. Tilton J.C. and Lawrence W.T., Interactive Analysis of hierarchical image segmentation, Proceesings of the IEEE IGARSS 2000. Wei William W. S., Time Series Analysis, Univariate and Multivariate Method, AddisonWesley, 1990.
10 Evolutionarybased Image Segmentation Methods Licheng Jiao
Xidian University, Xi'an, China 1. Introduction Nearly 72 years ago, Wertheimer [1] pointed out the importance of perceptual grouping and organization in vision and listed several key factors, such as similarity, proximity, and good continuation, which lead to visual grouping. However, even to this day, many of the computational issues of perceptual grouping have remained unresolved. Since there are many possible partitions of an image into subsets, how do you know which one is right? There are two aspects to be considered here. The first is that there may not be a single correct answer. The second aspect is that the partitioning is inherently hierarchical. Prior literature on the related problems of clustering, grouping and image segmentation is huge. Unfortunately, there is not a general method existing to solve the problem.[2] Image segmentation is one of the central problems in computer vision and pattern recognition. It refers to the process of assigning a label to every pixel in an image such that pixels with the same label share certain visual characteristics. The result of image segmentation is a set of segments (sets of pixels) that collectively cover the entire image. Pixels in the same region are similar with respect to some characteristics or computed properties, such as color, intensity, and texture. Adjacent regions are significantly different with respect to the same characteristics. The goal of segmentation is to simplify and/or change the representation of an image into something that is more meaningful and easier to analyze.[3] There are many generalpurpose approaches available for image segmentation such as threshold methods[4], edgebased methods[5], regionbased methods[6], and graphbased methods[7]. Threshold techniques make decisions based on local pixel information. Edgebased methods are based on connecting together broken contour lines. It is prone to failure in the presence of blurring. A regionbased method usually partitions an image into connected regions by grouping neighboring pixels of similar intensity levels. Adjacent regions are then merged under some characteristics. Graphbased techniques generally represent the problem in terms of a graph where each node corresponds to a pixel in the image, and an edge connects each pair of vertices. A weight is associated with each edge based on some property of the pixels that it connects, such as their image intensities. Hybrid techniques using a mix of the methods above are also popular. What listed above also exposed two basic questions: • What is the precise criterion for a good segmentation? • How can such a segmentation be computed efficiently?
180
Image Segmentation
In contrast to the heuristic nature of the methods above, one would formalize an objective criterion for evaluating a given segmentation. This would allow us to formulate the segmentation problem as an optimization problem. The objective function that one would seek to optimize is the interclass variance that is used in cluster analysis. An optimizer can lead to efficient solutions for optimal segmentation. But the objective function is usually not a monotone chain, therefore the problem is general NPhard. Following this way, some clustering methods have been applied to solve image segmentation problems. Among them, Kmeans algorithm [8] is the most popular and simplest one. It can partition an image into K clusters by using an iterative technique. Although it can be proven that the procedure will always terminate, the Kmeans algorithm does not necessarily find the most optimal configuration, corresponding to the minimum global objective function. The algorithm is also significantly sensitive to the initial randomlyselected cluster centers. As global optimization techniques, evolutionary algorithms (EAs) are likely to be good tools for image segmentation task. In the past two decades, EAs have been applied to image segmentation with promising results [916]. These algorithms exploited the metaphor of natural evolution in the context of image segmentation. In this chapter, in order to solve the image segmentation problem more efficiently, we propose two evolutionarybsed image segmentation algorithms with different objective functions.The first is a novel approach based on memetic algorithm (MISA). Watershed segmentation is applied to segment original images into nonoverlap small regions before performing the portioning process by MISA. MISA adopts a straightforward representation method to find the optimal combination of watershed regions under the criteria of interclass variance in feature space. After implementing clusterbased crossover and mutation, an individual learning procedure moves exocentric regions in current cluster to the one they should belong to according to the distance between these regions and cluster centers in feature space. In order to evaluate the new algorithm, six texture images, three remote sensing images and three natural images are employed in experiments. The experimental results show that MISA outperforms its genetic version, the Fuzzy cmeans algorithm, and Kmeans algorithm in partitioning most of the test problems, and is an effective approach when compared with two stateoftheart image segmentation algorithms including an efficient graphbased algorithm and a spectral clustering ensemblebased algorithm. The second is manifold evolutionary clustering (MEC). In MEC, the clustering problem is considered from a combinatorial optimization viewpoint. Each individual is a sequence of real integer numbers representing the cluster representatives. Each data item is assigned to a cluster representative according to a novel manifold distancebased dissimilarity measure which can measure the geodesic distance along the manifold. After extracting texture features from an image, MEC determines partitioning of the feature vectors using evolutionary search. We apply MEC to solve seven benchmark clustering problems of artificial data sets, three artificial texture image classification problems and two Synthetic Aperture Radar image classification problems. The experimental results show that in terms of cluster quality and robustness, MEC outperforms the KMeans algorithm, a modified KMeans algorithm using the manifold distancebased dissimilarity measure, and a genetic algorithmbased clustering technique in partitioning most of the test problems. In the following sections, we will give the descriptions for the two algorithms in detail. Finally, concluding remarks are presented.
Evolutionarybased Image Segmentation Methods
181
2. Natural and remote sensing image segmentation using memetic computing 2.1 Introduction This work focuses on image segmentation based on memetic computation. Memetic algorithms (MAs) are one of the recent growing areas in evolutionary computation [17]. They are now widely considered as populationbased metaheuristic search approaches which may be regarded as a marriage between an evolutionary or any populationbased global search and local improvement procedures. Recently, MAs have been well used across a wide range of problem domains. A lot of studies have demonstrated that MAs converge to highquality solutions more efficiently than their conventional counterparts in many realworld applications [1732]. MAs have also been applied in the image processing field [3338]. For example, Fernandez, Garana and Cabello [33] proposed a MAbased method for the correction of illumination inhomogeneities in images. Batenburg [34] designed an EA with hillclimb operator for finding a binary image that satisfies prescribed horizontal and vertical projections. Tirronen, Neri et al. [35] studied the defect detection in paper production by means of imageprocessing techniques based on memetic differential evolution frameworks. Gesù, Bosco et al. [36,37] introduced a new memetic approach for the reconstruction of binary images. Zhang, Wang and Zhang [38] proposed a novel image watermarking scheme using a MA and wavelet transform. In this method, in order to solve the image segmentation problem more efficiently, we propose a MAbased approach, Memetic Image Segmentation Algorithm (MISA), and compare the new method with its genetic version (MISA without learning), the Kmeans algorithm [8], fuzzy cmeans algorithm [39], and two stateoftheart image segmentation algorithms including an efficient graphbased algorithm [7] and a spectral clustering ensemblebased algorithm [40] in segmenting artificial texture images, remote sensing images and natural images. The rest of this section is organized as follows: Section 1.2 describes some related background including the technologies used in preprocessing the original image and the brief background of MAs. Section 1.3 describes the proposed MAbased image segmentation algorithm. In Section 1.4, we summarize and evaluate the experimental results. 2.2 Related background Before performing the portioning process by the proposed MISA, some preprocessing should be done on original images, including feature extraction based on the graylevel cooccurrence matrix (GLCM) [41] and wavelet decomposition [42], and over segmenting object images into small regions by watershed segmentation algorithm [6]. In this section, we will introduce these techniques briefly. The MAs will also be introduced in this section. 2.2.1 GLCM based statistic features GLCM was frequently used in texture analysis and extraction for images [41,43,44]. Texture features are demonstrated by the statistics over the GLCM. Four usually used statistics are angular second moment (also called energy), correlation, entropy, and inverse difference moment (also called inertia). In this method, the 14 statistics, including the previous four suggested by Haralick, Shanmugam and Dinstein [41] are exploited. There are four parameters that must be indicated to generate a GLCM, i.e., the interpixel orientation, distance, grey level quantization, and window size. Here we set the interpixel orientation to 0° for convenient
182
Image Segmentation
calculation. Short interpixel distances typically generate the preferred texture features in image analysis, so we set interpixel distance to 1. The role of different values for gray levels and windows size with respect to statistics from GLCM has been investigated in many literatures [43, 44]. According to their analysis and finetuned experiments, in this study, we set the image quantization to 16 and the window size to 9 × 9 . 2.2.2 Wavelet energy features Wavelet transform has the ability to examine a signal at different scales [42]. In this section, the undecimated waveletbased feature vector composed by the energies of the subband coefficients is used. Here we implement threelevel wavelet decomposition on each square local area with size of 16 × 16 . The features of each pixel can be represented as a 10dimension vector ( eLL − 1 , eLH − 1 , eHL − 1 , eHH − 1 , eLH − 2 , eHL − 2 , eHH − 2 , eLH − 3 , eHL − 3 , eHH − 3 ) , in which, for example eLL − 1 denotes the energy of the LL subimage in the first level. Wavelet transform generates localized spatial and spectral information simultaneously. The energy of the lowpass subimage eLL − 1 describes the spectral information. The other features except eLL − 1 characterize the textural properties. Incorporating all the features will enhance the performance. 2.2.3 Watershed segmentation We use watershed segmentation [6] to over segment object image into small regions. Each region is almost homogeneous in feature space, while neighboringregions possess different characters. We operate these regions as a basic unit during most of MISA unless it is necessary to come back to pixels for precise evaluation. The basic concept of watershed segmentation is described as follows. In an image, ideal step edges do not often exist since every edge is blurred to some contents. A blurred edge can be modeled by a ramp. For a ramp edge, a usual gradient operator will generate a slope of the edge. Thus, the ramp edge cannot be separated from noise if the slope of the edge is small. Wang proposed a multiscale gradient operator to solve the above problem [45]: MG( f ) =
1 n ∑ ⎡( ( f ⊕ Bi ) − ( f ΘBi ) ΘBi −1 )⎤⎦ n i =1 ⎣
(1)
where ⊕ and Θ denote dilation and erosion respectively, and Bi is called structural element of size (2i −1)×(2i −1), and f is the original image. If watershed regions are too large, big ones may contain more than one focused subject in image, so that the texture feature in that region may not be homogeneous. If the watershed regions are too small, the computational complexity will increase. This is because there will be more basic units to operate during most of MISA. In order to control the number of watershed regions, we use the watershed segmentation algorithm with markers [6]. In this method, a threshold can be adjusted to get an expected number of regions. In this method, the number of regions is about 1500 in a 256 × 256 image. 2.2.4 Memetic algorithms Evolutionary algorithms perform well for global searching because they are capable of quickly finding and exploiting promising regions of search space, but they take a relatively
Evolutionarybased Image Segmentation Methods
183
long time to converge to a local optimum. Local improvement procedures quickly find the local optimum of a small region of the search space, but are typically poor global searchers. Thus, several researchers proposed hybrid algorithms combining excellent global exploration characteristics of EAs and efficient refinement capabilities of local search algorithms [46,47]. These hybrid algorithms are known as memetic algorithms (MAs). MAs assume that combining the features of different methods in a complementary fashion may result in more robust and effective optimization tools. They are populationbased metaheuristic search approaches that have been receiving increasing attention in recent years. They are inspired by Darwinian’s principles of natural evolution and Dawkins’ notion of a meme defined as the basic unit of cultural transmission. In MAs, a meme is generally considered as an individual learning procedure capable of performing local refinements. Over the past 15 years, MAs have been a hot topic in the fields of both computer science and operational research [1738]. The typical issues pertinent to MA design [17] include (i) how often individual learning should be applied, (ii) on which solutions individual learning should be used, (iii) how long individual learning should be run, (iv) what maximum computational budget to allocate for individual learning, and (v) what individual learning method or meme should be used for a particular problem, subproblem or individual. 2.3 The proposed memetic algorithm for image segmentation 2.3.1 Preprocessing and representation Preprocessing is the first step of our method. In this procedure, we use graylevel cooccurrence matrix and wavelet decomposition for feature extraction of object images. This is done for every pixel in the object image. Some algorithms (such as [15]) operate pixels directly as basic units, which will lead to high computational cost, especially for large images. To reduce the computational complexity, we employ watershed segmentation to segment images into nonoverlap small regions. MISA operates these regions instead of every pixel as the basic unit during most of its evolution unless it is necessary to return back to pixels for more precise computation in fitness calculation, learning and crossover operations. Fig.1 illustrates the watershed segmentation process.
Fig. 1. Illustration of watershed segmentation In Fig.1, there is an example of an image of 10 × 10 pixels with each small square divided by the orange lines representing a pixel. We can see that this image contains 4 regions marked
184
Image Segmentation
by different colors as red, blue, yellow, and green. The watershed segmentation will segment the image into four regions divided by the black boldface lines. Then, these four regions are the basic unit that we operate during most of MISA. After preprocessing, we get the following data from the object image: (i) feature matrix, which stores the features of each pixel and is used in initialization with minimum spanning tree, learning and fitness calculation; (ii) coordinate of pixels, which are used in fitness calculation; (iii) pixel number of each region, which is used in learning; (iv) dissimilarity matrix, which is used in initialization with minimum spanning tree. Many ECbased clustering algorithms [4851] have used an indirect representation approach that borrows from the KMeans algorithm  the representation codes are for cluster center only, and each data item is subsequently assigned to a cluster representative according to an appointed dissimilarity measure. The most popular dissimilarity measure is the Euclidean distance. By using Euclidean distance as a measure of dissimilarity, these evolutionary clustering methods as well as the KMeans algorithm have good performance on the data set with compact supersphere distribution, but tend to fail on the data set organized in more complex and unknown shapes. This indirect representation approach can not overcome the main drawback of the Kmeans algorithm unless a more flexible dissimilarity measure is adopted [15]. Furthermore, this indirect representation approach is not suitable for individual evolutionary and learning operation. In this study, a straightforward encoding method is used here for image segmentation problem. In this coding, every locus represents the corresponding region in the image, and the gene on each locus is the cluster label of the corresponding region. In our method, coding length ( CL ) in MISA is the number of regions obtained from watershed segmentation of the object image. The serial labels of locus present the corresponding regions in image, that means the i th locus represents the i th regions. The segmentation of an image is expressed by encoding individual with cluster labels of regions. Cluster label of each region is put on its corresponding locus as the gene. Fig. 2 illustrates this code structure.
Fig. 2. Illustration of coding structure Here we exhibit the code of an image which owns 10 regions. The lower numbers denote the region labels. They are arranged in order from 1 to 10. The upper numbers are cluster labels corresponding to each region. In this individual, regions 1, 2, 4 and 10 belong to cluster 1, regions 3, 5 and 6 belong to cluster 2, and regions 7, 8 and 9 belong to cluster 3. This straightforward representation method is very convenient for evolutionary and learning operation as shown in Section 3.2. In order to guide the evolution to a meaningful direction related to optimal segmentation, Euclid distances in feature space between every pixel and the centroids of the cluster it belongs to are calculated. Then, we sum up all these distances and use the reciprocal of this summation as the fitness of MISA. This segmentation fitness is defined in Equation (2).
185
Evolutionarybased Image Segmentation Methods
fitness =
1 CN RN i PN j
∑∑ ∑ dis( pijk , mi )
(2)
i =1 j =1 k =1
where RN i is the number of regions in the ith cluster, PN j is the number of pixels in the jth region, pijk denotes the kth pixel in the jth region of the ith cluster, and mi is the centroid of all pixels of the ith cluster in feature space. Although we operate watershed regions of image during most of the evolution, it is necessary to return to every pixel, so that we can get a more precise segmentation evaluation. In Equation (2),
dis( pijk , mi ) =
∑( p FN
g =1
ijk g
− mi g
)
2
(3)
FN is the number of features and pijkg and mi g denote the gth feature of the current pixel and cluster centroid respectively. Therefore, the aim of MISA is to find an individual with maximum fitness. 2.3.2 Memetic image segmentation algorithm The main loop of MISA is as follows. Algorithm 1.1. Memetic Image Segmentation Algorithm (MISA) Step 1. Preprocessing. Feature extraction based on GLCM and wavelet decomposition, and over segmenting object image into nonoverlapping small regions by watershed segmentation. Step 2. Initialization. Giving the termination criterion, setting the initial parameters, generating a diverse population. Step 3. Crossover and Mutation. A clusterbased crossover operator and random mutation operator are implemented in this step. Step 4. Learning. A supervised local searching strategy is used to improve some individuals with probability. Step 5. Evaluation and Selection. Tournament selection with elitism strategy is used after fitness calculation. Step 6. Termination Test: If termination criterion is satisfied, export the individual with the highest fitness and return the corresponding image segmentation result, stop the algorithm; otherwise, go to Step 3. 2.3.2.1 Initialization
The following parameters have to be set in this step: population size (PS), the proportion of initial individuals generated using minimum spanning tree (MSTP), crossover probability (CP), mutation probability (MP), learning intensity (LI), and cluster number (CN). The termination criterion is to run until the number of generations reaches the maximum value (Gmax). The initial population is generated by two different methods to get a diverse population. Some initial individuals are generated based on minimum spanning tree (MST), and the others are generated randomly. The details are as follows: ⎡⎢ MSTP × PS ⎤⎥ individuals are generated based on minimum spanning tree (MST). The idea of MST is introduced to find a meaningful distribution of all regions. Dissimilarity matrix
186
Image Segmentation
from preprocessing is used to build a complete MST by Prim’s algorithm. In dissimilarity matrix, the number of the ith row and the jth column represents the dissimilarity degree of the ith region to the jth region. The smaller of the number is, the closer between the ith region and the jth region. Here the dissimilarity degree between the ith region ri and the jth region rj is calculated in Equation (4).
(
)
Dissimilarity ri , rj =
FN
∑ ( ri
k =1
k
− rik
)
2
(4)
Here FN is the number of features and rik and rik are the value of the kth feature of ri and rj . Based on MST, we break up links selected randomly to produce individuals. The whole MST represents one cluster which includes all the regions in the image; and removing n links leads to n more clusters. Breaking up different links will produce different distribution among clusters. Fig. 3 and 4 show this procedure with dissimilarity matrix given in Table 1. Region No. 1 2 3 4 5 6 7 8 9 10
1 Inf 0.7 4.67 0.69 4.73 4.72 4.63 2.56 2.44 2.46
2 0.7 Inf 4.42 0.58 4.53 4.52 4.33 2.28 2.25 2.21
3 4.67 4.42 Inf 4.63 0.53 0.67 6.27 4.79 5.29 4.99
4 0.69 0.58 4.63 Inf 4.75 4.77 4.58 2.5 2.45 2.31
5 4.73 4.53 0.53 4.75 Inf 0.29 6.17 4.76 5.25 4.98
6 4.72 4.52 0.67 4.77 0.29 Inf 6.19 4.78 5.26 5.03
7 4.63 4.33 6.27 4.58 6.17 6.19 Inf 2.16 2.22 2.41
8 2.56 2.28 4.79 2.5 4.76 4.78 2.16 Inf 0.64 0.52
9 2.44 2.25 5.29 2.45 5.25 5.26 2.22 0.64 Inf 0.73
10 2.46 2.21 4.99 2.31 4.98 5.03 2.41 0.52 0.73 Inf
Table 1. An example of dissimilarity matrix
Fig. 3. MST generated from dissimilarity matrix Fig. 4 Clusters generated from breaking up links
187
Evolutionarybased Image Segmentation Methods
Based on dissimilarity matrix shown in Table 1, we can get a Prim MST starting from region 3, which is illustrated in Fig. 3. If we break up the links between 2 and 3, 8 and 10, we can get a cluster distribution as {1, 2, 4,10} , {3, 5,6} and {7,8,9} , as shown in Fig. 4. The rest PS − ⎢⎡ MSTP × PS ⎥⎤ individuals are generated randomly, namely, for each locus and we put a random cluster label ranges from 1 to CN as its gene. 2.3.2.2 Crossover and Mutation
Based on the meaning of crossover that children should get part of the information from parents, we create a novel crossover operator, called clusterbased crossover (CC). In this algorithm, the role of crossover is to change the cluster distribution of one parent individual by the cluster distribution information of the other parent individual. To make the crossover meaningful for segmentation, we try to cross corresponding clusters of two parents, individuals A and B . For example, we divide an image into three clusters, where A is composed of clusters α 1 , α 2 , α 3 and B is composed of clusters β 1 , β 2 , β 3 . Here
α 1 = { ax ax ∈ cluster 1} ,
α 2 = { ay ay ∈ cluster 2} ,
α 3 = { az  az ∈ cluster 3} ,
β 1 = {bx bx ∈ cluster 1} , β2 = {by by ∈cluster 2} , β3 = {bzbz ∈cluster 3} , ( x ∈ (1, CL ),
y ∈ (1, CL ),
z ∈(1,CL), x ≠ y ≠ z ). First, mean values of every feature of each pixel in each cluster are calculated. Then, the modulus values of these mean values for each cluster are sorted from small to large. This calculation is similar to Equation (4) in section 3.2.1. For example, if the orders of clusters in A and B are α 2α 1α 3 and β 3 β 2 β 1 respectively, then, α 2 will be crossed with β 3 , α 1 with β 2 , and α 3 with β 1 . This idea is similar to multipoint crossover. The number of ‘points’ equals to the number of clusters. For convenient operation, we relabel α 2α 1α 3 and β 3 β 2 β 1 as α 1α 2α 3 and β 1 β 2 β 3 respectively, without removing regions of any cluster. Two individuals are chosen from the population randomly, and then the corresponding clusters between them are fixed and then crossed as mentioned above. The detailed steps are described as follows, where nc is the number of regions to be “crossed” in current cluster and α i j denotes the jth region in cluster α i and CN denotes the cluster number. Algorithm 1.2. Clusterbased Crossover (CC) Crossover: A cross B Step 1. Step 2. Step 3. Step 4.
Set i = 1 If i ≤ CN , go to Step 3, otherwise, stop Set nc = random number from 1 to the size of the i th cluster in A . Select nc regions randomly from the i th cluster of A , set j = 1 .
Step 5. If α i j does not exist in β i , then put i on the ai j th gene of B Step 6.
j = j + 1 , if j ≤ nc go to Step 5, otherwise, i = i + 1 , go to Step 2.
In CC, A crosses B and B crosses A , producing two children. Both of them inherit parts of the information from the two parents. We illustrate CC in Fig. 5.
188
Image Segmentation
Fig. 5. An illustration of clusterbased crossover Fig. 5 shows that in individual A, the cluster 1 to 3 are {1, 3,7, 5,10} , {2, 4} , and {6,8,9} , respectively. In individual B, the cluster distribution is {1, 4, 5,7} , {2, 3,6} and {8,9,10} . For example, we begin by selecting 3 regions in cluster 1 of A, which is {1, 3,7} with light red shadow. In individual B, region 1 and 7 are in cluster 1, but region 3 exists in cluster 2, so we move region 3 from cluster 2 to cluster 1. In the same way, after all the same procedures are applied on the rest clusters, we get the child individual illustrated in Fig.5(c). In the mutation phase, we change the cluster distribution randomly as follows, where A denotes an individual, A j is the jth gene of A. MP denotes mutation probability, CN
denotes the cluster number, and CL denotes coding length. Algorithm 1.3. Mutation Step 1. Set j = 1 Step 2. If j ≤ CL , set r = a random number range from 0 to 1, i = Aj , go to Step 3;
Otherwise, stop Step 3. If r < MP , set l = a random number range from 1 to CN ∧ l ≠ i . Step 4. A j = l , j = j + 1 , go to Step 2. In mutation, if a gene should be mutated due to probability, it will change into another cluster number generated randomly. This means the region corresponding to this locus will be moved into another cluster. 2.3.2.3 Individual Learning Method
Within the computational intelligence community, research on MA has since grown significantly and the term has come to be associated with the pairing of metaheuristics or populationbased methodologies with a separate lifetime learning process that materializes
189
Evolutionarybased Image Segmentation Methods
in various forms of individual learning methods or memes, which are generally regarded as lifetime learning procedures capable of generating refinement on given individual(s). [32] In MISA, we introduce an individual learning method based on cluster distribution as is described in Algorithm 1.4. The basic idea is firstly, identify the most distinct region in each cluster. Then, for each of these regions, we find the cluster with the most similar average feature values, and assign the region to this cluster. In Algorithm 1.4, AGi is the average feature values of the ith cluster and AGimi is the average feature values of the mi th region in the ith cluster where PS denotes population size, CN denotes the cluster number, LI denotes learning intensity, and CL denotes coding length. Algorithm 1.4. Individual Learning Step 1. Set a = 1 Step 2. If a ≤ PS , go to Step3, otherwise stop. Step 3. For the a th individual in population, r is a random number from 0 to 1, if r > 0.5 , go to Step4, otherwise, a = a + 1 , go to Step3. Step 4. Set c = 1 . Step 5. For 1 ≤ i ≤ CN , find mi =arg max AGi − AGimi
2
.
Step 6. For 1 ≤ i ≤ CN , move the mi th region to cluster j = arg min AG j − AGimi
2
. If i = j ,
that means the mi th region doesn’t need to be moved. Step 7. c = c + 1 . If c ≤ ⎣⎢LI × CL ⎦⎥ , go to Step 5, otherwise, a = a + 1 , go to Step 2.
For more precise evaluation, we return back to pixels to calculate AGi instead of using the average feature values of each region in cluster i directly, which is shown in Equation (5). As described above, AGi is a vector that comprises FN features. RN i PN j
AGi =
∑∑p j =1 k =1 RN i
ijk
(5)
∑ PN j j =1
where RN i is the number of regions in the ith cluster, PN j is the number of pixels in the jth region, pijk denotes the feature vector of the kth pixel in the jth region of the ith cluster, defined as Equation (6).
(
pijk = pijk1 , pijk2 ,..., pijkFN
)
(6)
Fig. 6 illustrates an example of the individual learning procedure. In Fig. 6, {1, 2, 3, 4, 5} , {6,7,8,9,10,11} and {12,13,14,15,16} demonstrate the three clusters before learning, we label them as cluster1, cluster2 and cluster3, respectively. After the calculation shown in Step 5 and Step 6 of Algorithm 1.4, region 4, 8 and 12 are shown to be the most distinct regions of each cluster. And cluster3, cluster1, and cluster2 are the ones with most similar features for region 4, 8 and 12 respectively. Therefore, the learning processing will remove region 4, 8 and 12 to cluster3, cluster1 and cluster2 respectively.
190
Image Segmentation
Fig. 6. Illustration of an example of the individual learning procedure From the description above, we can see that half of the whole population is selected randomly each time to learn. For each individual in learning, ⎣⎢LI × CL ⎦⎥ regions are detected. LI should be adjusted properly as if it is too small, the learning’s effect is not enough. However, it is also unnecessary to set the LI too big. This is because learning is a local searching strategy so when it comes to the local optimum, the following learning will not change any cluster, which makes it meaningless and a waste of the computation. 2.4 Experimental study 2.4.1 Experimental setup In order to validate the performance of MISA, we apply it to twelve image segmentation problems including six artificial texture images, three remote sensing images and three natural images. The results will be compared with the Kmeans algorithm (KM) [8], Fuzzy cmeans algorithm (FCM) [39], the genetic image segmentation algorithm (GISA, MISA without learning), and two stateoftheart image segmentation algorithms including an efficient graphbased image segmentation algorithm (EGSA) [7] and the spectral clustering ensemble algorithm (SCEA) [40]. Based on the parameter sensitivity analysis (as described in Section 1.4.5), for MISA, the parameters are set as follows: The maximum number of generations is 50, population size is 30, MST initialization proportion is 0.6, crossover probability is 0.8, mutation probability is 0.005, and learning intensity is 0.5. For GISA, the parameters are set as follows: The maximum number of generations is 200, population size is 30, MST initialization proportion is 0.6, crossover probability is 0.8, and mutation probability is 0.005. Under these parameter settings, the computational costs of MISA and GISA are about equivalent. For KM and FCM, the maximum iterative number is set to 500, and the stop threshold is 1010. The software of EGSA is downloaded directly from the author’s homepage (http://people.cs.uchicago.edu/~pff/segment/). SCEA was proposed by us in 2008. We
191
Evolutionarybased Image Segmentation Methods
will perform it under the tuned parameters. Here 30 component spectral clustering with Nyström method are combined, and 100 regions are sampled in each component spectral clustering algorithm. The scaling parameter for each component is randomly selected from the interval [3, 12].
Image1
Image2
Image3
Image4
Image5
Image6
Image7
Image8
Image9
Image10
Image11
Image12
Fig. 7. The original images used in this study The original images are shown in Fig. 7. Image1 to Image6 are artificial texture images with the size of 256 × 256 from the Brodatz texture images [52]. Their standard classification images are shown in Fig. 8. Image7 to Image9 are remote sensing images. Image7 is a Kuband SAR image of the Rio Grande River nearby Albuquerue, New Mexico, USA. Image8 is a Kuband image of the China Lake Airport, California, USA. Image9 is an optical remote sensing image of Shelter Island, San Diego. The sizes of these images are all 256 × 256 , too. Image10 to Image12 are natural images. The sizes of them are 256×256, 320×320, and 330×320, respectively. The watershed segmentation results of the twelve test images are shown in Fig. 9.
192
Image Segmentation
Image1
Image4
Image2
Image3
Image5
Image6
Fig. 8. The true partitioning of the artificial texture images 2.4.2 Results on artificial texture images Image1 and Image2 contain two textures, Image3 and Image4 contain three textures and Image5 and Image6 contain four textures. Gray level, GLCM and wavelet features are used. In these experiments, the true partitioning is known. Thus we can calculate the clustering correct ratio directly by contrasting a segmentation result to the corresponding true
Problem Image1 Image2 Image3 Image4 Image5 Image6
Correct Ratio (Standard Deviation) MISA GISA 0.9844 0.9837 (0.0069) (0.0021) 0.9770 0.9746 (0.0012) (0.0015) 0.9475 0.9442 (0.0018) (0.0026) 0.9539 0.9505 (0.0010) (0.0418) 0.9622 0.9588 (0.0008) (0.0014) 0.9493 0.9485 (0.0004) (0.0006)
EGSA 0.9705 (0) 0.9341 (0) 0.9517 (0) 0.9421 (0) 0.8326 (0) 0.8871 (0)
SCE 0.9842 (0.0005) 0.8630 (0.0158) 0.8983 (0.0950) 0.9614 (0.0014) 0.9114 (0.0783) 0.9319 (0.0408)
FCM 0.9841 (0) 0.9745 (0) 0.7863 (0.2938) 0.9502 (0) 0.7400 (0.2718) 0.8243 (0.2737)
KM 0.9835 (0) 0.9719 (0) 0.7807 (0.2883) 0.9505 (0.0001) 0.7711 (0.2608) 0.8243 (0.2642)
Table 2. Statistic results obtained from MISA, GISA, EGSA, SCEA, FCM and KM on the artificial texture images
193
Evolutionarybased Image Segmentation Methods
Image1
Image2
Image3
Image4
Image5
Image7
Image8
Image9
Image10
Image11
Image12
Image6
Fig. 9. The watershed segmentation results of Image 1 to Image 12.
194
Image Segmentation
partitioning. The average results of clustering correct ratios and standard deviations based on 30 independent runs are shown in Table 2. Fig. 10 shows the typical results of them obtained from the six algorithms. For each test image, the six segmentation images arranged from upper left to lower right are typical results of MISA, GISA, EGSA, SCEA, FCM and KM, respectively. Table 2 shows that MISA obtains higher correct ratio than GISA, FCM and KM do for all these six images, and gets the best correct ratio among all the six algorithms on Image1, Image2, Image5 and Image6. EGSA and SCEA get better correct ratios than MISA does on Image3 and Image4, respectively. EGSA is not a random algorithm, so all the standard deviations of its results are 0. MISA exceeds EGSA greatly on Image5 and Image6 and surpasses SCEA apparently on Image2 and Image3 in correct ratios. MISA exceeds KM and FCM greatly on Image3, Image5 and Image6 in both the correct ratios and standard deviation. Fig. 10 illustrates the above numerical comparisons visually.
MISA
SCEA
GISA
EGSA
FCM
KM
(a) Segmentation results of Image1
195
Evolutionarybased Image Segmentation Methods
MISA
SCEA
GISA
EGSA
FCM
KM
(b) Segmentation results of Image2
MISA
SCEA
GISA
FCM (c) Segmentation results of Image3
EGSA
KM
196
Image Segmentation
MISA
GISA
EGSA
SCEA
FCM
KM
(d) Segmentation results of Image4
MISA
SCEA
GISA
FCM (e) Segmentation results of Image5
EGSA
KM
197
Evolutionarybased Image Segmentation Methods
MISA
GISA
SCEA
FCM
EGSA
KM
(f) Segmentation results of Image6 Fig. 10. Typical results obtained from the compared algorithms on the artificial texture images 2.4.3 Results on remote sensing Images Fig. 11 shows the typical results on remote sensing images. For each test image, the six segmentation images arranged from upper left to lower right are typical results of MISA, GISA, EGSA, SCEA, FCM and KM respectively. Gray level and wavelet features are used. For these remote sensing images, there is no true partitioning for reference, thus numerical results could not be obtained here. For Image7, MISA makes the river bank clearer, and generates more homogeneous lower crops part than GISA does. But both MISA and GISA lose the crop line on the upper left corner. EGSA lose the crop part in the lower left part and the island in the river, but generates a clear bridge. SCEA and FCM might confuse the vegetations with the river. KM loses most of the bridge and other detailed information. For Image8, MISA gets very clear edges and integrates the building area, leaving some speckles in the center which are the trails of thin roads there. In the GISA results, some buildings are lost in the background. GISA produces some speckles because 200 generations are not enough for it to converge. EGSA and SCEA produce unacceptable results which mix up major cluster information. Both FCM and KM generate coarse edges of the main roads and lose detailed information of other narrow roads in the center and upper right. For Image9, MISA and GISA can distinguish details on the port area but generate some speckles on the land area. EGSA doesn’t confuse the land area on the left with water, but misses detailed information on the port area. SCEA, FCM and KM have worse results when compared with MISA on the port area and compared with EGSA on the land area.
198
Image Segmentation
MISA
GISA
EGSA
SCEA
FCM
KM
(a) Segmentation results of Image7
MISA
SCEA
GISA
FCM (b) Segmentation results of Image8
EGSA
KM
199
Evolutionarybased Image Segmentation Methods
MISA
SCEA
GISA
FCM
EGSA
KM
(c) Segmentation results of Image9 Fig. 11. Typical results obtained from the compared algorithms on the remote sensing images 2.4.4 Results on natural images Fig. 12 shows the typical results on natural images. For each image, the six segmentation images arranged from upper left to lower right are typical results of MISA, GISA, EGSA, SCEA, FCM and KM respectively. Gray level and wavelet features are used for experiments here. Image10 is segmented into three clusters; Image11 and Image12 are segmented into two clusters. Apparently, MISA produces the best results. GISA also gets better results than FCM and KM. EGSA, SCEA, FCM and KM lose a lot of details and even confuse very large parts in segmenting Image12. 2.4.5 Sensitivity in relation to parameters Fig.13 shows the convergence curves of MISA and GISA got from experiments on the artificial texture Image4, the remote sensing Image9, and the natural Image12. Each experiment is repeated 30 times for statistical evaluation, and the points in those figures are the average values. We can see that MISA converges within no more than 50 generations on all the experiments, while GISA can not converge within 100 generations.
200
Image Segmentation
MISA
GISA
EGSA
SCEA
FCM
KM
(a) Segmentation results of Image10
MISA
GISA
EGSA
SCEA
FCM
KM
(b) Segmentation results of Image11
201
Evolutionarybased Image Segmentation Methods
MISA
GISA
SCEA
FCM
EGSA
KM
(c) Segmentation results of Image12 Fig. 12. Typical results obtained from the compared algorithms on the natural images Fig. 14 shows the curves which show the influence of learning intensity. We make use of correct ratios of the six artificial texture images to exhibit the effect of learning intensity. The experiments are repeated 30 times, and we plot their statistical average values. Based on Fig. 14, we can see that the correct ratio increases as the learning intensity is reinforced before at about 0.2. After that，the curves fluctuate but do not increase apparently. This demonstrates that learning intensity should be adjusted properly as a small value is not enough for good convergency while a big one will lead to wasting of computation.
3. Image texture classification using a manifold distance based evolutionary clustering cethod 3.1 Introduction Image classification or segmentation based on texture features using unsupervised approaches has been a challenge topic. Texture is an important property of some images. A lot of texture feature extraction methods have been developed over the past three decades. These texture features can be categorized into four major categories [53, 54]: statistical, geometrical, modelbased, and signal processing. Among them, graylevel cooccurrence features, first proposed by Haralick, Shanmugam and Dinstein [55], are one of the most common features used in literature. In some images, the same object region may vary in appearance from image to image as well as within the same image. Thus, the selected training samples in a supervised algorithm may not be sufficient to include all the class variability throughout the image. Under these conditions, unsupervised classification, i.e. clustering, may be more effective. There are a variety of clustering approaches that could be
202
Fig. 13. The convergence curves of MISA and GISA
Image Segmentation
Evolutionarybased Image Segmentation Methods
203
Fig. 14. The influence of learning intensity versus correct ratio used to assign class labels to the feature vectors. These approaches can be categorized into two groups [56, 57]: hierarchical clustering and partitional clustering, where partitional clustering approaches, such as KMeans Algorithm [58], partition the data set into a specified number of clusters by minimizing certain criteria. Therefore, they can be treated as an optimization problem. As global optimization techniques, Evolutionary Algorithms (EAs) are likely to be a good choice for this task. EAs, including Genetic Algorithm (GA), Evolutionary Strategy (ES), Evolutionary Programming (EP), etc., have been used for clustering tasks commonly in literature [59~62]. A variety of EA representations for clustering solutions have been explored, such as the straightforward encoding with each gene coding for the cluster membership of the corresponding data item, and the locusbased adjacency representation [62]. Many researchers [59~61] have chosen to use a more indirect approach that borrows from the KMeans algorithm: the representation codes for cluster center only, and each data item is subsequently assigned to a cluster representative according to an appointed dissimilarity measure. The most popular dissimilarity measure is the Euclidean distance. By using Euclidean distance as a measure of dissimilarity, these evolutionary clustering methods as well as the KMeans algorithm have a good performance on the data set with compact supersphere distributions, but tends to fail in the data set organized in more complex and unknown shapes, which indicates that this dissimilarity measure is undesirable when clusters have random distributions. As a result, it is necessary to design a more flexible dissimilarity measure for clustering. Su and Chou [63] proposed a nonmetric measure based on the concept of point symmetry, according to which a symmetrybased version of the K
204
Image Segmentation
Means algorithm is given. This algorithm assigns data points to a cluster center if they present a symmetrical structure with respect to the cluster center. Therefore, it is suitable to clustering data sets with clear symmetrical structure. Charalampidis [64] recently developed a dissimilarity measure for directional patterns represented by rotationvariant vectors and further introduced a circular KMeans algorithm to cluster vectors containing directional information. In order to solve the texture classification task effectively, in this study, we design a novel evolutionary clustering method, named manifold evolutionary clustering (MEC). In MEC, we adopt an indirect encoding approach, namely, each individual is a sequence of real integer numbers representing the cluster representatives. Each data item is assigned to a cluster representative according to a novel dissimilarity measure which can measure the geodesic distance along the manifold. After extracting texture features from an image, MEC determines a partitioning of the feature vectors using evolutionary search. The effectiveness of MEC will be validated by comparing with the KMeans algorithm, a modified KMeans algorithm using the manifold distancebased dissimilarity measure [65], and the genetic algorithmbased clustering technique proposed by Maulik and Bandyopadhyay [60] in solving seven benchmark clustering problems of artificial data sets, three artificial texture image classification problems and two Synthetic Aperture Radar (SAR) images classification problems. The remainder of this section is organized as follows: Section 2.2 describes the novel manifold distancebased dissimilarity measure. Section 2.3 describes the evolutionary clustering algorithm based on the novel dissimilarity measure. In Section 2.4, we summary and evaluate the experimental results. 3.2 A novel manifold distancebased dissimilarity measure A meaningful measure of distance or proximity between pairs of data points plays an important role in partitional clustering approaches. Most of the clusters can be identified by their location or global characteristics. Through a large mount of observation, we have found the following two consistency characteristics of data clustering. a. Local consistency refers that data points close in location will have a high affinity. b. Global consistency refers that data points locating in the same manifold structure will have a high affinity. For realworld problems, the distribution of data points takes on a complex manifold structure, which results in the classical Euclidian distance metric can only reflect the local consistency, but fail to describe the global consistency. We can illustrate this problem by the following example. As shown in Fig. 15, we expect that the affinity between point a and point e is higher than the affinity between point a and point f. In other words, we are looking for a measure of dissimilarity according to which point a is closer to point e than to point f. In terms of Euclidian distance metric, however, point a is much closer to point f than to e, thus without reflecting the global consistency. Hence for complicated realworld problems, simply using Euclidean distance metric as a dissimilarity measure can not fully reflect the characteristics of data clustering. Here, we want to design a novel dissimilarity measure with the ability of reflecting both the local and global consistency. As an example, we can observe from the data distribution in Fig. 15 that data points in the same cluster tend to lie in the same manifold.
205
Evolutionarybased Image Segmentation Methods
Fig. 15. An illustration of that the Euclidian distance metric can not reflect the global consistency. At first, data points are taken as the nodes V of a weighted undirected graph G = (V , E) . Edges E = { Wij } reflect the affinity between each pair of data points. We expect to design a dissimilarity measure that assigns high affinity to two points if they can be linked by a path running along a manifold, and a low affinity if they cannot. This concept of dissimilarity measure has been shown in experiments to lead to significant improvement in classification accuracy when applied to semisupervised learning [66, 67]. The aim of using this kind of measure is to elongate the paths that cross different manifolds, and simultaneously shorten those that not cross. To formalize this intuitive notion of dissimilarity, we need first define a socalled manifold length of line segment. We have found a property that a distance measure describing the global consistency of clustering does not always satisfy the triangle inequality under the Euclidean metric. As shown in Fig. 15, to describe the global consistency, it is required that the length of the path connected by shorter edges is smaller than that of the direct connected path, i.e. ab + bc + cd + de < ae . In other words, a direct connected path between two points is not always the shortest one. Enlightened by this property, we define a manifold length of line segment as follows. Definition 1. The manifold length of line segment ( xi , x j ) is defined as L( xi , x j )
(
)
ρ
dist ( xi , x j )
−1
(7)
where dist xi , x j is the Euclidean distance between xi and x j ; ρ > 1 is the flexing factor. Obviously, the manifold length of line segment possesses the property mentioned above, thus can be utilized to describe the global consistency. In addition, the manifold length between two points can be elongated or shortened by adjusting the flexing factor ρ .
206
Image Segmentation
According to the manifold length of line segment, we define a new distance metric, called manifold distance metric, which measures the distance between a pair of points by searching for the shortest path in the graph. Definition 2. Let data points be the nodes of graph G = (V , E) , and p ∈ V l be a path of length l = p − 1 connecting the nodes p1 and p p , in which ( pk , pk + 1 ) ∈ E , 1 ≤ k < p . Let Pi , j
denote the set of all paths connecting data points xi and x j . The manifold distance between xi and x j is defined as p −1
D( xi , x j )
min p∈Pi , j
∑ L( pk , pk + 1 ) .
(8)
k =1
The manifold distance satisfies the four conditions for a distance metric, i.e. D( xi , x j ) = D( x j , xi ) ; D( xi , x j ) ≥ 0 ; D( xi , x j ) ≤ D( xi , xk ) + D( x k , x j ) for all xi , x j , xk ; and D( xi , x j ) = 0 if and only if xi = x j . As a result, the manifold distance metric can measure the geodesic distance along the manifold, which results in any two points in the same manifold being connected by a lot of shorter edges within the manifold while any two points in different manifolds are connected by a longer edge between manifolds, thus achieving the aim of elongating the distance among data points in different manifolds and simultaneously shortening the distance among data points in the same manifold. 3.3 Evolutionary clustering based on the manifold distance By using EAs to solving clustering tasks, it is necessary to design the individual representation method and the heuristic search operators, and formulate the objective function for optimization. 3.3.1 Representation and operators In this study, we consider the clustering problem from a combinatorial optimization viewpoint. Each individual is a sequence of real integer numbers representing the sequence number of K cluster representatives. The length of a chromosome is K words, where the first gene represents the first cluster, the second gene represents the second cluster, and so on. As an illustration, let us consider the following example. Example 1. Let the size of the data set is 100 and the number of clusters being considered is 5. Then the individual (6, 19, 91, 38, 64) represents that the 6th, 19th, 91st, 38th, and 64th points are chosen to represent the five clusters, respectively. This representation method does not mention the data dimension. If the size of the data set is N and the number of clusters is K, then the search space is NK. Crossover is a probabilistic process that exchanges information between two parent individuals for generating offspring. In this study, we use the uniform crossover [68] because it is unbiased with respect to the ordering of genes and can generate any combination of alleles from the two parents [62, 69]. An example of the operation of uniform crossover on the encoding employed is shown in example 2. Example 2. Let the two parent individuals are (6, 19, 91, 38, 64) and (3, 29, 17, 61, 6), randomly generate the mask (1, 0, 0, 1, 0), then the two offspring after crossover are (6, 29,
Evolutionarybased Image Segmentation Methods
207
17, 38, 64) and (3, 19, 91, 61, 64). In this case, the first offspring is not (6, 29, 17, 38, 6) because the 6 in bold is repeat, we keep it unchanged. Each individual undergoes mutation with probability pm as example 3. Example 3. Let the size of the data set is 100 and the number of clusters being considered is 5. Then the individual (6, 19, 91, 38, 64) can mutate to (6, 19+ ⎣⎢(10019) × random+1⎦⎥ , 91, 38, 64) or (6, 19 ⎢⎣(191) × random+1⎥⎦ , 91, 38, 64) equiprobably when the second gene is chosen to mutate, random denotes a uniformly distributed random number in the range [0,1). 3.3.2 Objective function Each data item is assigned to a cluster representative according to its manifold distance to the cluster representatives. As an illustration, let us consider the following example. Example 4. Let the 6th, 19th, 91st, 38th, and 64th points represent the five clusters, respectively. For the first point, we compute the manifold distance between it and the 6th, 19th, 91st, 38th, and 64th points, respectively. If the manifold distance between the first point and the 38th point is the minimum one, then the first point is assigned to the cluster represented by the 38th point. All the points are assigned in this way. Subsequently, the objective function is computed as follows:
Dev(C ) =
∑ ∑ D(i , μk )
(9)
C k ∈C i∈C k
where C is the set of all clusters, μ k is the representative of cluster C k , and D(i , μ k ) is the manifold distance between the ith data item of cluster C k and μ k . 3.3.3 Manifold evolutionary clustering algorithm In MEC, the processes of fitness computation, roulette wheel selection with elitism [70], crossover and mutation are executed for a maximum number of generations Gmax. The best individual in the last generation provides the solution to the clustering problem. The main loop of MEC is as follows.
Algorithm 2.1. Manifold Evolutionary Clustering (MEC) Begin 1. 2. 3.
t=0 randomly initialize population P(t) assign all points to clusters as the manifold distance and compute the objective function values of P(t) 4. if t< Gmax 5. t=t+1 select P(t) from P(t1) using roulette wheel selection with elitism 6. 7. crossover P(t) mutate P(t) 8. 9. go to step 3 10. end if 11. output the best and stop
End
208
Image Segmentation
The initial population in step 2 is initialized to K randomly generated real integer number in [1, N], where N is the size of the data set. This process is repeated for each of the P chromosomes in the population, where P is the size of the population. 3.4 Experimental study 3.4.1 Experimental setup In order to validate the performance of MEC, we first apply MEC to seven benchmark clustering problems of artificial data sets. The results will be compared with the KMeans algorithm (KM) [58], a modified KMeans algorithm using the manifold distancebased dissimilarity measure (DSKM) [65], and the genetic algorithmbased clustering technique (GAC) proposed by Maulik and Bandyopadhyay [60]. In all the algorithms, the desired number of clusters is set to be known in advance. In the second experiment, we will solve three artificial texture image classification problems using MEC, GAC, DSKM and KM, respectively. In the third experiment, we will solve the classification problems of one Xband SAR image and one Kuband SAR image by using MEC, GAC, DSKM and KM, respectively. In the image classification experiments (the second and third experiments), we will use the graylevel cooccurrence matrix (GLCM) [55] method to extract texture features from images. There are many statistics that can be determined from each GLCM, such as angular second moment, contrast, correlation, sum of squares, entropy, and so on. Following [54], in this study, we chose three statistics, dissimilarity, entropy and correlation which indicate the degree of smoothness of the texture, the homogeneity and the correlation between the gray level pair, respectively. There are four parameters that must be indicated in order to generate cooccurrence data, namely, interpixel orientation, interpixel distance, the number
of gray levels and window size. Typically, interpixel orientation is set to 0o , 45o , 90o , 135o since this is easiest to implement. Short interpixel distances have typically achieved the best success, so interpixel distance is 1 will be used. This combination of offset and orientation has characterized SAR texture well [54]. The role of varying the values of the number of gray levels and windows size with respect to GLCM statistics has been presented in many references [54, 71]. After their analysis and finetune experiments, in this study, we set the number of gray levels is 16 and the window size is 13 × 13 . The parameter settings used for MEC and GAC in our experimental study are given in Table 3. For DSKM and KM, the maximum iterative number is set to 500, and the stop threshold is 1010. Parameter Maximum Number of generations population size Crossover probability Mutation probability
MEC
GAC
100
100
50 0.8 0.1
50 0.8 0.1
Table 3. Parameter settings for MEC and GAC In the first two experiments, the true partitioning is known, we will evaluate the performance using two external measures, the Adjusted Rand Index [62, 72, 73] and the Clustering Error [65].
209
Evolutionarybased Image Segmentation Methods
The Adjusted Rand Index [72] is a generalization of the Rand Index [74] which takes two partitioning as the input and count the number of pairwise coassignments of data items between the two partitioning. Given a set of N points S = {p1 , p2 , , pN } , suppose U = {u1 , u2 , S such that
, uK } and V = {v1 , v2 , K
K
i =1
j =1
∪ ui = ∪ v j = S
, vK } represent two different partitions of the points in
and ui ∩ ui ' = v j ∩ v j ' = ∅
for 1 ≤ i ≠ i ' ≤ K , 1 ≤ j ≠ j ' ≤ K .
Suppose that U is the known true partition, and V is a clustering result. Let a be the number of pairs of points in the same class in U and in the same class in V, b be the number of pairs of points in the same class in U but not in the same class in V, c be the number of pairs of points in the same class in V but not in the same class in U, and d be the number of pairs of points in different classes in both partitions. The quantities a and d can be interpreted as a+d . The Rand agreements, and b and c as disagreements. Then the Rand Index is a+b+c+d Index lies between 0 and 1, when the two partitions agree perfectly, the Rand Index is 1. A problem with the Rand Index is that the expected value of the Rand index of two random partitions does not take a constant value (say zero). The Adjusted Rand Index proposed by Hubert and Arabie [72] assumes the generalized hypergeometric distribution as the model of randomness, i.e. the U and V partitions are picked at random such that the numbers of points in the classes are fixed. Let nij be the number of points that are in both class ui and class v j . Let ni i and ni j be the number of points in class ui and class v j respectively. Under the generalized hypergeometric model, it can be shown that: n n E ⎡⎢ ∑ i , j ( 2ij )⎤⎥ = ⎡⎢ ∑ i ( n2i• )i∑ j ( 2• j )⎤⎥ /( n2 ) ⎣ ⎦ ⎣ ⎦
(10)
Then the Adjusted Rand Index is given as nij
∑ i , j ( 2 ) − ⎡⎣⎢∑ i (n2 )i∑ j ( 2 )⎤⎦⎥ /(n2 ) R(U ,V ) = n n 1[ ( n ) + ∑ j ( 2 )] − [∑ i (n2 )i∑ j ( 2 )]/(n2 ) 2 ∑i 2 i•
n• j
i•
•j
i•
•j
(11)
The Adjusted Rand Index return values in the interval [0, 1] and is to be maximized. Let the known true partition be U = {u1 , u2 , , uK } and the clustering result be
V = {v1 , v2 ,
, vK } . ∀i , j ∈ {1, 2,
, K} , Confusion(i , j ) denotes the number of same data points
both in the true cluster ui and in the cluster v j produced. Then, the Clustering Error is defined as
CE(U ,V ) =
1 K K ∑∑ Confusion(i , j ) , N i =1 j =1
(12)
i≠ j
where N is the size of data set. Note that there exists a renumbering problem, so the Clustering Error is computed for all possible renumbering of V, and the minimum one is taken. The Clustering Error also returns values in the interval [0, 1] and is to be minimized.
210
Image Segmentation
3.4.2 Implementation results on benchmark clustering problems We first select seven artificial data sets, named Lineblobs, Long1, Size5, Spiral, Square4, Sticks, and Threecircles to study a range of different interesting data properties. The distribution of data points in these data sets can be seen in Fig. 16. We perform 30 independent runs on each problem. The average results of the two metrics, Clustering Error and Adjusted Rand Index, are shown in Table 4.
Problem lineblobs Long1 Size5 Spiral Square4 Sticks threecircles
Clustering Error MEC 0 0 0.010 0 0.065 0 0
GAC 0.263 0.445 0.023 0.406 0.062 0.277 0.569
DSKM 0.132 0 0.015 0 0.073 0 0.055
Adjusted Rand Index KM 0.256 0.486 0.024 0.408 0.073 0.279 0.545
MEC 1 1 0.970 1 0.835 1 1
GAC 0.399 0.011 0.924 0.034 0.937 0.440 0.033
DSKM 0.866 1 0.955 1 0.816 1 0.921
KM 0.409 0.012 0.920 0.033 0.816 0.504 0.044
Table 4. Results of MEC, GAC, DSKM and KM on artificial data sets where the results in bold are the best ones From Table 4, we can see clearly that MEC did best on six out of the seven problems, while GAC did best only on the Square4 data set. DSKM also obtained the true clustering on three problems. KM and GAC only obtained desired clustering for the two spheroid data sets, i.e. Size5 and Square4. This is due to that the structure of the other five data sets does not satisfy convex distribution. On the other hand, MEC and DSKM can successfully recognize these complex clusters, which indicate the manifold distance metric are very suitable to measure complicated clustering structure. When comparisons are made between MEC and DSKM, MEC obtained the true clustering on the Long1, Spiral, Sticks, Lineblobs and Threecircles in all the 30 runs, but DSKM can not do it on the Lineblobs and Threecircles. Further more, for the Size5 and Square4 problems, MEC did a little better than DSKM in both the Clustering Error and the Adjusted Rand Index. The main drawback of DSKM is that it has to recalculate the geometrical center of each cluster as the KMeans algorithm after cluster assignment which reducing the ability of reflecting the global consistency. MEC made up this drawback by evolutionary searching the cluster representatives from a combinatorial optimization viewpoint. In order to show the performance visually, the typical simulation results on the eight data sets obtained from MEC are shown in Fig. 16. 3.4.3 Implementation results on artificial texture image classification Image1 is a simple 256 × 256 bipartite image (Fig. 17(a)). The original image contains two textures, cork and cotton, selected from the Brodatz texture images [75]. Fig. 17(b) represents the true partitioning of Image1. Image2 also contains two textures as shown in Fig. 17(c), and Fig. 17(d) represents its true partitioning. Image3 is a more complicated texture synthesized image with 4 classes, and Fig. 17(e) and (f) represent the original image and the true partitioning, respectively. We perform 30 independent runs on each problem. The average results of the two metrics, Clustering Error and Adjusted Rand Index, are shown in Table 5. Fig. 18 to Fig. 20 are the typical implementation results obtained from the four algorithms, MEC, GAC, DSKM and KM, in clustering the three texture images, respectively.
Evolutionarybased Image Segmentation Methods
211
Fig. 16. The typical implementation results on the artificial data sets obtained from MEC. (a) Line blobs; (b) Long1; (c) Size5; (d) Spiral; (e) Square4; (f) Sticks; (G) Threecircles.
212
Image Segmentation
Fig. 17. Artificial texture images and their true partitioning. (a) Original Image1; (b) True partitioning of Image1; (c) Original Image2; (d) True partitioning of Image2; (e) Original Image3; (f) True partitioning of Image3.
213
Evolutionarybased Image Segmentation Methods
Problem Image1 Image2 Image3
Clustering Error MEC 0.0030 0.0037 0.1212
GAC 0.0069 0.1594 0.2554
DSKM 0.0035 0.0072 0.1858
Adjusted Rand Index KM 0.0071 0.2017 0.2899
MEC 0.9462 0.9376 0.8638
GAC 0.9115 0.9057 0.8012
DSKM 0.9437 0.9109 0.8117
KM 0.9113 0.8869 0.8094
Table 5. Results of MEC, GAC, DSKM and KM on artificial texture image classification where the results in bold are the best ones
Fig. 18. The typical implementation results obtained from (a) MEC, (b) GAC, (c) DSKM and (d) KM in clustering Image1.
214
Image Segmentation
As shown in Table 5, all the average values of Cluster Error obtained from MEC, GAC, DSKM and KM in clustering Image1 are less than 1%, so all the four algorithms are easily able to segment the Image1. The values of Cluster Error and Adjusted Rand Index and Fig. 18 also show that the results obtained from MEC and DSKM are much better than the results of GAC and KM because both MEC and DSKM assign data items according to the manifold distance while GAC and KM assign data items according to Euclidian distance. However, the computational cost of the manifold distance is much larger than that of Euclidian distance. MEC and DSKM have similar results in clustering Image1.
Fig. 19. The typical implementation results obtained from (a) MEC, (b) GAC, (c) DSKM and (d) KM in clustering Image2.
Evolutionarybased Image Segmentation Methods
215
Fig. 20. The typical implementation results obtained from (a) MEC, (b) GAC, (c) DSKM and (d) KM in clustering Image3. In clustering Image2, the average value of Cluster Error obtained from MEC is much smaller than the results obtained from GAC, DSKM and KM, and the average value of Adjusted Rand Index of MEC is obviously greater than the results obtained from GAC, DSKM and KM. So MEC does best in this problem. Fig. 19 also shows that the MEC result and DSKM result are obviously better than the GAC result and KM result, and the MEC result is better than the DSKM result. MEC segment the two textures better than DSKM may be due to MEC search the two cluster representatives using evolutionary searching but DSKM has to recalculate the geometrical center of each cluster after cluster assignment in each iteration which reduces the ability of reflecting the global consistency.
216
Image Segmentation
In clustering the more complicated texture image Image3, all the average values of Cluster Error are greater than 12%, so none of the four algorithms can segment the image very well based on GLCM features. However, Table 5 and Fig. 20 show that MEC does much better than the other three algorithms. 3.4.4 Implementation results on remote sensing image classification The first image, as shown in Fig. 21(a), is an Xband SAR image of a lakeside in Switzerland. The size of the image is 140 × 155 pixels. We want to classify the image into three clusters, namely, the lake, the city, and the mountainous region. The second image, as shown in Fig. 21(b), is a Kuband SAR image of the Rio Grande River nearby Albuquerque, New Mexico, USA. The size of the image is 256 × 256 pixels. We want to classify the image into three clusters, namely, the river, the vegetation, and the crop. Fig. 22 and Fig. 23 shows the clustering results obtained from the MEC, DSKM, GAC and KM in clustering these two SAR image respectively.
Fig. 21. Original SAR images. (a) Xband SAR image; (b) Kuband SAR image. Fig. 22 shows that all methods are readily able to perform the classification of the Xband SAR image. Fig. 22(b) and (d) show that many mountainous regions in the bottom left are recognized as lake by KM and GAC. Fig.22 (a) and (c) show that MEC can recognize these regions and DSKM can obviously reduce these error recognitions. Meanwhile, KM confuses many mountainous regions in the top left with city seriously. MEC reduce these errors mostly. Generally speaking, the MEC method outputs relatively better partitioning.
Evolutionarybased Image Segmentation Methods
217
Fig. 22. Implementation results obtained from (a) MEC, (b) GAC, (c) DSKM and (d) KM in clustering the Xband SAR image. Fig. 23 shows that the MEC, GAC, DSKM and KM generate different results, and all the methods do not perform as well as the first SAR image. Generally speaking, the two methods based on the manifold distance generate better partitioning than GAC and KM. The dissimilarity measure based on Euclidean distance tends to confuse the crop with river. Relatively, MEC and DSKM generate better partitioning of the river region. In distinguishing the vegetation and crop, the partitioning of GAC and KM appear more discontinuous than the results of MEC and DSKM method. GAC and KM tend to confuse the vegetation with crop along the river, delineating the crop more than it should. However, MEC and DSKM tend to identify the vegetation in the bottom left as the river, due to the nature of the graylevel of the leads in that region. DSKM also tends to confuse the vegetation with crop in the region along the river and the bottom left of the image. Generally speaking, MEC does better than DSKM, GAC does better than KM, and MEC and DSKM do much better than GAC and KM, in partitioning this Kuband SAR image. 3.4.5 Robustness and computing time In order to compare the robustness of these methods, we follow the criteria used by [76]. In detail, the relative performance of the algorithm m on a particular data set is represented by the ratio bm of its mean value of Adjusted Rand Index Rm and the highest mean value of Adjusted Rand Index among all the compared methods:
218
Image Segmentation
Fig. 23. Implementation results obtained from (a) MEC, (b) GAC, (c) DSKM and (d) KM in clustering the Kuband SAR image. bm =
Rm max Rk
(13)
k
The best method m* on that data set has bm* =1, and all the other methods have bm≤ 1. The larger the value of bm, the better the performance of the method m is in relation to the best performance on that data set. Thus the sum of bm over all data sets provides a good measurement of the robustness of the method m. A large value of the sum indicates good robustness. Fig. 24 shows the distribution of bm of each method over the ten problems. For each method, the 10 values of bm are stacked and the sum is given on top of the stack. Fig. 24 reveals that MEC has the highest sum value. In fact, the bm values of MEC are equal or very close to 1 on all the test problems, which denotes MEC performs very well in different situations. Thus MEC is the most robust method among the compared methods.
Evolutionarybased Image Segmentation Methods
219
Fig. 24. Robustness of the compared algorithms Fig. 25 illustrates the sum of the computing time of the four algorithms in solving the twelve problems at an IBM IntelliStation M Pro 6233. From Fig. 25, it can be seen that the computing time of MEC is obviously longer than the computing time of GAC and KM. The main computational cost of MEC lies in computing the manifold distance between each pair of data points.
Fig. 25. Computing time of the compared algorithms.
4. Concluding remarks In the first method, we proposed a novel image segmentation approach based on memetic algorithm called MISA. MISA applies the idea of clustering to achieve image segmentation task. In preprocessing phase, graylevel cooccurrence matrix and wavelet decomposition are used for feature extraction. The watershed segmentation is employed to segment images into nonoverlap small regions. MISA tries to find the optimal combination of the watershed regions
220
Image Segmentation
under the criteria of interclass variance in feature space by a memetic algorithm. In MISA, after implementing clusterbased crossover and mutation, the individual learning procedure moves exocentric regions in current cluster to the one they should belong to according to the distance between these regions and cluster centers in feature space. Then, tournament selection and elitism strategy are used for producing the next generation. If stop criterion is satisfied, the segmentation result is outputted directly using the best individual in population. In order to evaluate the new algorithm, six artificial texture images, three remote sensing images and three natural images are employed in experiments. The EGSA, SCEA, GISA, FCM and KM are our compared algorithms. We exhibit typical segmentation results, convergence curves for all kinds of images, and numerical results on artificial texture images for which the true partitioning is known. Experimental results showed that MISA outperformed GISA on most of the tested images. The only difference between GISA and MISA lies in the individual learning strategy. Thus the new improvement of MISA could benefit from the learning operator. The comparisons between MISA and the compared algorithms showed that MISA was an effective image segmentation approach. Image segmentation remains a challenging problem. The main contribution of this study is to make substantial progress through the introduction of memetic computing methods to solving this problem. This study also shows that MAs provide useful computational tools. However we only designed one local search technique for image segmentation problem in this method. We will try to design more individual learning methods to cooperate together as well as higher order learning strategy in our future work. In this study, we have attempted to illustrate the power of MA for segmenting the three kinds of images, namely texture images, remote sensing images and natural images. In fact, it is difficult or impossible to design an always powerful generalpurpose algorithm. Thus, applying the proposed algorithm for practical applications such as magnetic resonance imaging (MRI) image segmentation and synthetic aperture radar (SAR) image segmentation with domainspecific knowledge is also planned in our future work. In the second method, we proposed the manifold evolutionary clustering using a novel representation method and a manifold distancebased dissimilarity measure to solve unsupervised image classification based on texture features. The experimental results on seven artificial data sets with different manifold structure, three artificial texture images and two SAR images showed that the novel manifold evolutionary clustering algorithm outperformed the KM, GAC and DSKM in terms of cluster quality and robustness. MEC made up the drawbacks of DSKM by evolutionary searching the cluster representatives from a combinatorial optimization viewpoint instead of recalculating the center of each cluster after cluster assignment. The manifold evolutionary clustering algorithm is a tradeoff of flexibility in clustering data with computational complexity. The main computational cost for the flexibility in detecting clusters lies in searching for the shortest path between each pair of data points which makes it much slower than GAC and KM. To find a fast or approximate computing method of the manifold distance is also our future work.
5. References [1] M. Wertheimer, “Laws of Organization in Perceptual Forms(partial translation),” A Sourcebook of Gestalt Psycychology, W.B.Ellis, ed., pp 7188, Harcourt, Brace, 1938. [2] J. Shi, J. Malik. Normalized cuts and image segmentation.IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI, vol. 22, no. 8, pp 888905, 2000.
Evolutionarybased Image Segmentation Methods
221
[3] L. G. Shapiro and G. C. Stockman. Computer Vision. New Jersey, PrenticeHall, ISBN 0130307963, pp 279325, 2001. [4] K.V. Mardia and T.J. Hainsworth. A Spatial Thresholding Method for Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 10, no. 6, pp. 919927, 1988. [5] P. Perona and J. Malik. Scalespace and edge detection using anisotropic diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.12, no. 7, pp. 629639, 1990. [6] K. Haris, S. N. Efstratiadis, N. Maglaveras, and A. K. Katsaggelos. Hybrid Image Segmentation Using Watersheds and Fast Region Merging. IEEE Transactions on Image Processing, vol. 7, no. 12, pp. 16841699, 1998. [7] P. F. Felzenszwalb and D. P. Huttenlocher. Efficient GraphBased Image Segmentation. International Journal of Computer Vision, vol. 59, no. 2, pp. 167181, 2004. [8] J. A. Hartigan and M. A.Wong. A KMeans clustering algorithm. Applied Statistics, vol. 28, pp. 100108, 1979. [9] B. Bhanu, S. Lee, J. Ming. Adaptive image segmentation using a genetic algorithm. IEEE Transactions on Systems, Man and Cybernetics, vol. 25, no. 12, pp. 15431567, 1995. [10] S. M. Bhandarkar, H. Zhang. Image segmentation using evolutionary computation. IEEE Transactions on Evolutionary Computation, vol. 3, no. 1, pp. 121, 1999. [11] P. Andrey. Selectionist relaxation: Genetic algorithms applied to image segmentation. Image and Vision Computing, vol. 17, no. 34, pp. 175187, 1999. [12] C. J. Veenman, M. J. T. Reinders, E. Backer. A cellular coevolutionary algorithm for image segmentation. IEEE Transactions on Image Processing, vol. 12, no. 3, pp. 304313, 2003. [13] M. Koppen, K. Franke, R. VicenteGarcia. Tiny GAs for image processing applications. IEEE Computational Intelligence Magazine, vol. 1, no. 2, pp. 1726, May 2006. [14] K. E. Melkemi, M. Batouche, and S. Foufou. A multiagent system approach for image segmentation using genetic algorithms and extremal optimization heuristics. Pattern Recognition Letters, vol. 27, no. 11, pp. 12301238, 2006. [15] Maoguo Gong, Licheng Jiao, Liefeng Bo, Ling Wang, Xiangrong Zhang. Image Texture Classification Using a Manifold Distance based Evolutionary Clustering Method. Optical Engineering, vol. 47, no. 7, pp. 0772011–10, 2008. [16] U. Maulik. Medical Image Segmentation Using Genetic Algorithms. IEEE Transactions on Information Technology in Biomedicine, vol. 13, no. 2, pp. 166173, 2009. [17] Q. C. Nguyen, Y. S. Ong and M. H. Lim. A Probabilistic Memetic Framework. IEEE Transactions on Evolutionary Computation, vol. 13, no. 3, pp. 604623, 2009. [18] K. W. C. Ku, M. W. Mak, and W. C. Siu. A study of the Lamarckian evolution of recurrent neural networks. IEEE Transactions on Evolutionary Computation, vol. 4, no. 1, pp. 3142, 2000. [19] S. A. Kazarlis, S. E. Papadakis, J. B. Theocharis, V. Petridis. Microgenetic algorithms as generalized hillclimbing operators for GA optimization. IEEE Transactions on Evolutionary Computation, vol. 5, no. 3, pp. 204217, 2001. [20] Y. S. Ong and A. J. Keane. Metalamarckian learning in memetic algorithms. IEEE Transactions on Evolutionary Computation, vol. 8, no. 2, pp. 99–110, 2004. [21] N. Krasnogor and J. Smith. A tutorial for competent memetic algorithms: model, taxonomy, and design issues. IEEE Transactions on Evolutionary Computation, vol. 9, no. 5, pp. 474–488, 2005.
222
Image Segmentation
[22] A. Caponio, G. L. Cascella, F. Neri, N. Salvatore, and M. Sumner. A fast adaptive memetic algorithm for online and offline control design of pmsm drives. IEEE Transactions on Systems, Man and Cybernetics  Part B, vol. 37, no. 1, pp. 2841, 2007. [23] M. Tang and X. Yao. A memetic algorithm for VLSI floorplanning. IEEE Transactions on Systems, Man and Cybernetics  Part B, vol. 37, no. 1, pp. 62–69, 2007. [24] J. Tang, M. H. Lim, and Y. S. Ong. Diversityadaptive parallel memetic algorithm for solving large scale combinatorial optimization problems. Soft Computation, vol. 11, no. 9, pp. 873888, 2007. [25] Z. Zhu, Y. S. Ong and M. Dash. WrapperFilter Feature Selection Algorithm Using A Memetic Framework. IEEE Transactions On Systems, Man and Cybernetics  Part B, vol. 37, no. 1, pp. 7076, Feb 2007. [26] S. Hasan, R. Sarker, D. Essam, and D. Cornforth. Memetic algorithms for solving jobshop scheduling problems. Memetic Computing, vol. 1, no. 1, pp. 6983, 2009. [27] JongHwan Kim, YeHoon Kim, SeungHwan Choi, InWon Park. Evolutionary multiobjective optimization in robot soccer system for education. IEEE Computational Intelligence Magazine, vol. 4, no. 1, pp. 3141, February 2009 . [28] Shinkyu Jeong, S. Hasegawa, K. Shimoyama, S. Obayashi. Development and investigation of efficient GA/PSOHYBRID algorithm applicable to realworld design optimization. IEEE Computational Intelligence Magazine, vol. 4, no. 3, pp. 3644, Aug. 2009. [29] S. C. Chiam, K. C. Tan, and A. Al. Mamun. A memetic model of evolutionary PSO for computational finance applications. Expert Systems With Applications, vol. 36, no. 2, pp. 36953711, March 2009. [30] K. C. Tan, S. C. Chiam, et al. Balancing exploration and exploitation with adaptive variation for evolutionary multiobjective optimization. European Journal of Operational Research, vol. 197, no. 2, pp. 701713, 2009. [31] J. H. Ang, K. C. Tan, and A. A. Mamun. An evolutionary memetic algorithm for rule extraction. Expert Systems With Applications, vol. 37, no. 2, pp. 13021315, March 2010. [32] Y. S. Ong, M. H. Lim, X Chen. Research Frontier: Towards Memetic Computing. Technical Report, School of Computer Engineering, Nanyang Technological University, Singapore, December 2009. [33] E. Fernandez, M. Grana, J. Cabello. An instantaneous memetic algorithm for illumination correction. In: Proceedings of the IEEE Congress on Evolutionary Computation, vol. 1, pp 11051110, 2004. [34] K. Batenburg. An evolutionary algorithm for discrete tomography. Discrete Applied Mathematics, vol. 151, no. 13, pp 3654, 2005. [35] V. Tirronen, F. Neri, T. Kärkkäinen, K. Majava, T. Rossi. An enhanced memetic differential evolution in filter design for defect detection in paper production. Evolutionary Computation, vol. 16, no. 4, pp 529555, 2008. [36] V. D. Gesù, G. L. Bosco, F. Millonzi, C. Valenti. A Memetic Algorithm for Binary Image Reconstruction. Lecture Notes in Computer Science, vol. 4958, Springer, pp 384395, 2008. [37] V. D. Gesù, G. L. Bosco, F. Millonzi, C. Valenti. Discrete tomography reconstruction through a new memetic algorithm. Lecture Notes in Computer Science, vol. 4974, Springer, pp 347352, 2008.
Evolutionarybased Image Segmentation Methods
223
[38] Qingzhou Zhang, Ziqiang Wang and Dexian Zhang. Memetic AlgorithmBased Image Watermarking Scheme. Lecture Notes in Computer Science, vol. 5263, Springer, pp 845853, 2008. [39] J. C. Bezdek. Pattern Recognition with Fuzzy Objective Function Algoritms. Plenum Press, New York, 1981. [40] Xiangrong Zhang, Licheng Jiao, Fang Liu, Liefeng Bo, Maoguo Gong. Spectral Clustering Ensemble Applied to Texture Features for SAR Image Segmentation. IEEE Transactions on Geoscience and Remote Sensing, vol. 46, no. 7, pp 21262136, 2008. [41] R. M. Haralick, K. Shanmugam and I. Dinstein. Textural features for images classification. IEEE Transactions on Systems, Man, and Cybernetics, vol. 3, no. 6, pp. 610621, 1973. [42] S. Fukuda and H. Hirosawa. A waveletbased texture feature set applied to classification of multifrequency polarimetric SAR images. IEEE Transactions on Geoscience and Remote Sensing, vol. 37, no. 8, pp. 22822286, 1999. [43] D. A. Clausi and B. Yue. Comparing Cooccurrence Probabilities and Markov Random Fields for Texture Analysis of SAR Sea Ice Imagery. IEEE Transactions on Geoscience and Remote Sensing, vol. 42, no. 1, pp. 215228, 2004. [44] D.A. Clausi. An analysis of cooccurrence texture statistics as a function of grey level quantization,” Canadian Journal of Remote Sensing, vol. 28, no. 1, pp. 4562, 2002. [45] D. Wang. A Multiscale Gradient Algorithm for Image Segmentation Using Watersheds. Pattern Recognition, vol. 30, no. 12, pp. 20432052, 1997. [46] A. Hoeffler, U. Leysner, and J. Weidermann. Optimization of the layout of trusses combining strategies based on Mitchel’s theorem and on biological principles of evolution. In: Proceedings of the Second Symposium on Structural Optimization, Milan, Italy, 1973. [47] P. Moscato. On Evolution, Search, Optimization Algorithms and Martial Arts: Towards Memetic Algorithms. Report 826, Caltech Concurrent Computation Program, California Institute of Technology, Pasadena, 1989. [48] L. O. Hall, I. B. Ozyurt and J. C. Bezdek. Clustering with a genetically optimized approach. IEEE Transactions on Evolutionary Computation, vol. 3, no. 2, pp. 103112, 1999. [49] U. Maulik and S. Bandyopadhyay. Genetic algorithmbased clustering technique. Pattern Recognition, vol. 33, no. 9, pp. 14551465, 2000. [50] H. Pan, J. Zhu and D. Han. Genetic algorithms applied to multiclass clustering for gene expression data. Genomics, Proteomics & Bioinformatics, vol. 1, no. 4, pp. 279287, 2003. [51] Maoguo Gong, Licheng Jiao, Ling Wang, Liefeng Bo. DensitySensitive Evolutionary Clustering. In: Proceedings of the 11th PacificAsia Conference on Knowledge Discovery and Data Mining, PAKDD07. SpringerVerlag, Lecture Notes in Computer Science, LNAI, vol. 4426, pp. 507514, 2007. [52] P. Brodatz. Textures: A Photographic Album for Artists and Designers. Dover Publications, New York, 1966. [53] M. Tuceryan and A. K. Jain, "Texture analysis," In Handbook of Pattern Recognition and Computer Vision, C. Chen, L. Pau and P. Wang, Eds., pp. 235276, World Scientific, Singapore (1993). [54] D. A. Clausi and B. Yue, "Comparing Cooccurrence Probabilities and Markov Random Fields for Texture Analysis of SAR Sea Ice Imagery," IEEE Transactions on Geoscience and Remote Sensing, 42(1), 215228 (2004).
224
Image Segmentation
[55] R. M. Haralick, K. Shanmugam and I. Dinstein, "Textural features for images classification," IEEE Transactions on Systems, Man, and Cybernetics, 3(6), 610621 (1973). [56] H. Frigui and R. Krishnapuram, "A Robust Competitive Clustering Algorithm with Applications in Computer Vision," IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(5), 450465 (1999). [57] Y. Leung, J. Zhang and Z. Xu, "Clustering by SpaceSpace Filtering," IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(12), 13961410 (2000). [58] J. A. Hartigan and M. A.Wong, "A KMeans clustering algorithm," Applied Statistics, 28, 100108 (1979). [59] L. O. Hall, I. B. Ozyurt and J. C. Bezdek, "Clustering with a genetically optimized approach, " IEEE Transactions on Evolutionary Computation, 3(2), 103112 (1999). [60] U. Maulik and S. Bandyopadhyay, "Genetic algorithmbased clustering technique, " Pattern Recognition, 33(9), 14551465 (2000). [61] H. Pan, J. Zhu and D. Han, "Genetic algorithms applied to multiclass clustering for gene expression data," Genomics, Proteomics & Bioinformatics, 1(4), 279287 (2003). [62] J. Handl, J. Knowles, "An evolutionary approach to multiobjective clustering," IEEE Transactions on Evolutionary Computation, 11(1), 5676 (2007). [63] M.C. Su and C. H. Chou, "A modified version of the KMeans algorithm with a distance based on cluster symmetry," IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(6), 674680 (2001). [64] D. Charalampidis, "A Modified KMeans Algorithm for Circular Invariant Clustering," IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(12), 18561865 (2005). [65] L. Wang, L. F. Bo and L. C. Jiao, "A modified KMeans clustering with a densitysensitive distance metric," Lecture Notes in Computer Science, 4062, 544551 (2006). [66] A. Blum and S. Chawla, "Learning from labeled and unlabeled data using graph mincuts," In Proceedings of the Eighteenth International Conference on Machine Learning, 1926 (2001). [67] O. Bousquet, O. Chapelle and M. Hein, "Measure based regularization," Advances in Neural Information Processing Systems 16 (NIPS), MIT Press, Cambridge, MA (2004). [68] G. Syswerda, "Uniform crossover in genetic algorithms," In Proceedings of the Third International Conference on Genetic Algorithms, pp29, Morgan Kaufmann Publishers, San Francisco, CA (1989). [69] D. Whitley, "A genetic algorithm tutorial," Statistics and Computing, 4, 6585 (1994). [70] D. E. Goldberg, Genetic Algorithms in Search, Optimization, and Machine Learning, AddisonWesley, Massachusetts (1989). [71] D. A. Clausi, "An analysis of cooccurrence texture statistics as a function of grey level quantization," Can. J. Remote Sens., 28(1), 4562 (2002). [72] L. Hubert and P. Arabie, "Comparing partitions," Journal of Classification, 193218 (1985). [73] K. Y. Yeung and W. L. Ruzzo, "Principal component analysis for clustering gene expression data," Bioinformatics, 17(9), 763774 (2001). [74] W. Rand, "Objective criteria for the evaluation of clustering methods," Journal of the American Statistical Association, 66(336), 846850 (1971). [75] P. Brodatz, Textures: A Photographic Album for Artists and Designers, Dover Publications, New York (1966). [76] X. Geng, D. C. Zhan and Z. H. Zhou, "Supervised nonlinear dimensionality reduction for visualization and classification," IEEE Transactions on Systems, Man, and Cybernetics  Part B: Cybernetics, 35(6), 10981107 (2005).
11 Segmentation of Handwritten Document Images into Text Lines Vassilis Katsouros and Vassilis Papavassiliou
Institute for Language and Speech Processing/R.C. “Athena” Greece 1. Introduction There are many governmental, cultural, commercial and educational organizations that manage large number of manuscript textual information. Since the management of information recorded on paper or scanned documents is a hard and timeconsuming task, Document Image Analysis (DIA) aims to extract the intended information as a human would (Nagy, 2000). The main subtasks of DIA (Mao et al. 2003) are: i) the document layout analysis, which aims to locate the “physical” components of the document such as columns, paragraphs, text lines, words, tables and figures, ii) the document content analysis, for understanding/labelling these components as titles, legends, footnotes, etc. iii) the optical character recognition (OCR) and iv) the reconstruction of the corresponding electronic document. The proposed algorithms that address the abovementioned processing stages come mainly from the fields of image processing, computer vision, machine learning and pattern recognition. Actually, some of these algorithms are very effective in processing machineprinted document images and therefore they have been incorporated in the workflows of wellknown OCR systems. On the contrary, no such efficient systems have been developed for handling handwritten documents. The main reason is that the format of a handwritten manuscript and the writing style depend solely on the author's choices. For example, one could consider that text lines in a machineprinted document are of the same skew, while handwritten text lines may be curvilinear. Text line segmentation is a critical stage in layout analysis, upon which further tasks such as word segmentation, grouping of text lines into paragraphs, characterization of text lines as titles, headings, footnotes, etc. may be developed. For instance, a task for textline segmentation is involved in the pipeline of the Handwritten Address Interpretation System (HWAIS), which takes a postal address image and determines a unique delivery point (Cohen et al., 1994). Another application, in which text line extraction is considered as a preprocessing step, is the indexing of George Washington papers at the Library of Congress as detailed by Manmatha & Rothfeder, 2005. A similar document analysis project, called the Bovary Project, includes a textline segmentation stage towards the transcription of the manuscripts of Gustave Flaubert (Nicolas et al., 2004a). In addition, many recent projects, which focus on digitisation of archives, include activities for document image understanding in terms of automatic or semiautomatic extraction and indexing of metadata such as titles, subtitles, keywords, etc. (Antonacopoulos & Karatzas, 2004, Tomai et al., 2002). Obviously, these activities include textline extraction.
226
Image Segmentation
Fig. 1. Handwritten document images included in the training set of ICDAR 2007 Handwriting Segmentation: 024.tif, 016.tif, 010.tif, 044.tif, 068.tif and 026.tif
Segmentation of Handwritten Document Images into Text Lines
227
This chapter is a comprehensive survey of methods exploited to segment handwritten document images into text lines during the last two decades. The main underlying assumption is that the nontextual information has been removed and the document image comprises only plain text. Even though this hypothesis seems to simplify the task, textline segmentation has to face many challenges, such as the touching or overlapping text lines and the variation of skew angles. Some typical examples of handwritten document images are illustrated in fig. 1. In the next sections, we will describe, in detail, how the proposed methods try to overcome these difficulties. The main idea in textline segmentation is to consider the foreground pixel density and employ one of the following three broad classes of techniques (Razak et al., 2008). The first includes traditional methods that have been applied to printed documents and is based on the analysis of projection profiles. The second class incorporates grouping techniques, also known as bottomup strategies that attempt to build text lines by considering the alignments of foreground pixels or connected components. The third category includes smearing approaches that aim to enhance the text lines structure by applying linear or morphological filters and exploiting wellknown image segmentation methods, such as level sets, scalespace analysis (Lindeberg, & Eklundh, 1992), etc. Moreover, there are some methods that exploit a combination of these techniques with the purpose to improve further the segmentation results. In section 2, the problem definition and the main challenges of the task are described. Several techniques and the contributions of the most effective algorithms within each class are presented in section 3. The available recourses for validating relative methods and the comparative results of recent contests are reported in section 4. Finally, the chapter is concluded with a discussion of the main outcomes.
2. Background When creating a manuscript, the author selects the writing instrument and the paper in such a way as to produce a readable document, namely a document with high contrast between the traces of the pen (foreground) and the paper (background). As a consequence, the digitisation of these documents in most of the cases generates binary images. In the case of greyscale document images, most of the proposed methods for textline extraction incorporate an initial processing stage of binarization by employing global (Otsu, 1979) or local thresholding (Niblack, 1986, Sauvola & Pietikäinen, 2000). However, some recent techniques combine the results of processing both the greyscale and the binary versions of the document image. In binary document images, the traces of the writing instrument are represented by pixels that have value one and constitute the text. The other pixels have value zero, corresponding to the background. The convention for using the values 1 and 0 for foreground and background pixels respectively is very common in studies related to the binary images. 2.1 Definitions Considering that a twodimensional binary image is defined on the discrete plane Z 2 and by selecting a square grid and a certain type of connectivity (e.g. 8n denotes all the neighbours of a pixel, while 4n indicates only the cross neighbours), we could represent objects or shapes of the image as groups of neighbouring pixels with the same value. In the view of set theory, a binary image is modelled by the corresponding set S as follows:
228
Image Segmentation
S = {x ∈ Z 2 : s( x ) = 1}
(1)
where x denotes the coordinates of a pixel and s is a binary function s : Z 2 → {0,1} (Soille,
2004). Then, the shapes in the binary image are defined as the maximal connected subsets of the image foreground pixels, called connected components (CCs). Therefore, the CCs in a document image could be noise specks, dots, single symbols, groups of touching characters, parts of a character that is broken, etc. The extraction of CCs is accomplished by applying a connected component operator that assigns the same value to every pixel of each distinct CC. A common algorithm for identifying CCs is outlined in (Haralick & Shapiro, 1992). A text line could be considered as a group of CCs that are adjacent, relative close to each other and correspond to occurrences of text elements. By adopting this simple definition, textline segmentation produces an image, in which each text pixel has a value that identifies the proper textline (fig. 2, left). Alternatively, a text line could be represented by a large CC that covers the corresponding part of the image, or by a closed curve that represents the boundary of each text line (fig. 2, right).
Fig. 2. Representation of text lines as: groups of text pixels with the same value (left) and parts of the document image (right) 2.2 Challenges The main challenges of textline segmentation of handwritten documents arise from the variation of the skew angle between text lines or along the same text line, the existence of overlapping and/or touching lines, the variable character size, the variation of intraline and interline gaps and the nonmanhattan layout. To overcome these difficulties, an efficient textline segmentation algorithm should represent the boundaries of each text line by a closed curve instead of enclosing a text line with a rectangular. In addition, such an algorithm should incorporate procedures for cutting CCs which running along two or more text lines. These are the major differences in handling manuscripts rather than machineprinted documents.
Segmentation of Handwritten Document Images into Text Lines
229
Actually, text line segmentation in printed documents could be seen as a solved problem (Plamondon & Srihari, 2000), which is equivalent with the estimation of the document’s skew angle. A comprehensive survey and the annotated bibliography on skew detection of printed document images are presented in (Hull, 1998). To this point, it is supposed that text lines in a printed document have a unique skew angle. Thus, the proper rotation of the image will result to horizontal text lines that could be easily located. A wellknown and efficient method for layout analysis of printed documents, called Docstrum, is outlined in (O’Gorman, 1993). The main assumption is that the distances between characters of the same text line are smaller than the distances between characters of successive text lines. The fact that this assumption does not hold for manuscripts, explains why Docstrum cannot handle handwritten documents successfully, as shown in fig. 3. As mentioned above, this task focuses on text elements only. Therefore, noise removal is the first preprocessing step. In binary document images which incorporate merely textual information, noise removal is equivalent with the elimination of CCs that do not represent text elements but mainly occur due to misses at the digitisation phase. In document images which are considered as normally “clear”, simple methods adopting median filters or heuristics based on geometrical and topological properties of the CCs are employed to remove the noisy data. For example, a large CC, which is lying on the edges of the image arises due to the inaccurate placement of the manuscript to the scanner and need to be removed. However, the extraction of actual text elements from digitised historical archives might be a significant issue. Actually, historical documents suffer from smudges, smears, faded print and bleedthrough of writing from the opposite side of a page (LikformanSulem et al., 2007). Document images are captured in high resolution (about 300dpi) in order to be suitable for OCR engines. However, text lines have an underlying texture that is manifest in printed documents at low resolutions about 40dpi (Bloomberg, 1996). Hence, subsampling methods that prevent aliasing are also applied in textline segmentation.
3. Proposed methods Handwritten documents are characterised by high variability of writing styles. Thus, most of the existing methods adapt to the properties of a document image and eliminate the use of prior knowledge. According to the adopted strategy, the existing methods are classified in three categories, which are discussed in this section. In general, textline segmentation techniques are script independent. However, some special scripts such as Indian and Arabic incorporate many characters with diacritical points that require great care in CCs assignment. 3.1 Projectionbased methods Given that an image A with height M and width N could be considered as a matrix of the same dimensions, the projection profile of the image is defined as follows: N
P(i ) = ∑ A(i , j ), i = 1,..., M
(2)
j =1
Therefore, the projection is a onedimensional signal that denotes the amount of text pixels per row. Consequently, the lobes (valleys) of the projection correspond to foreground
230
Image Segmentation
Fig. 3. Textline segmentation of a machineprinted and a handwritten document image by exploiting the Docstrum method (O’Gorman, 1993). The figure is reprinted (Li et al., 2008) with permission from the author.
Segmentation of Handwritten Document Images into Text Lines
231
(background) areas of the image. Supposing that the text lines have the same skew angle, the amplitude and the frequency of the projection are maximized when the skew of the text is zero. Based on this characteristic, many proposed approaches rotate the image through a range of angles, calculate the projection for each angle and estimate the global skew angle according to a properly selected criterion. Such a criterion could be based on the variance of the projections (Bloomberg et al., 1993) and the sum of the coefficients of the power spectrum (Postl, 1988). After estimating the unique skew angle and rotating the image appropriately, the local minima of the projection allocate the positions of textline separators. In machineprinted documents, the separators will be horizontal lines lying within the space between adjacent text lines. Apparently, this might be occurred in some manuscripts written with great care and consistency. In fact, this technique is adopted for the process of 1000 sampled documents from the George Washington corpus at the Library of Congress (Manmatha & Rothfeder, 2005). The additional processing step is the smoothing of the projection by applying a Gaussian low pass filter in order to eliminate false alarms (i.e. insignificant minima) and reduce the noise. A similar method (Santos et al., 2009) includes a postprocessing stage for labeling candidate text lines as false or actual, according to their geometrical features (i.e. lines, which correspond to very narrow lobes of the projection, should be removed). Although this approach has been tested in 150 images from the IAM offline handwritten database (Marti & Bunke, 2002) and showed almost excellent results, it is worth to mention that the textline segmentation in documents of this database seems to be a straightforward task. A common feature of manuscripts is the overlapping of successive text lines due to the ascenders and/or descenders of some characters. Hence, the formulation of a horizontal line as a separator is often not feasible. With the purpose to overcome this difficulty, some researchers exploit the projections in order to locate the areas (i.e. the areas between two successive maxima) in which the separators should be allocated. Considering ascenders/descenders as obstacles, the algorithms try to find a path from the left to the right edge in each area, by attempting to move around the obstacles (fig. 4). If the deviation is too high, the algorithm intersects the character and continues forward. Such segmenters could be based on predefined constrains (Yanikoglu & Sandon, 1998) or on the minimization of a proper cost function (Weliwitage et al., 2005).
Fig. 4. The line separator should be lying in the gray area between the text lines (left, reprinted from Yanikoglu & Sandon, 1998). The final line separator (right, reprinted from Weliwitage et al., 2005). Since variation of skew angles between text lines or along the same text line is common in manuscripts, the global projections based approaches cannot provide a general solution. Piecewise projections can be seen as a modification of global projections to the properties of handwritten documents, by which the separators between text lines are drawn in staircase function fashion along the width of the document page. The main idea of piecewise projections is to divide the document image in vertical nonoverlapping equiwidth zones
232
Image Segmentation
and find the critical local minima of each projection. The selection of the width of the zones is a tradeoff between the local skew and the text density. In other words, if the width was large enough, the skew should not be considered as constant. Furthermore, a narrow width would produce zones, which do not include adequate amount of text. Relative experiments showed that a zone width equal to 5% of the document image width seems to be an appropriate value. However, the nonmanhattan layout of manuscripts will result in vertical zones without enough foreground pixels for every text line. In such cases, some local minima may be lost and the results of two adjacent zones may be ambiguous. To deal with these problems, we calculate a smooth version of the projections influenced by the neighboring zones and introduce a separatordrawing algorithm that combines separators of consecutive zones according to their proximity and the local text density as shown in fig. 5 (Papavassiliou et al. 2010).
0
02
0
02
04
06
08
1
04
06
08
1
12
12
14
14
Fig. 5. The separatordrawing algorithm (reprinted from Papavassiliou et al., 2010). The separators in green are ambiguous (first column). New separators in these areas (second column) should be located at the global minima of the metric function (third column) influenced by the local foreground density and the proximity of the separators. The separators with the same colors are associated (fourth column). The last challenge that projectionbased methods have to face is the assignment of CCs to the proper textlines. In most of the cases, this is a straightforward task since the majority of CCs lie between two line separators. However, some CCs either overlap with two text lines (i.e. characters with ascenders/descenders) or run along two text lines (touching lines). In order to preserve the ascending/descending symbols from been corrupted by arbitrary cuts, several heuristics based on the geometrical and topological properties such as the height, the length, the distance of neighboring CCs, etc. have been proposed. An interesting approach models the text lines by bivariate Gaussian densities considering the coordinates of the pixels of the CCs that have been already assigned. Then, the probabilities that the CC under consideration belongs to the upper or lower text lines are estimated and the decision is made by comparing the probabilities (Arivazhagan et al., 2007). This method has been tested on 720 documents including English, Arabic and children’s handwriting and performed a detection rate of 97.31%. In the case that a character should be split, the segmentation occurs at a proper cross point of the skeleton (Lam et al., 1992) by taking account the distance from the separator (fig. 6) as well as the slope and curvature of the stroke (Kuzhinjedathu et al., 2008).
Segmentation of Handwritten Document Images into Text Lines
233
Fig. 6. Segmentation of a CC running along two text lines (reprinted from Papavassiliou et al., 2010). 3.2 Grouping methods Grouping approaches, also known as bottomup techniques, were very popular in textline segmentation of handwritten documents due to their success in prior tasks concerning the process of machineprinted documents (Simon et al., 1997). These methods try to group CCs considering geometrical and topological characteristics of the CCs such as their distances, locations and orientation. The common strategy is to represent each CC with an appropriate vector (e.g. the coordinates of its gravity centre), calculate the distances between that point and the corresponding points of its neighbouring CCs and compare the distances with a proper predefined value. If the constrain is satisfied the CCs are grouped (Khandelwal et al., 2009). Since such methods strongly depend on the values of the thresholds, they cannot handle variation in writing styles. In fact, (Feldbach & Tonnies, 2001) report that a similar method tested on historical church registers achieved a 97% recall rate when the thresholds values are adjusted to specific authors but decreased to 90% when these parameters remained constant for various authors. As a result, many recent methods produce an adjacency graph constructed by linking the pairs of neighbouring CCs with edges. Then, they try recursively to find the minimum spanning tree, which likely crosses CCs of the same text line (Nicolas et al., 2004b). Additionally, the orientations of the edges that connect these points are also examined. Supposing that CCs in the same text line could be represented by almost collinear points, Hough transform (Duda, & Hart, 1972) has been applied on handwritten documents. Although Houghbased approaches locate text lines with different skew angles correctly, they are not flexible to follow variation of skew along the same text line (fig. 7 left). 3.3 Smearing methods In general, smearing approaches include two main processing steps. The first stage aims to enhance text areas by blurring the input image. The second step concerns the modification and use of wellknown image segmentation methods in order to formulate the text lines. Li et al. (2008) apply an anisotropic Gaussian filter to smooth the image and provide a greyscale “probabilistic” image that denotes the text line distribution. It is worth to mention that the horizontal dimension of the filter is greater than the vertical in order to advance the mainly horizontal textline orientation. Then, they locate the initial boundaries of text lines or parts of them using Niblack’s algorithm for binarization and finding the contours of the CCs in the produced binary image. Next, the level set method (Osher & Fedkiw, 2003) is adopted and the boundaries evolve concerning the local curvature and density with the purpose of moving fast along the horizontal direction and towards areas with high
234
Image Segmentation
Fig. 7. (left) Results of Houghbased transform (reprinted from Louloudis et al., 2008); (right) Segmentation of CCs based on mincut/maxflow algorithm (reprinted from Kennard & Barret, 2006). probability being text. This method has been tested on several manuscripts in different scripts and performed pixellevel hit rates varying from 92% to 98%. As expected, the proposed method fails when the gap between two neighbouring text lines is smaller than the vertical dimension of the Gaussian filter. A similar approach adopts the MumfordShah model to locate text lines and then applies morphological operations to either segment merged text lines or join parts of the same text line (Du et al., 2008). Instead of using erosions and dilations Yin & Liu (2009) apply a modification of the variational Bayes framework to the downsampled input image. The binary image (after downsampling, smoothing and binarization) is considered as a mixture model in which each CC is a Gaussian component. Since, a CC may correspond to more than one text lines, a CC is split according to the second eigenvalue of the covariance (i.e. thick CCs are candidate to be segmented). Other smearing strategies enhance the text areas by estimating an adaptive local connectivity map (ALCM). This map is actually a greyscale image in which each pixel has a value that denotes the amount of text pixels in the proximity of the pixel under consideration. By converting this image to a binary one, the resulting CCs represent the text areas of the document image. In most cases, these CCs include many text lines and should be split. Considering such a CC as a graph with candidate source (sink) nodes the pixels in the upper (lower) part, the mincut/max flow algorithm has been proposed for segmenting the CC to its main components (Kennard & Barret, 2006), as illustrated in fig. 7 right. Alternatively, the ALCM could be replaced by another image produced by applying the run length smoothing algorithm (RLSA). In this image, the value of a pixel is the distance of that pixel from the nearest text pixel. As previously, text areas are represented by pixels with small (dark) values while background corresponds to bright areas. Then, dark areas are grouped according to their orientation and proximity in order to formulate text lines (Shi & Govindaraju, 2004).
235
Segmentation of Handwritten Document Images into Text Lines
4. Evaluation Research groups test their method either on their own collection of handwritten documents or on a public available database. As a result, many such test sets have been constructed. In this section, we refer to three wellknown collections that could be used for evaluating various processing steps such as textline segmentation, word segmentation and character recognition. The IAM handwriting database1 consists of 1539 pages of scanned text containing 13353 and 115320 isolated and labelled text lines and words respectively. Although, this database is an excellent resource for validating word segmentation and character recognition algorithms, the textline extraction seems not to be a complex task due to significant gaps between successive text lines in many images. Another famous database is the NIST Handprinted Forms and Characters Database2, which includes handwritten sample forms from 3600 writers. This collection is mainly used for evaluating character recognition techniques but it could be also employed to assess textline segmentation algorithms. The other two benchmarking databases are the training and test sets constructed for the Handwriting Segmentation Contests in the context of ICDAR 20073 and 20094. The first collection consists of 100 images (20 and 80 for training and test, respectively). The second database includes these images (as the training set) and 200 images that construct the test set. The documents are either modern manuscripts written by several writers in several languages (English, French, German and Greek) or historical handwritten archives, or document samples selected from the web. It is worth to mention that none of the documents includes any nontext elements (lines, drawings, etc.) The comparative results of the algorithms, which participated in ICDAR 2007 Handwriting Segmentation Contest (Gatos et al., 2007) or have been tested on this dataset, are presented in Table 1. Detection Rate (DR) denotes the ratio between the number of text lines detected correctly and the number of groundtruth lines (1771). Similarly, Recognition Accuracy (RA) is calculated by dividing the number of correctly detected lines with the total number of detected text lines. FM denotes the harmonic mean of DR and RA.
BESUS (Das et al., 1997) DUTHARLSA ILSPLWSeg (Papavassiliou et al., 2010) PARC UoAHT (Louloudis et al., 2008) PROJECTIONS RidgesSnakes (Bukhari et al., 2009) Shredding (Nikolaou & Gatos, 2009)
DR(%) 86.6 73.9 97.3 92.2 95.5 68.8 97.3 98.9
RA(%) 79.7 70.2 97.0 93.0 95.4 63.2 95.4 98.3
FM% 83.0 72.0 97.1 92.6 95.4 65.9 96.3 98.6
Table 1. Evaluation results of algorithms tested on the database of ICDAR2007 Handwriting Segmentation Contest. 1
http://www.iam.unibe.ch/fki/databases/iamhandwritingdatabase http://www.nist.gov/srd/nistsd19.cfm 3 http://users.iit.demokritos.gr/~bgat/HandSegmCont2007/resources.htm 4 http://users.iit.demokritos.gr/~bgat/HandSegmCont2009/resources.html 2
236
Image Segmentation
The evaluation results of algorithms participated in ICDAR2009 Handwriting Segmentation Contest are presented in Table 2. We mention that the test set consists of 200 binary images and their dimensions vary from 650x825 to 2500x3500 pixels. The total number of text lines included in this dataset is 4034.
CASIAMSTSeg (Yin & Liu, 2008) CMM CUBS (Shi et al., 2009) ETS ΙLSPLWSeg09 Jadavpur Univ LRDE PAIS AegeanUniv (Kavallieratou et al., 2003) PortoUniv (Cardoso et al., 2008) PPSL REGIM
DR(%)
RA(%)
FM(%)
95.86
95.51
95.68
98.54
98.29
98.42
99.55
99.50
99.53
86.66 99.16 87.78 96.70 98.49
86.68 98.94 86.90 88.20 98.56
86.67 99.05 87.34 92.25 98.52
77.59
77.21
77.40
94.47
94.61
94.54
94.00 40.38
92.85 35.70
93.42 37.90
Table 2. Comparative results of ICDAR2009 Handwriting Segmentation Contest. PAIS and ILSP are based on piecewise projections and achieved high results. On the other hand, similar methods presented poor results because they either adopt global projections (PROJECTIONS) or divide the image into only three vertical zones (AegeanUniv). Ten participating methods are classified as grouping approaches. In particular, five methods (Jadavpur Univ, CASIAMSTSeg, CMM, PPSL and REGIM) introduce constrains on the topological and geometrical properties of the CCs in order to create groups of CCs that correspond to text lines. Since, these approaches require many predefined thresholds, the selection of appropriate (improper) values results in good (poor) results. Three approaches (BESUS, ETS and PARC) apply morphological operations to produce new CCs by merging the initial neighbouring CCs and then adopt similar constrains. Another grouping approach is UoAHT, which exploits the Hough transform. As expected, the algorithm is not very effective when the skew of a text line varies along its width. Although, two methods (DUTHARLSA and CUBS) exploit the RLSA algorithm, their results differ significantly. The reason is that CUBS applies the RLSA algorithm in five directions (20o, 10o, 0o, 10o, and 20o) and combines the results in order to calculate the local skew of each text line. PortoUniv proposes a tracing algorithm that tries to find proper paths that connect the edges of the image without cutting the textual elements. A similar approach (Shredding) includes a preprocessing step for blurring and then exploits the tracing algorithm. LRDE is a fast algorithm that enhances the test areas by anisotropic Gaussian filtering, smoothes the image by applying morphological operations and segments it by using the watershed transform (Vincent & Soille, 1991). A recent method (RidgesSnakes) uses a multioriented anisotropic Gaussian filter bank for smoothing, approximates the ridges as the central lines of the text parts and then the ridges evolve until they overlap the CCs of the manuscript.
Segmentation of Handwritten Document Images into Text Lines
237
5. Conclusions After reviewing the existing methods for textline segmentation we conclude that there are pros and cons for each approach. For example, piecewise projection based methods can handle text lines with varying skew angles, but fail when the document includes high degree of curl text lines. In addition, the benefit of some grouping strategies is that they succeed to extract text lines from a complex layout but may fail to segment touching text lines. Regarding smearing approaches, some of them seem to be promising since they exploit image segmentation algorithms that have been already applied on other kinds of images. However, they may merge two successive text lines if the gap between them is not large enough. As a conclusion, we report that the existing methods do not generalize very well to all possible variations encountered in handwritten documents. Thus, textline segmentation of handwritten documents remains an open issue. This fact explains why the number of relative papers and contests is increasing. Since different methods can face different challenges of this task, we foresee that a combination of complementary techniques could result in a generalized solution.
6. References Antonacopoulos A. & Karatzas D (2004), Document Image analysis for World War II personal records, Proceedings of International Workshop on Document Image Analysis for Libraries DIAL’04, pp. 336341, ISBN 076952088X, Palo Alto, USA, January 2324, 2004 Arivazhagan, M.; Srinivasan, H. & Srihari, S. (2007). A statistical approach to line segmentation in handwritten documents, Proceedings of Document Recognition and Retrieval XIV SPIE, Vol.6500, Νο.1, ISSN 0277786X, San Jose, CA, USA, January 30, 2007 Bloomberg, D.S. & Kopec, G. (1993). Method and apparatus for identification and correction of document skew. Xerox Corporation, U.S. Patent 5,563,403, October 8, 1996. Bloomberg, D.S. (1996). Textured Reductions for Document Image Analysis, Proceedings of IS&T/SPIE EI ’96, Conference 2660: Document Recognition III, pp. 160174, ISBN: 9780819420343, San Jose, USA, March 7, 1996 Bukhari, S.S.; Shafait, F. & Bruel, T.M. (2009). ScriptIndependent Handwritten Textlines Segmentation using Active Contours, Proceedings of International Conference on Document Analysis and Recognition, pp. 446450, ISSN 15205363, Barcelona, Spain, July 2629, 2009 Cardoso, J.S.; Capela, A.; Rebelo, A. & Guedes, C. (2008). A Connected Path Approach for Staff Detection on a Music Score, Proceedings of International Conference on Image Processing, pp.10051008, ISSN 15224880, San Diego, CA, USA, October 215, 2008 Cohen, E.; Hull, J.J. & Srihari, S.N. (1994). Control Structure for Interpreting Handwritten Addresses. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.16, No.10, (October 1994), pp. 1049 1055, ISSN 01628828 Das, A.K.; Gupta, A. & Chanda, B. (1997). A Fast Algorithm for Text Line & Word Extraction from Handwritten Documents, Image Processing & Communications, Vol. 3, No. 12, pp. 8594
238
Image Segmentation
Du, X.; Pan, W. & Bui, T.D. (2009). Text Line Segmentation in Handwritten Documents Using MumfordShah Model, Pattern Recognition, Vol. 42, No. 12, (December 2009), pp. 31363145, ISSN 00313203 Duda, R.O. & Hart, P. E. (1972). Use of the Hough Transformation to Detect Lines and Curves in Pictures, Communications of the ACM, Vol. 15, No. 1, (January 1972), pp. 1115, Feldbach M. & Tonnies, K.D. (2001) Line Detection and Segmentation in Historical Church Registers, Proceedings of International Conference on Document Analysis and Recognition, pp. 743747, ISBN 0769512631, Seattle, WA , USA, September 1013, 2001 Gatos, B.; Antonacopoulos, A. & Stamatopoulos, N. (2007). ICDAR2007 Handwriting Segmentation Contest, Proceedings of International Conference on Document Analysis and Recognition, pp. 12841288, Curitiba, Brazil, September 2326, 2007 Gatos, B.; Stamatopoulos, N. & Louloudis, G. (2009). ICDAR2009 Handwriting Segmentation Contest, Proceedings of International Conference on Document Analysis and Recognition, pp. 13931397, ISSN 15205363, Barcelona, Spain, July 2629, 2009 Haralick, R.M. & Shapiro, L.G. (1992). Computer and Robot Vision, AddisonWesley Longman Publishing Co., Inc. Boston, MA, USA, ISBN:0201569434 Hull, J.J. (1998). Document Image Skew Detection: Survey and Annotated Bibliography, In: Document Analysis Systems II, Hull, J.J. & Taylor, S.L. (Eds.), pp. 4064, World Scientific Publishing Co. Pte. Ltd, ISBN 9789810231033 Kavallieratou, E.; Dromazou, N.; Fakotakis, N. & Kokkinakis, G. (2003). An Integrated System for Handwritten Document Image Processing, International Journal of Pattern Recognition and Artificial Intelligence, Vol. 17, No. 4, pp. 101120, DOI 10.1.1.107.7988 Khandelwal, A.; Choudhury, P.; Sarkar, R.; Basu, S.; Nasipuri, M. & Das, N. (2009). Text Line Segmentation for Unconstrained handwritten Document Images Using Neighborhood Connected Components Analysis, Lecture Notes in Computer Science, Vol. 5909/2009, pp. 369374, DOI: 10.1007/9783642111648_60 Kuzhinjedathu, K.; Srinivansan, H. & Srihari, S. (2008). Robust Line Segmentation for Handwritten Documents, Proceedings of Document Recognition and Retrieval XV ST/SPIE Annual Symposium, Vol. 6815, San Jose, CA, January 2008 Lam, L.; Lee, S.W. & Suen, C.Y. (1992). Thinning MethodologiesA Comprehensive Survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol 14, No. 9, (September 1992), pp. 869885, ISSN 01628828 Li, Y.; Zheng, Y.; Doermann, D. & Jaeger, S. (2008). ScriptIndependent Text Line Segmentation in Freestyle Handwritten Documents, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.30, No.8, (August 2008), pp. 13131329, ISSN 01628828 LikformanSulem, L.; Zahour, A. & Taconet, B. (2007). Textline Segmentation of Historical Documents: a Survey, International Journal on Document Analysis and Recognition, Vol. 9, No. 2, (April 2007), pp. 123–138, ISSN:14332833 Lindeberg, T. & Eklundh, J.O. (1992). ScaleSpace Primal Sketch: construction and experiments, Image and Vision Computing, Vol. 10, No.1, (JanuaryFebruary 1992), pp. 3–18, doi 10.1016/02628856(92)90079I Louloudis, G.; Gatos, B. & Halatsis, C. (2008). Text line detection in handwritten documents, Pattern Recognition, Vol.41, No.12, (December 2008), pp. 37583772, ISSN 00313203
Segmentation of Handwritten Document Images into Text Lines
239
Manmatha, R. & Rothfeder, J.L. (2005). A Scale Space Approach for Automatically Segmenting Words from Historical Handwritten Documents. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 27, No.8, (August 2005), pp. 12121225, ISSN 01628828 Mao, S.; Rosenfeld, A. & Kanungo, T. (2003). Document Structure Analysis Algorithms: A Literature Survey, Proceedings of Document Recognition and Retrieval X SPIE, Vol. 5010, pp. 197207, ISBN 0819448109, Santa Clara, California, USA, January 2223, 2003 Marti, U.V. & Bunke, H. (2002). The IAMDatabase: an English sentence database for offline handwriting recognition, International Journal on Document Analysis and Recognition, Vol.5, No. 1, (), pp. 3946, DOI: 10.1007/s100320200071 Nagy, G. (2000). Twenty Years of Document Image Analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, No. 1, (January 2000), pp. 38–62, ISSN 01628828 Niblack, W. (1986). An Introduction to Digital Image Processing, pp. 115116, Prentice Hall, ISBN13: 9780134806747 Nicolas, S.; Paquet, T. & Heutte, L. (2004a). Enriching Historical Manuscripts: The Bovary Project, In: Document Analysis Systems VI, LNCS 3163, pp. 135146, Marinai, S. & Dengel, A. (Eds.) SpringerVergal, ISBN 3540230602, Berlin/Heidelberg Nicolas, S.; Paquet, T. & Heutte, L. (2004b). Text Line Segmentation in Handwritten Document Using a Production System, Proceedings of International Workshop on Frontiers in Handwriting Recognition, pp. 245–250, ISSN 15505235, Tokyo, Japan, October 2629, 2004 Nikolaou, A. & Gatos, B. (2009). Handwritten Text Line Segmentation by Shredding Text into its Lines, Proceedings of International Conference on Document Analysis and Recognition, pp. 13931397, ISSN 15205363, Barcelona, Spain, July 2629, 2009 O’Gorman, L. (1993). The document spectrum for page layout analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.15, No.11, (November 1993), pp. 1162– 1173, ISSN 01628828 Osher, S. & Fedkiw, R. (2003). Level Set Methods and Dynamic Implicit Surfaces, Springer, ISBN 9780387954820 Otsu, N. (1979). A Threshold Selection Method From GrayLevel Histograms. IEEE Transactions on Systems, Man and Cybernetics, Vol.9, No.1, (January 1979), pp. 6266, ISSN 00189472 Papavassiliou, V.; Stafylakis, T.; Katsouros, V. & Carayannis, G. (2010). Handwritten Document Image Segmentation into Text Lines and Words, Pattern Recognition, Vol.43, No.1, (January 2010), pp. 369377, ISSN 00313203 Plamondon, R. & Srihari, S.N. (2000). OnLine and OffLine Handwriting Recognition: A Comprehensive Survey, IEEE Transactions on Pattern Analysis Machine Intelligence, Vol.22, No.1, (January 2000), pp. 63–84, ISSN 01628828 Postl, W. (1988). Method for automatic correction of character skew in the acquisition of a text original in the form of digital scan results. Siemens AG, U.S. Patent 4,723,297, February 2, 1988 Razak, Z.; Zulkifee, K.; Idris, M.Y.I.; Tamil, E.M.; Noor, M.N.M.; Salleh, M.; Yaakob, M.; Yusof, Z.M. & Yaacob, M. (2008). Offline Handwriting Text Line Segmentation: A
240
Image Segmentation
Review, International Journal of Computer Science and Network Security, Vol. 8, No. 7, (July 2008), pp. 1220, http://paper.ijcsns.org/07_book/200807/20080703.pdf Santos, R.P.; Clemente, G.S.; Ren, T.I. & Calvalcanti, G.D.C. (2009). Text Line Segmentation Based on Morphology and Histogram Projection, Proceedings of International Conference on Document Analysis and Recognition, pp. 651655, ISBN 9781424445004, Barcelona, Spain, July 2629, 2009 Sauvola, J. & Pietikäinen, M. (2000). Adaptive document image binarization, Pattern Recognition, Vol. 33, No. 2, (February 2000), pp. 225236, doi:10.1016/S00313203(99)000552 Shi, Z. & Govindaraju, V. (2004). Line Separation for Complex Document Images Using Fuzzy Runlength, Proceedings of International Workshop on Document Image Analysis for Libraries, pp. 306312, ISBN : 076952088X, Palo Alto, USA, January 2324, 2004. Shi, Z.; Seltur, S. & Govindaraju, V. (2009). A Steerable Directional Local Profile Technique for Extraction of Arabic Text Lines, Proceedings of International Conference on Document Analysis and Recognition, pp. 176180, ISBN 9781424445004, Barcelona, Spain, July 2629, 2009 Simon, A.; Pret, J.C. & Johnson, A.P. (1997). A Fast Algorithm for BottomUp Document Layout Analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 19, No. 3, (March 1997) pp. 273277, ISSN 01628828 Soille, P. (2004). Morphological Image Analysis Principles and Applications, (2nd ed.), SpringerVerlag New York, Inc. Secaucus, NJ, USA, ISBN:3540429883 Tomai C. I.; Zhang B. & Govindaraju V. (2002), Transcript Mapping for Historic Handwritten Document Images, Proceedings of International Workshop on Frontiers in Handwriting Recognition IWFHR2002, pp. 413  418, ISBN 0769516920, Ontario, Canada, August 68, 2002. Vincent, L. & Soille, P. (1991). Watershed in Digital Spaces: An Efficient Algorithm Based on Immersion Simulations, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 13, No. 6, (June 1991) pp. 583598, ISSN 01628828 Weliwitage, C.; Harvey, A.L. & Jennings, A.B. (2005). Handwritten Document Offline Text Line Segmentation, Proceedings of Digital Image Computing: Techniques and Applications Conference (DICTA2005), pp. 27, ISBN: 0769524672, Cairns, Australia, December 68, 2005 Yanikoglu, B. & Sandon, P.A. (1998). Segmentation of Offline Cursive Handwriting Using Linear Programming, Pattern Recognition, Vol.31, No.12, (December 1998), pp. 18251833, doi:10.1016/S00313203(98)000818 Yin, F. & Liu, C.L. (2008). Handwritten Text Line Segmentation by Clustering with Distance Metric Learning, Proceedings of International Conference on Frontiers in Handwriting Recognition (ICFHR 2008), pp.229234, Montreal, Quebec, Canada, August 1921, 2008 Yin, F. & Liu, C.L. (2009). A Variational Bayes Method for Handwritten Text Line Segmentation, Proceedings of International Conference on Document Analysis and Recognition, pp. 436440, ISSN: 15205363, Barcelona, Spain, July 2629, 2009
12 IR Image Segmentation by Combining Genetic Algorithm and Multiscale Edge Detection Li Zhaohui1,2 and Chen Ming1
1Northwestern
2Chinese
Polytechnical University; Flight Test Establishmen China
1. Introduction The Canny operator applied in a multiscale edge detection for images is the most optimal step edge detection operator. However the Canny edge detection operator is optimal only for the step edge detection influenced by white noises because the Canny operator utilizes the first order derivative of a Gauss function to get a better compromise between noises restraint and edges detection. The Canny operator based edge detection finishes the implementation per the three criteria, namely, detecting (no losing important edges and no false edges), locating (with the minimal deviation between the real edges and the edges detected) and responding singly (reduce multiresponse into a single edge response). Because the Canny operator is based on a derivative operator (the first order derivative) to seek for the derivatives in two directions and their directions about image grey scale, it determines the maximum gradient and local maximum by means of nonmaximum restraint to the gradient. But the function with the maximum gradient and local maximum meeting with the above three standards limits is in fact the optimizing process of a multivariate function. Generally if an object function is continually differential, the space equation of a solution is simpler, and the general analysis is effective. However for the image segmentation with a complex configuration or sophisticated texture, it is very difficult to describe the multivariable object function satisfied with the above three standard limits by use of simple analysis functions because the image function itself is a multivariable nonlinear random function. So it has only an approximate and local sense to get the maximum gradient and local maximum of a function in traditional analysis methods. For multiscale based Canny operator edge detection, Canny once presented the algorithm to get a multiscale edge image based upon the above three standards for final edges composed, but it is related to the problem about the composition of different scale detections[1]. Jeong and Kim proposed a method for a single pixie to select local optimized scales by minimizing a scalespace energy function. Although this method is able to detect step edges exactly, delete false edges, restrain random noises relatively, and parry the problem mentioned above, the experiments showed that this method is not ideal compared with some fixed large scale methods. The reason is that the algorithm successive over relaxation (SOR) used for the solution of a minimizing energy function especially depends on initial given values resulting in not converging to the whole optimum solution[2,3]. Because there are some individual differences about homogenous properties and edges
242
Image Segmentation
vagueness between IR (infrared) images and general grey scale images, the former region segmentation mainly depends on the temperature field distribution of an equivalent black body for a target and the edge contour tends to blurring. It is more convenient to deal with the IR image edge detection by use of the individual fitness and mutation operation of the genetic algorithm. The individual fitness evaluation may be applied to the evaluation of a region segmentation with no segmentation reference. And the variation operation may be integrated with a kind of local contrast to design a dynamic variation operator as to measure the fuzziness of region edge information. This paper proposed a new multiscale edge detection method based upon an optimized genetic searching algorithm, and put it into the image segmentation of IR target signature.
2. Optimized genetic searching algorithm The genetic algorithm is introduced into an image segmentation based on edge detection. Firstly we determine the selection of a coding mode and a fitness function. Here according to the n cities’ universal rank as a genetic coding, we take the inverse of Hamilton ring length, Td, as a fitness function because there are always legal restraints of a hidden TSP problem in the initialization for possible solutions, crossover operation and mutation operation. f = 1 /(Td + α ⋅ N t )
(1)
where Nt is the measurement for TSP route illegality, α is a penalty coefficient. The application of crossover strategy: based upon the above TSP rank coding, the OX [4] method proposed by Davis is used in order to reduce the space searched. Firstly select a region to be matched:
A=9 8 4  5 6 7  1 3 2 0 B=8 7 1  2 3 0 9 5 4 6
(2)
In accordance with the relation mapped of a region to be matched, H is signed at the relative position outside the region, thus A′ = 9 8 4  5 6 7  1 H H H B′ = 8 H 1  2 3 0  9 H 4 H
(3)
Move the region matched to the initial position, preleave the space (H number) equal to the region matched afterwards, and then range other codes behind the region preleft in accordance with their relative rank, thus A′′ = 5 6 7 H H H 1 9 8 4 B′′ = 2 3 0 H H H 9 4 8 1
(4)
Finally interchange the father’s A, B regions, put them into A″, B″ regions preleft, and then get two descendant generations: A′′′ = 5 6 7  2 3 0  1 9 8 4 B′′′ = 2 3 0  5 6 7  9 4 8 1
(5)
IR Image Segmentation by Combining Genetic Algorithm and Multiscale Edge Detection
243
Path length (mile)
The mutation is an operation on backgrounds in the whole sense of genetic algorithms. For the TSP problem, the “Inverse mutation” operation is used in order to keep the individual of population varied so that there is a great change in possible solutions rank. Randomly select two points and inversely insert the subset between the points into the original positions. Because the above mutation operation for the TSP problem made the TSP ring length changed around adjustments and this change led to the finest adjustments, the local accuracy arrive at a better level. From the optimization of genetic algorithms the individual with a superior fitness has more opportunities to breed in a limited scale of population. The another feature of the genetic algorithm is not good enough to optimize locally. In practical applications GA generally converges to some possible solution which is not surely a optimized point as a whole or even not a local optimized point. To improve the deficiency of a basic genetic algorithm in local optimization and upgrade the quality of a solution as a whole, this paper proposed an improved hybridized algorithm SGA (Simple genetic algorithm)+SA (Simulated anneal)+TABU, in which SGA is integrated with a heuristic searching algorithm. The alternative optimization strategy applied is as follows, 1. Utilize a stochastic method to produce many different possible solutions for an initial possible solution population. 2. For one half of the individuals of possible solutions, execute TABU searching to get a local optimization solution. 3. For the other half of the individuals of possible solutions, execute SA searching to get a local optimization solution. 4. For the local solution from 2) and 3) steps, execute genetic selection and crossover operations. 5. Repeat 2), 3) and 4) steps until the objective condition of an algorithm is finally satisfied. Fig.1 shows the comparison between SGA+SA+TABU and SGA performances. Where the SGA optimization as a whole is better but poor at local. Relatively the SGA+SA+TABU algorithm intensifies the local ability of the simplex SGA. And compared with the simplex SA and simplex TABU, the SGA+SA+TABU algorithm extends the local optimization range of SA and TABU, and strengthens the whole optimization ability of simplex SA and simplex TABU. SGA+TABU SGA+SA0.63 SGA+SA0.9+TABU TABU SA0.63 SA0.9 OPTI
21.7 21.6 21.5 21.4 21.3 21.2 0
4
8 12 Time /s
16
Fig. 1. Performances comparison of optimization algorithms
20
244
Image Segmentation
3. Genetic algorithm + Canny multiscale edges detection Because an image segmentation is different from the simple optimization on the shortest route of n cities, the latter is only the construction of a closed route. But the former is about a population optimization among route groups composed of many closed and open routes. So it is necessary to combine the radiation signature of IR images when the cleavage and mergence are used to segment an image into regions in accordance with the homogeneity firstly and then execute edges detection based upon the Canny multiscale in each region, searching for the shortest route problem of n cities TSP composed of wavelet transformation module maximum values. The specific steps are as follows, 1. According to the equivalent black body temperature distribution of an IR radiation image, specify the initial regions conformed to homogeneity requirements. Supposing the maximum number of an image segmentation is n1, the individual chromosome coding is an integer sequence Ik={rii=1, 2, …, n1}. Where ri is a subregion order number and k is an individual number. Crossover operation: it is completed by use of the improved PMX crossover method. Here the gene codes in sequence coding may be repeated after finishing a crossover operation. Just based upon gene codes generate repeatedly, an image region is able to execute the further cleavage and mergence. If the region ri is merged with an adjacent region rj , Ik[i]=I, Ik[j]=i , the region rl merged with the region rj is also combined with the region ri , Ik[l]=i . So the gene code of the region combined with the region ri is i . Mutation operation: it is the design of a dynamic mutation operator in conjunction with a local contrast. The operator is used to measure the information vagueness degree of the region borders. The distance between the rejoin ri and its adjacent region rj is dij =
1 nij
nij
∑xijk − x jik 2
(6)
k =1
where xijk( j=1,2,…, ni; k=1,2,…,nij ) is the kth pixel on the border between the region ri and its adjacent region rj ; nij is the total pixels number of the border between the region ri and its adjacent region rj . The relative distance between the region ri and its adjacent region rj is ni
uij = dij /( ∑ dij )
(7)
j =1
Because uij is conformed to uij∈[0, 1], it may be used as a probability of the choice between the cleavage and mergence. According to the definition of a relative distance between regions, define the local contrast in any region, min j (uij ) ⎧ , min j ≠ max j ⎪min( max j (uij ) − min j (uij ) Ci = ⎨ ⎪ 1 , min j = max j ⎩
(8)
Individual fitness estimation: the region segmentation estimation, with no a segmentation reference, includes the region homogeneity measurement and edge vagueness measurement.
IR Image Segmentation by Combining Genetic Algorithm and Multiscale Edge Detection
•
245
Region homogeneity measurement The region homogeneity measurement is defined as follows AI i ≈
1 nk nl
nk nl
∑ ∑ ( gkl − μi )2
(9)
k =1 l =1
where gkl is the grey scale of the pixels in the region ri , μi is the average grey scale in the region ri . And the reason why the above equation is not an equality is that the region ri is not always a rectangle. • Edge vagueness measurement The integrated measurement definition for all region edges vagueness is as follows, uij (∇G)
E=
∑u
(i , j)
ij (∇G′)
(10)
where ∇G is the grey scale gradient of the pixels after cleavage and mergence. ∇G′ is the grey scale at the pixels in an initial region division. Because there exists always ∇G( i , j )≤∇G′(i , j ), so here is 0≤E≤1. 2.
For each region above divided take a Canny multiscale edge detection. Suppose the unit vectors n j (u) = (cos Af ( u , 2 j ), sin Af (u , 2 j )) and ∇( f ∗ θ 2 j )(u) are linear each other. At a 2j scale, the Mf(u, 2j) in the direction of u = v + λ n j ( v ) gets a local maximum
3.
4.
5.
at a point u=v，when λ becomes small enough. Such a point is the edge point in this region, and is also called the wavelet transformation module maximum point. Take the wavelet module maximum point from 2) as the universal point for n cities and constitute a TSP problem. The GA+SA+TABU is applied to the solution of an optimal route curve and to the formation of an optimal edge curve. Change the scale 2j and return to 2). Continue the edge detection and refind the n wavelet module maximum points as the universal points for cities, and then go to the next optimization step. Here is the measurement for a region uniformity computation. The uniformity measurement of an image segmentation denotes that the weighting sum for the uniformity measurement of all regions: G=
1 n ∑ wi ( pi )Gi n i =1
(11)
where n is the sum of segmented regions. wi(pi) is the weighting coefficient of a region area. pi is a region area. The computation of a weighting coefficient is in accordance with the following eq. ⎧2 pi2 / β 2 , α ≤ pi < 2 / β ⎪⎪ 2 2 wi ( pi ) = ⎨1 − 2( pi − β ) / β , 2 / β ≤ pi < β ⎪1 , pi ≥ β ⎪⎩ where α and β denote the minimum and maximum area respectively.
(12)
246
Image Segmentation
For image segmentation based on a genetic algorithm the multiplication of the equations (10) and (11) is applied to a composite individual fitness estimation. F=E·G
(13)
When the value got from eq. (13) is satisfied with the setting objective value the optimization searching ceases, otherwise returns to the 2) step for continuous optimization computations.
4. Experiment results and analysis In the course of a FLIR target image detection, take a target point source as the initial information of a potential target and determine an initial growing point of signature regions according to the sequence images inputed. The five threshold values of equal intervals[5] are employed in this paper, that is tmax＞t0＞t1＞t2＞tmin . where tmax is towards the maximum equivalent black body radiance temperature of a target image, tmin towards the minimum grey temperature scale of a target image. The first level of isothermal region is determined by tmaxt0 interval, and the second isothermal region tmaxt1 is determined by tmaxt1 interval. For the division of isothermal regions by use of a genetic algorithm, any grey scale interval in the above regions may be used as a seeding region for a target pixel seed till one of the following two conditions is satisfied. 1. the region grows large enough to reach the edge; 2. the equivalent black body radiance temperature or radiance intensity of a searching region degrades too much. The seed regions are defined as {tmaxt0, tmaxt1, tmaxt2, tmaxtmin} where {tmax, t0, t1, t2, tmin} is used, in a genetic algorithm, as five inner nodes of each segmentation region for construction of four layers of equivalent black body temperature differences. Fig.2(a) shows the original long wave FLIR image of a running truck, and Fig.2(b) and Fig.2(c) respectively illustrate the target IR image segmentation results by means of LaplaceGauss edge detection or wavelet nervenetwork algorithm[6]. Although the LaplaceGauss edge detection algorithm is able to locate target contours, a great deal of target details are lost. For the PCA analysis about a target region the wavelet nervenetwork algorithm embedded in a wavelet timefrequency analysis could keep the high frequency details of region contours. Fig.2(d) shows the image segmentation based on the genetic and wavelet multiscale edge detection. Compared with the former segmentation algorithms this detection algorithm could not only extract edges of a target clearly but also the edges succession behaves better because of the “nonmaximum restraint” for pixels gradient in course of optimization. From Fourier transformation frequency spectra Fig.3(a), Fig.3(b) and Fig.3(c) relative to Fig.2(b), Fig.2(c) and Fig.2(d) the effect of the LaplaceGauss edge detection[7] could be evaluated. Their energy is basically along the main axis and lower inside quadrants with a great loss of details information. The high energy of an image processed by a wavelet nervenetwork algorithm is distributed along the two axes from the original point, but the energy distribution inside each quadrant exists low state. So the number of little targets with bad pixels is available to be restrained. The energy spectrum gotten from the genetic and a wavelet multiscale edge detection is focused around the spectrum center and basically exists low value distribution with no mixture of medium value energy. This signature of the frequency spectrum distribution has an instant relationship with the segmentation structure
IR Image Segmentation by Combining Genetic Algorithm and Multiscale Edge Detection
247
of an image because the latter algorithm pays more attention to the edges segmentation inside image textures. As is showed in Fig.4 the Pratt quality factor is used as the evaluation of edges detection. Under a low signalnoise ratio the Pratt quality factor of a genetic and wavelet multiscale edge detection algorithm is much more superior than other two detection algorithms, and more suitable to an image with complex texture and fine construction.
(a) Original image
(b) LaplaceGauss edge detection
(c) WaveletNeroNET (d) Genetic+multiscale edge detection
Fig. 2. Experimental results 8
16
4
6
12
2
4
8
0
2
2
0
(a) Spectrum to Fig.2(b)
(b) Spectrum to Fig.2(c)
6
4
(c) Spectrum to Fig.2(d)
Fig. 3. Fourier transform spectrum LaplaceGauss WaveletNNet GeneticMultiscale
0.70 0.60 0.50 0.40 0.30 0.20 0.10 0
10
20
30
40
50%SNR
Fig. 4. Pratt qualities for different algorithms
5. Conclusions The image segmentation based upon a genetic and wavelet multiscale edge detection is very much suitable for the contour extraction of a dynamic IR target in clutters. It could restrain clutter background signals of a target image while reserving its high frequency signature. From the view of energy spectra the energy spectra of an IR target image after a
248
Image Segmentation
genetic and wavelet multiscale edge detection procession could be expressed according to the IR radiance energy levels of the target so as to avoid the loss of fine details (high frequency contents) inside the target image while its image edge is extracted.
6. References [1] Tan H L, Gelfand S B. A cost minimization approach to edge detection using simulated annealing [J]. IEEE Transaction on Pattern Analysis and Machine Intelligence (S01628828), 1992, 14(1): 318. [2] Jeong H, Kim C I. Adaptive determination of filter scales for edge detection [J]. Pattern Analysis and Machine Intelligence (S01628828), 1992, 14(5): 579585. [3] ZHONG Fan, ZHOU Jiliu, LANG Fangnian, et al. Adaptive Scale Filtering for Edge Detection [J]. Acta Automatica Sinica, 2007, 33(8): 867870. [4] WANG Xiaotong, DAVIS Lawrence, FU Chunsheng. Genetic Algorithms And Finegrained Topologies For Optimization [C]// Proceedings of the Genetic and Evolutionary Computation Conference, New York, USA, July 913, 2002: 981988. [5] Correia B, Dinis J, Davies R. Automatic Detection and Recognition of Stationary Motorized Vehicles in Infrared Images [J]. Proc. of SPIE(S0277786X), 1999, 3718: 140151. [6] LI Zhaohui, CHEN Ming. FLIR image segmentation based on wavelet neural networks with adaptive learning [J]. Journal of Computer Applications, 2005, 25(8): 17601763. [7] Greenspan H, Anderson C H, Akber S. Image Enhancement by Nonlinear Extrapolation in Frequency Space [J]. IEEE Transactions on Image Processing(S10577149), 2000, 9(6): 10351048.
13 Segmentation of Remotely Sensed Imagery: Moving from Sharp Objects to Fuzzy Regions Ivan Lizarazo1 and Paul Elsner2
1 Universidad
Distrital Francisco Jose de Caldas College, University of London 1 Colombia 2 United Kingdom
2 Birkbeck
1. Introduction The increasing availability of remotely sensed imagery data with high spatial resolution is demanding more efﬁcient, and more accurate methods for image analysis. Traditional perpixel classiﬁers focus on analysing spectral characteristics of single pixels and ignore contextual properties from surrounding pixels (Townshend et al., 2000). This inability to integrate neighbourhood information triggered the need of developing image analysis algorithms able to go ‘beyond pixels’ and take also into account spatial information (Jensen, 2005). New approaches for land cover characterization from high spatial resolution images include objectbased image analysis (OBIA), also referred to as geospatial object based image analysis (GEOBIA) (Blaschke, 2010). OBIA uses segmentation techniques to group pixels into discrete image objects as a ﬁrst stage of the image analysis process. By analysing image objects rather than individual pixels, it is then possible to include spatial and/or textural properties of image objects into the process (Blaschke et al., 2006). Recent research demonstrates that this segmentation based approach for image classiﬁcation produces higher thematic accuracy than the traditional perpixel methods (Blaschke et al., 2006; Lang et al., 2006; Platt & Rapoza, 2008; Thomas et al., 2003). The OBIA approach is based on the assumption that image objects produced by segmentation can be unambiguously linked to the geographic objects of interest and has proven valuable in a number of applications, see, for example, Shackelford & Davis (2003); Wei et al. (2005); Zhou et al. (2007). However, it can not be considered to be a silver bullet: when classes overlap spectrally, high classiﬁcation accuracy is still difﬁcult to achieve (Platt & Rapoza, 2008). The aim of creating meaningful image objects may be affected by blurring and distortion problems inherent to the image acquisition process (Bezdek et al., 1999). A central stage of OBIA is the segmentation stage. Standard image segmentation usually requires considerable parameterisation effort to ﬁnd the right sizes and homogeneity criteria that produce meaningful image objects for a given scene and application. In many situations, image segmentation becomes a time consuming task which requires iterative processing, and may not always succeed (Lang et al., 2006; Schiewe et al., 2001a). Depending on the complexity of the landscape, the quality of the image and the parameterisation procedure, image segmentation may produce image objects that represent realworld objects, part of
250
Image Segmentation
objects or just noise. Such image segmentation approach is a subjective and user driven task which esentially prevents automated processing of large data sets. The traditional image segmentation looks for delineating discrete image objects with sharp boundaries. The underlying assumption of this hard segmentation is that it is always possible to determine spatial boundaries between land cover classes. However, many geographic objects, both natural and man made, may not appear clearly bounded in remotely sensed images. Therefore, a fuzzy segmentation approach which takes into account the fuzziness of the real world and the ambiguity of remote sensing imagery is potentially more appropriate than a hard segmentation to resolve the spectral and spatial confusion which characterizes urban landscapes. Over the last decades, geographic information systems (GIS) researchers have applied fuzzy concepts to deal with the vagueness and imprecision widespread among geographic objects (Burrough, 1989; Burrough & Frank, 1996). Remote sensing researchers have applied perpixel fuzzy classiﬁcation to study geographic phenomena that are continuous and lack sharp boundaries (Fisher & Pathirana, 1990; Foody, 1992; Wang, 1990). However, fuzzy concepts have not been applied yet in the segmentation process for environmental remote sensing image analysis. As remote sensing applications include a wide spectrum of geographic phenomena ranging from discrete objects to continuous ﬁelds, a fuzzy segmentation approach can be potentially useful for land cover characterization on natural and manmade landscapes. This chapter proposes a method for land cover characterization based on fuzzy image segmentation. It argues that, in order to handle uncertainty of real world landscapes, image segmentation should output fuzzy image regions rather than sharp image objects. The chapter has been structured in two parts. In the ﬁrst part, basic principles and assumptions of OBIA are explained. It is highlighted that a successful application of OBIA depends on the quality of the discrete image objects produced at the segmentation stage. In the second part, a fuzzy image segmentation approach is proposed in more detail.
2. Traditional Objectbased image classification A basic assumption underlying traditional objectbased image classiﬁcation is that it is always possible to identify groups of pixels that can be related to geographic objects. These groups of pixels with meaning in the real world are referred to as image objects (Schneider & Steinwender, 1999). Thus, image objects are basic entities, composed of similar digital values, and possessing intrinsic sizes,shapes and geographic relationships with the realworld scene they model (Hay et al., 2001). The standard objectbased image analysis (OBIA) approach for image classiﬁcation can be represented as shown in Figure 1 using a three stage workﬂow (Benz, 2001): 1. image segmentation which creates meaningful image objects; 2. feature analysis which measures spectral, spatial, and contextual attributes of image objects; and 3. classiﬁcation which allocates image objects to target classes Image segmentation, the ﬁrst stage, outputs image objects that hopefully represent, completely or partially, the structural properties of the geographic objects under study. Feature analysis, the second stage, aims to select a set of attributes (also referred to as a feature vector) able to differentiate the classes of interest (i.e. the target classes). This stage requires users to establish correspondence between the image objects and the realworld objects (classes), and also to determine which properties can be relevant for the problem under
Segmentation of Remotely Sensed Imagery: Moving from Sharp Objects to Fuzzy Regions
251
Pixels
1 Image Segmentation
ImageObjects
2 Feature Analysis
Knowledge Input
Feature Vector
3 Classification
Land cover classes
Fig. 1. Flowchart of the OBIA image classiﬁcation process. study. Classiﬁcation, the last stage, assigns classes to image objects applying appropriate classiﬁcation rules which can be set up automatically or manually. This classiﬁcation may be conducted using conventional (e.g. maximum likelihood or minimum distance to mean) or nonconventional techniques (e.g. neural networks, fuzzy logic, nearest neighbor) (Gao, 2009; Jensen, 2005). The ﬁnal outcome of the process are digital objects representing the categories of interest. The OBIA process is iterative rather than linear and strictly subsequent, and knowledge input may occur at any stage of the process (Lang, 2008). 2.1 Image segmentation
Image segmentation is recognized as the critical step of the OBIA approach because its results affect directly all the subsequent tasks (Carleer et al., 2005; Neubert et al., 2006; Rekik et al., 2007). Image segmentation aims to partition the image into a set of regions which are distinct and uniform with respect to some intrinsic property of the image, such as grey level, texture or color (Freixenet et al., 2002). The selection of an appropriate segmentation technique depends greatly on the type of data being analyzed and on the application area (Pal & Pal, 1993). A successful segmentation produces image objects which can be unambiguously linked with ground objects. Typically, the segmentation stage is not concerned with the thematic identity
252
Image Segmentation
of the image objects as they are class labelled later in the classiﬁcation stage (Rosenfeld & Kak, 1992). The result of image segmentation is determined by the purpose of the study, the grouping strategy and speciﬁc decisions about the desired levels of homogeneity and (or) the expected size of the resulting segments (Pal & Pal, 1993). Image segmentation is therefore a highly subjective task which has to be adapted to the speciﬁc interest of users and the spatial and spectral constraints of the available images (Bock & Lessing, 2002). This poses signiﬁcant problems for the transferability of image segmentation methods and the development of automated classiﬁcation rules (Bock & Lessing, 2002). Traditional image segmentation can be referred to as crisp or hard segmentation because it produces image objects with clearly deﬁned boundaries. In a crisp segmentation, it is assumed that the image can be partitioned into spatially continuous, disjoint and homogeneous groups of pixels which represent the relative homogeneity of geographic phenomena (Blaschke et al., 2006). However, realworld landscapes are comprised by objects of different sizes which may exhibit internal heterogeneity (Herold et al., 2003). As single scale segmentations may not sufﬁce to capture such complexity, it has been suggested that image segmentation be conducted in a nested hierarchy of scales modeling relationships between subobjects, objects and superobjects (Burnett & Blaschke, 2003). While it is an appealing approach, it can bring further complications to the classiﬁcation stage due to the additional need for userdriven individual parameterization. 2.1.1 Image segmentation in computer vision
Computer vision is concerned with the theory for building artiﬁcial systems that obtain information from images. The image data can take many forms, such as a video sequence, views from multiple cameras, or multidimensional data from medical scanners (Azad et al., 2008). As opposed to multispectral remote sensing, computer vision deals mainly with single band data sets, i.e. greyscale images. Image segmentation is an active ﬁeld of research in computer vision and ‘hundreds of segmentation algorithms have been proposed in the last 30 years’ (Freixenet et al., 2002) (p. 408). This proliferation of algorithms shows how elusive a satisfactory segmentation is. There is an almost inﬁnite number of ways an image can be subdivided, they are all technically correct, but most of them are not what users want (Shi & Malik, 2000). Comprehensive reviews of image segmentation techniques have been published in the last two decades in the computer vision literature, including, for example Freixenet et al. (2002); Haralick & Schapiro (1992); Pal & Pal (1993). In computer vision, image segmentation is the process of extracting objects from background or separating the image into several regions which are considered to be coherent and meaningful (Pal & Pal, 1993). Generally, image segmentation is a process of clustering pixels in an image based on some rules, e.g. pixels with similar attributes should be together (Pal & Pal, 1993). In general terms, image segmentation techniques may be divided into two basic categories: edgebased and regionbased (Pal & Pal, 1993). Edgebased segmentation methods look for detecting image discontinuities. In this category, the assumption is that boundaries of regions are sufﬁciently different from each other and from the background to allow boundary detection based on local discontinuities in grey level intensity. On the another hand, regionbased segmentation methods are based on partitioning an image into regions that are similar according to a set of predeﬁned criteria such as grey level, shape or texture.
Segmentation of Remotely Sensed Imagery: Moving from Sharp Objects to Fuzzy Regions
253
Two common approaches for regionbased segmentation are either merging individual image objects (bottomup approach) or recursively splitting regions starting from the whole image (topdown approach) (Pal & Pal, 1993). Although the edge and region approaches are considered to yield good results, some researchers have argued the impossibility of extracting complete information of either aspect of the image independent of the other and have proposed hybrid approaches which integrate boundary and region information (Moigne & Tilton, 1995; Pal & Pal, 1993). One example of these approaches is the watershed algorithm which sees an image as a topographic surface and, after identifying maxima (ridges) and minima (valleys), attempts ‘ﬂooding’ the terrain to obtain catchment regions (Dam, 2000). 2.1.2 Image segmentation in remote sensing
Image segmentation is a relatively recent ﬁeld of research in environmental remote sensing, despite the fact that it has been applied extensively in neighboring disciplines such as computer vision and medical imaging (Carleer et al., 2005) . The main problem was the inability of computer vision algorithms to process color or multiband image data sets. However, in the last decade, the remote sensing community has adapted and enhanced the computer vision segmentation approach for dealing with multispectral data sets. A number of operational software tools are now available to conduct segmentationbased image classiﬁcation for remote sensing. This includes Deﬁniens Developer, previously known as eCognition, ENVI’s Feature Analysis, ERDAS’s Objective, and IDRISI’s Segmentation. Although all of these programs offer a number of similar functionalities, Deﬁniens Developer implements the most advanced algorithms and is a popular choice in OBIA applications (Blaschke et al., 2006) and will therefore be reviewed as example for current standard ’crisp’ OBIA implementations. Deﬁniens implements the fractal net evolution algorithm which provides multiresolution segmentation capabilities ‘applicable and adaptable to many problems and data types’ (Baatz & Schape, 2000) (p. 1). Multiresolution segmentation is a bottom up, regionmerging technique which starts building onepixel image objects which grow by merging adjacent objects based on heterogeneity criteria (Yan et al., 2006). These objects may be extracted from the image in a number of hierarchical levels and each subsequent level yields image objects of a larger size by combining objects from a level below, which represents information on different scales simultaneously (Baatz & Schape, 2000). Image object heterogeneity can be spectral heterogeneity, hspectral, or shape heterogeneity, hshape (Baatz & Schape, 2000). Spectral heterogeneity is a function of userassigned layer weights, number of pixels comprising the objects, and standard deviation of pixel values within each layer. Shape heterogeneity is based upon the change in object shape before and after an eventual merge. Object shape is described in two ways: (i) compactness and (ii) smoothness. Compactness C is measured as indicated in Equation 1: C = sn ∗ ln /bn
(1)
where sn is the size of each image object, ln is the perimeter of the image object and bn is the perimeter of a minimum box bounding each image object. Smoothness S is measured as indicated in Equation 2:
√ S = sn ∗ ln / sn
(2)
254
Image Segmentation
Spectral and shape heterogeneity are summarized by a single fusion value ( f ), which indicates the potential merge between two image objects given by Equation 3 (Zhang, 2006): f = (1 − w) ∗ hspectral − w ∗ hshape
(3)
where w is the weight associated with shape heterogeneity (Deﬁniens, 2007). Throughout a single segmentation step, the underlying optimization procedure minimizes the heterogeneity of resulting image objects weighted by their size. A segmentation step is ﬁnished when every original image object is assigned to the optimal higher level image object. To achieve adjacent image objects of similar size and thus of comparable quality, the procedure simulates the growth of objects over a scene in each step and also for the ﬁnal result (Yan et al., 2006). Other alternatives to multiresolution segmentation are provided by Deﬁniens. The chessboard algorithm produces a regular grid of segments with a predeﬁned size. The quadtree segmentation splits an image domain into squares, and then into smaller squares, until the spectral heterogeneity of every image object falls below a userdeﬁned threshold. The spectral difference segmentation merges adjacent image objects that do no exceed a userdeﬁned threshold for a weighted spectral difference (Esch et al., 2008). An approach to address the problems associated with multilevel segmentation is the move from multilevel to singlelevel segmentation approaches (Corcoran & Winstanley, 2006). This approach is advantageous due to two main reasons. First, it looks for conducting segmentation in a way closer to human visual perception. Second, it helps to solve practical problems for evaluation of multilevel segmentation such as complex hierarchical classiﬁcation schemas and complicated quality evaluation procedures. In fact, it has been argued that a robust set of intensity and texture features could be extracted and integrated to represent urban land cover in a "true form" with just one level segmentation (Corcoran & Winstanley, 2006). In summary, the traditional approach for image segmentation is problematic due to the following reasons: (i) remotely sensed images portray an ambiguous representation of geographic objects which often prevents the formation of meaningful image objects; (ii) the current implementations of such segmentation are highly dependent on complicated parameterisation procedures which are both labour intensive and time consuming; (iii) linking image objects to real world classes, and identifying appropriate attributes, are not trivial tasks, and usually require a trial and error approach. This means that the effectiveness of crisp segmentation for land cover classiﬁcation is at least partly depending on effort and skill of the individual user. 2.1.3 Segmentation quality
Image segmentation can be seen as an improvement of the analysis process of remotely sensed imagery. It provides an alternative means to conduct image classiﬁcation. However, it has been observed that objectbased image classiﬁcation also has limitations (Song et al., 2005): • classiﬁcation accuracy depends on the quality of the image segmentation (i.e. if objects are extracted inaccurately, subsequent classiﬁcation accuracy will not improve); • classiﬁcation error could be accumulated due to error in both image segmentation and classiﬁcation process; and • once an object is missegmented, all pixels in this object will be misclassiﬁed.
Segmentation of Remotely Sensed Imagery: Moving from Sharp Objects to Fuzzy Regions
255
Image segmentation is therefore the critical stage in OBIA, but it is also viewed as an illposed problem in the sense that it has no unique solution: a minor change of the homogeneity measure leads to different segmentation outcome (Hay & Castilla, 2008). However, not all segmentation methods are good enough for a particular type of images and users of each application have to evaluate the quality of the image output from a given segmentation algorithm. Thus, the problem is to ensure that the evaluation of the segmentation results is an objective process. A common approach, in computer vision applications, is to create a vector of comparison measures between the segmented image and a ’ground truth’ segmentation (Pal & Pal, 1993). A simple way to evaluate the quality of segmentation, provided that such ground truth segmentation is available, is to use an overlapping area matrix (OAM) and the following metrics (Ortiz & Oliver, 2006): • percentage of correctly grouped pixels (CG); • percentage of oversegmentation (OS); and • percentage of undersegmentation (US). In the ﬁeld of environmental remote sensing, similar concepts have been proposed to evaluate segmentation quality. A good segmentation is achieved when the overall differences between the segmentation results and the associated reference objects are as low as possible (Meinel & Neubert, 2004; Neubert et al., 2006). In general, the quality of image segmentation may be evaluated using both qualitative and quantitative methods. Qualitative measures are visual evaluations of general criteria such as the delineation of varying land cover types, the segmentation of linear objects, or the ocurrence of faulty segmentation. Quantitative measures make a comparison between clearly deﬁned reference areas (varying in location, form, texture, contrast, land cover type) and segmentation √ results using geometric properties such as area Ai , perimeter Pi , and shape index Si = Pi /4 Ai of the image object i (Meinel & Neubert, 2004; Neubert et al., 2006). As presented, all of the methods proposed for quantitative evaluation of segmentation quality of remotely sensed images rely on a reference segmentation. A main issue is how to deﬁne such ground truth or reference segmentation. Some researchers have suggested adopting a library of reference images with their corresponding segmentation reference image, as computer vision community does, in order to evaluate different segmentation algorithms (Corcoran & Winstanley, 2006). Using that approach, a group of researchers have evaluated the performance of different segmentation algorithms using a common set of remotely sensed images and measures (Neubert et al., 2008). However, this initiative is just a ﬁrst step on the path to well established criteria and methods for judging the quality of segmentation for remote sensing applications. For practical purposes, users need to build the reference segmentation suitable for their own applications by visual interpretation. In this case, expert humans partition the image into spectrally homogeneous regions, adjust them to accommodate spatial or contextual characteristics and, then, use that ideal segmentation to evaluate segmentation results. A problem with this approach is that the whole process of image classiﬁcation becomes more dependent on visual interpretation and harder to automate. Segmentation evaluation becomes more complex with multiscale segmentation which produces a number of segments at different scales. It seems impractical to deﬁne an ideal reference segmentation for each scale. A sensible approach in this case would be a qualitative evaluation through visual inspection of each segmentation level. However, a subjectivity
256
Image Segmentation
problem may arise with visual evaluations. This problem suggests that, while multiscale segmentation seems valuable to capture both the coarse and the ﬁne scales of real world objects (Benz, 2001), it may demand intensive user interaction to validate the consistency of the different segmentation levels. An additional issue could emerge as a byproduct of changing the classiﬁcation paradigm from pixels to image objects. Although quality assessment of pixelbased classes can be done without many problems using proven techniques based on the well known misclassiﬁcation or error matrix, it could be not entirely right for objects obtained using the objectbased approach. It could be necessary to build a new framework for assessing the quality of ground objects derived from remotely sensed data. In that direction, Zhan et al. (2005) have proposed adopting a socalled perobject method of quality assessment able to measure spatial or contextual attributes of the objectclasses not just the label assigned to each pixel. In summary, the objective evaluation of segmentation quality remains problematic. As an example, a detailed evaluation of four different segmentation algorithms (i.e. optimal edge detector, watershed segmentation, multilevel thresholding, and fractal net evolution) concluded that the ‘miraculous segmentation’ method which segments in a correct way for all types of landscape does not exist (Carleer et al., 2005). 2.1.4 Uncertainty in image segmentation
A useful model for understanding uncertainty in image analysis is the image chain approach which considers the remote sensing process as a chain linking subsequent stages from image collection to the ﬁnal thematic mapping (Schott, 1997). In such a view, image analysis is only as strong as the weakest link in the chain and limitations existing at each step affect the entire process (Woodcock, 2002). In the case of objectbased image analysis, image segmentation is not only the weakest link in the chain but also the earliest one. Understanding uncertainties associated with the image classiﬁcation process may be useful to propose alternative ways for conducting image segmentation. Atkinson & Foody (2002) suggested dividing uncertainty between ambiguity and vagueness. Ambiguity is the uncertainty associated with crisp sets, for example, when hard classiﬁcation is conducted to allocate pixels to one of several possible land cover classes. Vagueness, on the other hand, is expressed by the degree of incompleteness of land cover classiﬁcation schemas or the eventual fuzziness needed to deal with borderline cases (e.g. the dividing line between a sprawling shrub and a woody vine which is indeﬁnite). Such ambiguity intensiﬁes in urban areas, where mixed pixels occur in images with different spatial resolution (Herold et al., 2003; Mesev, 2003). This ambiguity is due to the complex composition of urban landscapes where the same materials may be present in different land cover classes, as for example in roads and rooftops made of asphalt and concrete. In such cases, any attempt to produce crisp image objects representing one or another land cover class can fail easily. As a simple illustration, Figure 2 shows eight crisp segmentations obtained from a high spatial resolution image over an urban area. In this example, image segmentation was conducted using a watershed segmentation algorithm which needs three parameters as input: (i) denoising factor (conductance of an anisotropic diffusion ﬁlter), (ii) scale (size of the image objects), and (iii) dissimilarity criteria (threshold value to merge the image objects). A visual assessment of the resulting image segmentations suggests that none of these single level segmentations produces meaningful image objects. Morever, it demonstrates that parameterisation of a meaningful segmentation is really a complicated and hard task. This
Segmentation of Remotely Sensed Imagery: Moving from Sharp Objects to Fuzzy Regions
257
Fig. 2. Examples of crisp image objects produced by a hard image segmentation of a multispectral image using the watershed algorithm and different combinations of parameters example illustrates that, very often, a discrete image segmentation does not help to resolve the ambiguity present in urban image data sets. On the other side of uncertainty is vagueness, a component which refers to the application of fuzzy sets concepts, for example, when fuzzy classiﬁcation is used to establish degrees of membership to natural land cover that varies continuously (Atkinson & Foody, 2002). This topic is reviewed in the next section. 2.2 Fuzzy sets and image classification
Fuzzy classiﬁcation attempts to address the fact that geographic information is imprecise, meaning that the boundaries between different phenomena are fuzzy, or there is heterogeneity within a class, perhaps due to physical differences. A fuzzy classiﬁcation takes into account that there are pixels of mixed composition, that is, a pixel cannot be deﬁnitively assigned to one or another category (Fisher & Pathirana, 1990; Wang, 1990). A hard classiﬁcation algorithm is based on classic set theory, which requires precisely deﬁned set boundaries for which an element (i.e. a pixel) is either a member (true = 1) or not a member (false = 0) of a given class. In contrast, a fuzzy classiﬁcation allows degrees of membership of image objects to each of the different classes. A main advantage of fuzzy classiﬁers is that they provide a more informative and potentially more accurate alternative to hard classiﬁcation (Atkinson & Foody, 2002). Thus, a fuzzy classiﬁer is a realistic way to take into account the ambiguity problem described in the previous section. Fuzzy approaches for perpixel image classiﬁcation have been proposed since 1990 (Foody, 1996; Wang, 1990; Zhang & Foody, 1998). It has been argued that, because of mixed pixels, remotely sensed images provide an ambiguous representation of the landscape (Fisher, 1997). As a consequence, identiﬁcation of thematic classes from images can not be achieved with a high level of certainty (Molenaar & Cheng, 2000). Hence, a hard classiﬁcation of images into welldeﬁned classes has been evaluated as inappropriate and inaccurate (Doan & Foody, 2007). A fuzzy or soft image classiﬁcation can be obtained by using either a perpixel or a perﬁeld approach (i.e. using existing boundary data) (Aplin & Atkinson, 2001; Zhang &
258
Image Segmentation
Stuart, 2001). A fuzzy classiﬁcation is especially useful for analyzing and detecting changes in complex landscapes (Burrough & Frank, 1996; Fisher et al., 2006). Pixelbased fuzzy classiﬁcation is typically conducted using three main stages: 1. fuzziﬁcation, in which membership functions to classes are established for a number of features, 2. fuzzy rules deﬁnition, in which different membership functions are combined into a rule set base and used for classiﬁcation, and 3. defuzziﬁcation, in which the fuzzy classiﬁcation result is discretized to produce the eventual crisp allocation/classsiﬁcation.
Fig. 3. Crisp (rectangular) and fuzzy (i.e. triangular and normal) membership functions for feature z. Fuzziﬁcation is a term used to describe the transition from crisp (classic) sets to fuzzy sets. In a fuzzy system, every class is represented as a fuzzy set and memberships take values anywhere in the range [0, 1] (Zadeh, 1965). Figure 3 shows that , for a particular feature z, the membership value μ to a given class can be deﬁned by either a crisp membership function as the rectangular function or by a fuzzy membership function such as the triangular or the bellshaped functions. In a crisp function, such as the rectangular one, the membership value μ is full (1.0) if b1 ≤ z ≤ b2 , or null (0.0), otherwise. In a fuzzy function, such as the triangular in (a) or the normal in (b), the membership value μ can be any value between 1.0 and 0.0, depending on the value of z and the speciﬁc nature of the selected function. The second stage is the deﬁnition of fuzzy rules, i.e. if  then rules in which, if a condition is fulﬁlled, an action takes place. Referring to Figure 4 the following rules apply: If the value of feature z is below 25, then the pixel has a full membership to a given land cover class (e.g. class A). If the value of feature z is 50, then the pixel has 0.5 membership to class A and 0.5 membership to class B. A fuzzy rule base produces a fuzzy classiﬁcation which consists of discrete return values of membership of pixels for each of the output classes. In defuzziﬁcation, the last stage of the process, every pixel is assigned to one of the classes of interest. A common way of defuzziﬁcation is to assign each pixel to the class with the highest membership degree value. If this value is below a certain threshold (e.g. less than 0.50), no classiﬁcation is performed to ensure a minimum reliability of such allocation.
Segmentation of Remotely Sensed Imagery: Moving from Sharp Objects to Fuzzy Regions
259
Fig. 4. The fuzzy membership functions for feature z deﬁne the fuzzy set L (low), M (medium), and H (high) of membership to classes A, B and C. It should be noted that the use of fuzzy sets techniques in the context of conventional OBIA is not new. However, fuzziness has not been considered at the segmentation stage but later in the process chain, that is, at the classiﬁcation stage. In this stage, the richness of fuzzy sets concepts is used to handle the uncertainty carried by sharp image objects and hopefully to increase the ﬁnal thematic accuracy (Benz et al., 2004). Fuzzy classiﬁcation is, however, the last stage of the process, and it may be too late to take into account uncertainty. As an example, Deﬁniens software provides capabilities for fuzzy classiﬁcation of image objects. However, in such implementation users need to manually select fuzzy membership functions and experiment with classiﬁcation rules. This particular setup of the defuzziﬁcation step leads to time consuming trial and error experiments which add signiﬁcant effort to the whole classiﬁcation process. Fuzzy sets concepts have been implemented in the segmentation stage of OBIA only very recently (Lizarazo & Elsner, 2009). This chapter links into this work and pursues the argument that a much more natural way for dealing with uncertainty is to apply fuzzy set concepts at the very beginning of the OBIA process, that is in the segmentation stage. By doing that, the earliest (and regularly weakest) link could be strengthened and therefore the entire classiﬁcation process be improved. In the next section , a general framework for objectbased image classiﬁcation based on fuzzy segmentation will be proposed.
3. Geographic objects and image objects In the previous section, ‘image objects’ were introduced as the core spatial units for conducting objectbased image analysis (OBIA). As segmentation, the OBIA’s critical stage, produces such image objects which aim to identify real world objects, it is important to review how geographic objects are conceptualised in digital environments. At its most basic level, geographic objects can be modeled as discrete objects or continuous ﬁelds, depending on a number of considerations, including purpose and scale of analysis (Couclelis, 1992; Goodchild, 1992; Goodchild et al., 2007). Discrete objects are nonoverlapping partitions of the geographic space which can be used to conceptualise sharply bounded objects like buildings, roads or real estate units. Continuous
260
Image Segmentation
ﬁelds are functions that map locations to a given property (attribute) which may serve to conceptualise spatially variant phenomena like elevation, soil type, agricultural suitability or ﬂood risk. For some applications, a given geographic object can be modeled simultaneously as being a ﬁeld and an object, e.g. an agricultural parcel can be conceptualised as an enclosed area of land whose soil acidity varies continuously (Bian, 2007; Goodchild et al., 2007). Thus, geographic objects can be represented using three main categories: (i) geoobjects for discrete objects, (ii) geoﬁelds for continuous ﬁelds, and (iii) ﬁeldobjects for spatial regions with internal heterogeneity (Goodchild et al., 2007). All categories can be derived from the concept of a geoatom which associates a point location in spacetime and a property. In formal language, a geoatom is a tuple x, Z, z( x ) where x deﬁnes a point in spacetime, Z identiﬁes a property and z( x ) deﬁnes the particular value of the property at that point. A geoobject is therefore an aggregation of points in spacetime whose geoatoms have uniform values for certain properties. A geoobject can have either natural (also known as bona ﬁde) boundaries or cognitioninduced (also known as ﬁat) boundaries (Goodchild et al., 2007). Examples of bona ﬁde boundaries are buildings, roads, or rivers. Examples of ﬁat boundaries are county borders or propertylines. In geoﬁelds, the values for a particular property of geoatoms are allowed to vary. A scalar geoﬁeld describes a single property such as elevation; a vector geoﬁeld describes both magnitude and orientation of continuous phenomena such as wind or temperature. A ﬁeldobject is a mixed object which describes the variation of a continuous property across an areal geoobject such as changes on biomass in a forest. In current GIS practice, geoobjects can be represented in the two dimensional space using points, lines and polygons. Geoﬁelds are commonly represented by raster grid cells or triangular irregular networks (Goodchild et al., 2007). Fieldobjects, however, are not very well represented yet in GIS systems (Bian, 2007). In the realm of OBIA, the segments output by standard ‘hard’ segmentation procedures can be related to geo objects. However, segmentation could be deﬁned in a broader way to be compatible with other types of geographic objects as well. Such a more generic view of the image segmentation process would allow it to produce image regions able to resemble both discrete geoobjects and continuous geoﬁelds. It is proposed here that image segmentation should be understood as the process of grouping pixels either spatially or thematically. In the ﬁrst case, spatial segments can be identiﬁed as crisp image objects which are discrete groups of pixels with clearly delimited boundaries but no thematic description. These are the geoobject type segments that are produced by current standard OBIA approaches. In the second case, segments can be conceptualised as image ﬁelds, whose pixels store individual values to a given property. Image ﬁelds are not single layer output such as the spatial segments of standard OBIA approaches. Rather, image ﬁelds consist of as many layers as properties of interest, e.g. target land cover classes, exist. A set of image ﬁelds representing respective membership to a given set of thematic categories and individual image ﬁelds can be understood as continuous image regions. Figure 5 illustrates both spatial and thematic image segmentation: (a) an example of spatial segmentation with four discrete image objects labelled as 1,2,3, and 4; (b) three layers of continuous image regions, identiﬁed as target land cover classes A, B, and C. In (c), the hypothetical thematic categories A, B, and C, are shown for reference. Typically, discrete image objects, or spatial segments, are crisply bounded, have individual identity but lack any thematic description. Crisp image objects are created using intrinsic homogeneity criteria which usually do not refer to thematic similarity. Thus, discrete image objects are simply crisp
261
Segmentation of Remotely Sensed Imagery: Moving from Sharp Objects to Fuzzy Regions
(c)
(b)
(a)
Fig. 5. Two different types of segments output by image segmentation: (a) discrete image objects with crisp boundaries but no thematic description; and (b) continuous image regions with indeterminate boundaries but thematic description. Three hypothetical thematic categories A, B and C are shown in (c) for reference. regions whose spatial and spectral properties need to be analysed at a later stage of the OBIA process to produce an eventual thematic allocation. Continuous image regions, or thematic segments, on the other hand, are a set of image ﬁelds with no spatial boundaries but holding a membership value to a given set of thematic categories. Thematic segments have no sharply deﬁned boundaries, thus they can be considered as fuzzy or vague regions. As illustrated in (b), there are as many layers of image regions as target classes exist. Each pixel in a image region holds membership values to every one of target classes A, B and C. A given pixel, for example the one to the top right corner, holds the following membership values: 0.7 to class A, 0.0 to class B, and 0.3 to class C (in this example, it is assumed that membership values add up to 1.0 which is not always the case). 3.1 Spatial image segmentation and sharp image objects
Spatial or crisp image segmentation that is undertaken in traditional OBIA analysis can formally be described as follows. Let R represent the entire geographic region occupied by an image. Spatial segmentation can be seen as a process that partitions R into n homogeneous subregions R1 , R2 , . . . , Rn such that: n
Ri = R.
(4)
is a connected set, i = 1, 2, . . . , n.
(5)
i =1
R
Ri ∩ R j = ∅ for all i and j, Pred( Ri ) = TRUE for Pred( Ri ∪ R j ) = FALSE
i = j.
(6)
i = 1, 2, . . . , n.
for any adjacent regions Ri
(7) and R j .
(8)
262
Image Segmentation
In equations 4 to 8, Pred( Rk ) is a logical predicate deﬁned over the points in set Rk , and ∅ is the null set. The symbols ∪ and ∩ represent set union and intersection, respectively. A region is a connected set of pixels. Two regions Ri and R j are considered adjacent if pixels lying on their boundaries are neighbours. A pixel may have either 4 neighbours (i.e. two horizontal and two vertical) or 8 neighbours (when the four diagonal neighbors are also considered). Equation 4 indicates that every pixel is allocated to a region. Equation 5 requires that pixels in a region be 4 or 8 connected. Equation 6 indicates that the regions must be disjoint (i.e. they have no pixel in common). Equation 7 indicates what properties must be satisﬁed by the pixels in a segmented region –for example, Pred( Ri ) = TRUE if all pixels in Ri have their intensity level within certain interval. Finally, equation 8 denotes that two adjacent regions Ri and R j must be different in the sense of predicate P.
Fig. 6. Spatial or crisp image segmentation produces one single image composed of sharp image objects with clearly deﬁned boundaries. In a ﬁeld representation of sharp image objects, as the one depicted to the right, pixels hold full or null membership to image objects. In a perfect segmentation, discrete image objects can be unequivocally associated to target land cover classes. Discrete image segmentation divides an image into a set of nonoverlapping regions or image objects whose union is the entire image (Haralick & Schapiro, 1992). Figure 6 illustrates that crisp image objects are discrete structures with clearly deﬁned boundaries which cover the totality of the imaged geographic area. Image objects are an aggregation of (raw or preprocessed) pixels whose digital values, located at one or several spectral bands, meet one or several homogeneity criteria deﬁned by predicate P. Pixels aggregated as image objects satisfy predicate P –which usually involves properties of single pixels but may also include constraints related to the segment under construction like minimum or maximum size. As outlined earlier, the deﬁnition of predicate P is a subjective decision which usually entails a trial and error parameterisation procedure in established OBIA workﬂows. However, Figure 6 demonstrates that such image objects can be expressed as well in the ﬁeld view of thematic segmentation. This is done as three image regions in which the membership of pixels to image objects 1 to 4 is expressed in the binary 0/1 membership to each individual region. In a perfect segmentation each image object 1, 2, 3, and 4 can be unequivocally linked
Segmentation of Remotely Sensed Imagery: Moving from Sharp Objects to Fuzzy Regions
263
to the target classes A, B, and C (as shown in Figure 5). This demonstrates that the spatial segments can also be conceptualised as extreme examples of thematic segmentation. 3.2 Fuzzy image segmentation and fuzzy image regions
Thematic segmentation offers the opportunity to widen the binary crisp membership attributes that are inherent in the spatial segmentation approach by allowing not just membership values of 0 or 1 but also any value in between, i.e. expressing fuzzy membership values. This is the central paradigm of the proposed fuzzy image segmentation approach. Figure 7 shows vague (or fuzzy) image regions output by a fuzzy segmentation. In this hypothetical example, the input image is a multispectral image composed of n spectral bands. The output of the fuzzy classiﬁcation is a set X composed of m image regions, where m is deﬁned by the number of target land cover classes of the OBIA process. In the example of Figure 7, Xm comprises of three land cover classes A, B, and C. Partial membership values μ ( Xm ) are represented by continuous values ranging from 0.0 (no membership) to 1.0 (full membership). Class
Fig. 7. Thematic or fuzzy image segmentation outputs fuzzy image regions holding multiple membership values to target thematic categories A, B, and C. Fuzzy image regions are continuous or thematic segments which take into account the inherent ambiguity of remotely sensed images. A fuzzy segmentation partitions the image R into overlapping image ﬁelds with indeterminate boundaries and uncertain thematic allocation. This means that equations 5, 6 and 8 that apply to discrete segmentation do not hold for continuous or thematic segmentation. Instead, there is no condition of spatial connectedness (Equation 5); each pixel belongs to all m fuzzy image regions with varying degrees of membership (Equation 6); and there is no condition of dissimilarity between regions ( Equation 8). In addition, membership values can be anywhere in the range from 0.0 to 1.0. The concept of a fuzzy image region can be further extended in order to allow pixels to store not only a single membership value per class (a scalar) but a multivalued membership per class (a vector). This extension may be useful to store, for example, minimum, mean and maximum estimated values of membership. This additional type of fuzzy image region can be referred to as a composed fuzzy image region. A composed image region can be seen as a
264
Image Segmentation
ﬁnite set of possible realizations of a fuzzy imageregion where each realization is the value inferred using one speciﬁc processing technique. Once fuzzy image regions have been established, there are two alternative approaches for conducting the subsequent OBIA classiﬁcation process: • direct discretization which transforms fuzzy image regions into discrete objects or classes skipping the feature analysis stage (i.e. using a similar approach to traditional pixelwise fuzzy classiﬁcation), or • objectoriented discretization which introduces a feature analysis stage in which fuzzy image regions properties are measured before proceeding to the defuzziﬁcation/classiﬁcation stage. 3.3 A general workflow for image classification
A general framework to conduct fuzzy segmentationbased image analysis is proposed in Figure 8. The image classiﬁcation process can be understood as the sequential development of three distinct and interrelated stages: (i) fuzzy segmentation in which n spectral bands are transformed into m fuzzy image regions, (ii) feature analysis in which properties of fuzzy image regions are measured to build a set of relevant features, and (iii) defuzziﬁcation (or classiﬁcation) in which fuzzy image regions are allocated to one of m land cover classes. 3.3.1 Fuzzy segmentation
In this stage, fuzzy image regions are created from raw or preprocessed pixels. As outlined in the previous section, fuzzy imageregions have membership values in the range [0, 1]. Such values represent degrees of belongingness of every pixel to the classes under study. Fuzzy segmentation can be understood here as a supervised regression task in which, training samples are used to infer membership values to classes for the whole image. Thus, following such a concept, any statistical technique able to ﬁt a supervised regression model may be used to produce fuzzy image regions. Once a set of membership greylevel images has been produced, there will be one fuzzy image region available for each target class. 3.3.2 Feature analysis
This stage aims to deﬁne, select and extract a relevant set of image regions’ properties and relationships suitable to be used as a feature vector to infer appropriate decision rules to resolve the spectral ambiguity of land cover classes. An example metric is the absolute normalized difference index (ANDI) deﬁned in Equation 9: ANDI =  μiA − μiB  /(μiA + μiB )
(9)
where μiA and μiB are the membership values of the i th pixel to the classes A and B, respectively. The ANDI value is an indicator of the overlap existing between two speciﬁc fuzzy image regions. ANDI values range in [0, 1]. ANDI values close to 0 represent areas of thematic confusion. Another metric is the sum of logarithms index (SOL) deﬁned in Equation 10: SOL = lnμiA + lnμiB
(10)
where ln is the natural logarithm. The SOL index measures the overlap between two fuzzy image regions A and B and highlights areas exhibiting high membership values to more than one target class.
Segmentation of Remotely Sensed Imagery: Moving from Sharp Objects to Fuzzy Regions
265
Fig. 8. A general workﬂow for image classiﬁcation based on fuzzy segmentation. 3.3.3 Defuzzification
This stage aims to infer (and apply) appropriate decision rules to assign either full membership from the fuzzy image regions to the target landcover classes (in the case of categorical qualitative analysis) or compositional values representing proportion of land cover classes at every pixel (in the case of quantitative analysis). For qualitative analysis, a common defuzziﬁcation technique used in fuzzy applications is the allocation using the maximum membership values (Wang, 1990), which has been applied in perpixel classiﬁcation in the last two decades. Other approaches include the centroid method, the weighted average method and the mean max membership method (Ross, 2004). However, all these established techniques do not exploit the potential richness of information carried by fuzzy image regions (and quantiﬁed in the feature analysis stage). Thus, in the approach proposed here, the process of defuzziﬁcation is addressed as a problem of supervised learning in which a variety of non parametric classiﬁcation algorithms could be applied.
266
Image Segmentation
For quantitative analysis, the defuzziﬁcation task can be understood as a supervised regression problem in which appropriate training samples are used to infer compositional land cover values for the whole image (in this case, fuzzy image regions need to add to 1.0). For solving such a problem, a double regression is conducted: one for the fuzzy segmentation, and the other one for the ﬁnal classiﬁcation. 3.4 Application of the proposed framework
The proposed framework has been applied to qualitative and quantitative land cover analysis. The following sections summarize main results of applying the fuzzy image segmentation approach for classiﬁcation of a number of remotely sensed urban datasets. 3.4.1 Categorical land cover analysis
Lizarazo & Elsner (2009) applied the fuzzy segmentation method to classify the University image, a hyperspectral dataset of the University of Pavia (Italy) that was collected by the Hysens project on 8th July 2002 (Gamba, 2004). The University data set size is 610x339 pixels. Spatial resolution is 1.2 m. There are 112 hyperspectral channels ranging from 400 to 1260 nm. This data set was chosen because it was used in a previous OBIA study using crisp image segmentation (Aksoy, 2006). Seven spectral channels which roughly correspond to the center wavelength of Landsat Thematic Mapper (TM) channels were selected as input for the image classiﬁcation experiment. The University data set includes training and testing samples for nine land cover classes: asphalt, meadows, gravel, trees, (painted) metal sheets, bare soil, bitumen, selfblocking bricks, and shadow. The training sample comprises 3921 pixels and the testing sample 42762 pixels. For this experiment, the fuzzy segmentation stage was conducted using the generalized additive method (GAM) which output nine fuzzy image regions (i.e. one image region per target land cover class) from the seven spectral channels. For the feature analysis stage, the ANDI indices were calculated for the following pairs of classes as they were visually identiﬁed as potencial source of spectral confusion: asphalt and gravel, asphalt and bitumen, asphalt and bricks, meadows and trees, and meadows and bare soil. For the ﬁnal defuzziﬁcation stage, a support vector machine (SVM) was used to output the intended land cover classiﬁcation. The overall classiﬁcation accuracy reported by Lizarazo & Elsner (2009) was a 0.95 conﬁdence interval of KIA (Kappa Index of Agreement) equals to [0.764, 0.784]. The performance of the proposed fuzzy segmentation method was evaluated by using an earlier classiﬁcation of the University data set (Aksoy, 2006) as benchmark. In that work, a three stage crisp segmentation OBIA procedure was applied. Input to the crisp segmentation comprised of 24 bands as follows: 8 linear discriminant analysis (LDA) bands, 10 principal components analysis (PCA) bands, and 16 Gabor texture bands. The feature analysis step used a feature vector of 4 values for each image object by clustering spectral statistics from the 24 input bands and from 10 shape attributes. The ﬁnal classiﬁcation was based on Bayesian classiﬁers. The overall classiﬁcation accuracy reported by Aksoy (2006) was a 0.95 conﬁdence interval of KIA equals to [0.788, 0.814]. It is apparent that the work of Aksoy (2006) achieved slightly higher accuracies. However, it also required signiﬁcant user input and has very limited potential for automation. Fuzzy segmentationbased OBIA as implemented by Lizarazo & Elsner (2009) required very little userinput and could easily be automated for the analysis of large data sets.
Segmentation of Remotely Sensed Imagery: Moving from Sharp Objects to Fuzzy Regions
267
3.4.2 Quantitative land cover analysis
The traditional OBIA methods, based on crisp segmentation, are mainly used to delineate and classify land cover efﬁciently (Duveiller et al., 2008). As crisp segmentation deals with the delineation of meaningful sharp image objects, it is clear that its main advantage relies on its ability to produce qualitative land cover units. The OBIA approach focuses on qualitative interpretation of remote sensing images. Often, however, it is preferable to use a continuous representation of land cover (i.e. a quantitative analysis) rather than a discrete classiﬁcation (i.e. a qualitative classiﬁcation). In a continuous representation of land cover, the different compositional classes vary continuously not only in space but also in time. By contrast, in the discrete representation of land cover, each spatial unit is represented by a single categorical value which is stable over time (Lambin, 2000). Quantitative land cover clasiﬁcation is of particular relevance where a individual land surface characteristic is of interest, e.g. the fraction of impervious surface in each pixel, or the percentage of tree cover at each location. Lizarazo (2010) applied the fuzzy segmentation method for estimation of impervious surface area (ISA) values in Montgomery County, Maryland, USA, from LandsatTM ortorectiﬁed images. The Landsat images, collected in 1990 and 2000, comprised of seven spectral channels with pixel size of 28.5 m at 50 m root mean square error (RMSE) positional accuracy. As part of an extensive study on urbanization in the Chesapeake Bay Watershed, Jantz et al. (2005) obtained accurate ISA maps and qualitative land cover maps for the study area. These maps were used as ground reference for training sample collection and accuracy evaluation. The estimation of quantitative land cover involved four main stages: (i) preprocessing for relative radiometric normalization of the two LandsatTM scenes, (ii) fuzzy segmentation, (iii) feature analysis, and (iv) ﬁnal regression. For the fuzzy segmentation stage, a supervised regression SVM model was ﬁtted using land cover class memberships as a response variable and six independent analysis components (ICA) as predictor variables. Training sample comprised 1000 randomly selected pixels representing 0.27% of the study area. As a result of this stage, the following ﬁve fuzzy image regions were created: water, urban, grass, trees, and bare soil. For the feature analysis stage, ANDI indices were calculated for the following pair of fuzzy image regions: urban and water, urban and grass, urban and trees, and urban and bare soil. For the ﬁnal regression stage, the SVM regression technique was used to predict ISA values from the ﬁve vague image regions and the four SOL indices. As a ﬁnal step, the accuracy of the fuzzy segmentation method was evaluated by comparing the predicted ISA values with the ground reference data using the standard correlation index (Jensen, 2005). The correlation index between the predicted ISA images and the ground reference images were 0.75 for the 1990 date, and 0.79 for the 2000 date. The fuzzy segmentation method overestimated ISA values by 10% and 8% respectively. This estimation level can be considered to be a good approximation to the real impervious surface area. Minor misprediction problems were detected only after a careful visual assessment. Unlike traditional OBIA methods, the fuzzy segmentation approach can therefore be used for both qualitative and quantitative land cover analysis. This characteristic of the proposed approach is clearly one of its central advantages. This potential is very important for remote sensing image analysis, as there is an increasing need to estimate biophysical parameters to better understand the environmental dynamics at local and national scales (CampsValls et al., 2009). In this context, there seems to be an urgent need for robust and accurate regression methods in order to overcome the challenges posed by the inversion of analytical models which are often considered as too complex, computationally intensive, or sensitive to noise (CampsValls et al., 2009; Kimes et al., 2000).
268
Image Segmentation
Consequently, the fuzzy segmentation approach can be included in the list of emerging methods which use empirical models to learn the relationship between the acquired imagery and actual ground measurements. Furthermore, the fuzzy segmentation approach can act as an unifying framework which contributes to bridging the gap between the two traditional branches of remote sensing analysis, i.e. those methods that study categorical variables and those methods focused on estimating continuous variables, which are often considered as opposite, and independent worlds (Liang, 2007).
4. Conclusions Fuzzy image regions can serve as a general conceptual framework for producing both accurate quantitative and qualitative characterizations of land cover. The application of the proposed method in several case studies illustrated that fuzzy image segmentation provides a robust, accessible and easy to use method for urban land cover analysis. The contribution of this chapter, which was discussed in last section, can be summarized as follows: 1. The potential of fuzzy sets concepts to extend the traditional perspective of image segmentation was evaluated. 2. The suitability of the fuzzy segmentation approach to produce both categorical and compositional characterization of land cover was tested. The proposed method therefore enhances the range of OBIA alternatives and contributes to improving the remote sensing image analysis process. It addresses the concerns of Platt & Rapoza (2008) that traditional OBIA approaches commonly have difﬁculties dealing with spectrally overlapping classes. This is particularly true in urban landscapes which usually comprise objects of different sizes with high levels of withinclass heterogeneity (Herold et al., 2007). In such landscapes, where spectral complexity is a common issue, the fuzzy segmentation approach represents a suitable alternative to the hard image segmentation approach, a trial and error process which may not always succeed (Frauman & Wolff, 2005; Lang et al., 2006; Robson et al., 2006; Schiewe et al., 2001b). In addition, in contrast to image classiﬁcation based on hard segmentation, the proposed method is able to deal, throughout the whole process, with the inherent imperfections of remote sensing data such as sensor noise, local shading and highlights which often prevent the creation of meaningful image objects (Bezdek et al., 1999). Lastly, standard crisp OBIA implementations require signiﬁcant user input for the parameterization of segmentation models. This introduces signiﬁcant subjectivity into the process and makes the eventual classiﬁcation performance at least partly a function of individual user skill and effort. Fuzzy segmentationbased OBIA on the other hand requires very little user input. This makes the approach more robust, objective, and reproducible.
5. References Aksoy, S. (2006). Spatial techniques for image classiﬁcation, in C. C. H. (ed.), Signal and Image Processing for Remote Sensing, Taylor & Francis, pp. 491–514. Aplin, P. & Atkinson, P. (2001). Subpixel land cover mapping for perﬁeld classiﬁcation, International Journal of Remote Sensing 22(14): 2853–2858. Atkinson, P. M. & Foody, G. M. (2002). Uncertainty in remote sensing and GIS: Fundamentals, in G. M. Foody & P. M. Atkinson (eds), Uncertainty in Remote Sensing and GIS, Wiley, chapter 1, pp. 1–18.
Segmentation of Remotely Sensed Imagery: Moving from Sharp Objects to Fuzzy Regions
269
Azad, P., Gockel, T. & Dillmann, R. (2008). Computer Vision  Principles and Practice, Elektor International Media BV. Baatz, M. & Schape, A. (2000). Multiresolution segmentation: An optimization approach for high quality multiscale image segmentation, in J. S. et al. (ed.), Angewandte Geographische Informationsverarbeitung XII, AGI Symposium, pp. 12–23. Benz, U. (2001). Deﬁniens imaging GmbH: Object oriented classiﬁcation and feature detection, IEEE Geoscience and Remote Sensing Society Newsletter . Benz, U., Hofmann, P., Willhauck, G., Lingenfelder, I. & Heynen, M. (2004). Multiresolution, objectoriented fuzzy analysis of remote sensing data for GISready information, ISPRS Journal of Photogrammetry and Remote Sensing 58(3–4): 239–258. Bezdek, J., Pal, M., Keller, J. & Krisnauram, R. (1999). Fuzzy Models and Algorithms for Pattern Recognition and Image Processing, Springer. Bian, L. (2007). Objectoriented representation of environmental phenomena: Is everything best represented as an object?, Annals of the Association of American Geographers 97(2): 267–281. Blaschke, T. (2010). Object based image analysis for remote sensing, ISPRS Journal of Photogrammetry and Remote Sensing 65: 2–16. Blaschke, T., Burnett, C. & Pekkarinen, A. (2006). Image segmentation methods for objectbased analysis and classiﬁcation, in S. de Jong & F. van der Meer (eds), Remote Sensing Image Analysis: Including the Spatial Domain, Springer, pp. 211–236. Bock, M. & Lessing, R. (2002). Remote sensing, formation of objects and determination of quality, Technical report, DELPHIIMM. Burnett, C. & Blaschke, T. (2003). A multiscale segmentation/object relationship modeling methodology for landscape analysis, Ecological Modelling 168: 233–249. Burrough, P. A. (1989). Fuzzy mathematical methods for soil survey and land evaluation, Journal of Soil Science 40: 477–492. Burrough, P. A. & Frank, A. U. (1996). Geographic Objects with indeterminate boundaries, Taylor and Francis. CampsValls, G., MuñozMarí, J., GómezChova, L., Richter, K. & CalpeMaravilla, J. (2009). Biophysical parameter estimation with a semisupervised support vector machine, IEEE Geoscience and Remote Sensing Letters . Carleer, A., Debeir, O. & Wolff, E. (2005). Assessment of very high spatial resolution satellite image segmentations, Photogrammetric Engineering and Remote Sensing 71: 1285–1294. Castilla, G. & Hay, G. J. (2008). Image objects and geographic objects, Objectbased Image Analysis, Springer, pp. 91–110. Corcoran, P. & Winstanley, A. (2006). Using texture to tackle the problem of scale in landcover classiﬁcation, International conference on object based image analysis, Salzburg. Couclelis, H. (1992). Theory and Methods of SpatialTemporal Reasoning in Geographic Space, Springer, chapter People manipulate objects (but cultivate ﬁelds): beyond the raster debate in GIS, pp. 65–77. Dam, E. (2000). Evaluation of diffusion schemes for multiscale watershed segmentation. MSc Thesis in Computer Science, University of Copenhagen, Denmark. Deﬁniens (2007). Developer 7 reference book. Doan, H. T. X. & Foody, G. M. (2007). Increasing soft classiﬁcation accuracy through the use of an ensemble of classiﬁers, International Journal of Remote Sensing 28(20): 4609–4623. Duveiller, G., Defourny, P., Desclee, B. & Mayaux, P. (2008). Deforestation in central africa: Estimates at regional, national and landscape levels by advanced processing
270
Image Segmentation
of systematicallydistributed landsat extracts., Remote Sensing of Environment 112(5): 1969 – 1981. Esch, T., Thiel, M., Bock, M., Roth, A. & Dech, S. (2008). Improvement of image segmentation accuracy based on multiscale optimization procedure, IEEE Geosciences and Remote Sensing Letters 5(3): 463–467. Fisher, P. (1997). The pixel: a snare and a delusion., International Journal of Remote Sensing 18: 679–685. Fisher, P., Arnot, C., Wadsworth, R. & Wellens, J. (2006). Detecting change in vague interpretations of landscapes, Ecological Informatics 1: 163–178. Fisher, P. & Pathirana, S. (1990). The evaluation of fuzzy membership of land cover classes in the suburban zone, Remote Sensing of Environment 34: 121–132. Foody, G. M. (1992). A fuzzy set approach to the representation of vegetation continua from remotely sensed data: an example from lowland heath, Photogrammetric Engineering and Remote Sensing 58: 221–225. Foody, G. M. (1996). Approaches for the production and evaluation of fuzzy land cover classiﬁcations from remotely sensed data, International Journal of Remote Sensing 17: 1317–1340. Frauman, E. & Wolff, E. (2005). Segmentation of very high spatial resolution satellite images in urban areas for segmentsbased classiﬁcation, International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences. Freixenet, J., Muñoz, X., Raba, D., Martí, J. & Cufí, X. (2002). Yet another survey on image segmentation: Region and boundary information integration, Lecture Notes in Computer Science 2352: 408–422. Friedl, M. A., McGwire, K. C. & McIver, D. K. (2001). An Overview of Uncertainty in Optical Remotely Sensed Data for Ecological Applications, Springer, chapter 12, pp. 258–283. Gamba, P. (2004). A collection of data for urban area characterization, Geoscience and Remote Sensing Symposium. Gao, J. (2009). Digital Analysis of Remotely Sensed Imagery, Mc Graw Hill. Goodchild, M. (1992). Geographical data modeling, Computer and Geosciences 18(4): 401–408. Goodchild, M., Yuan, M. & Cova, T. (2007). Towards a general theory of geographic representation in GIS, International Journal of Geographical Information Science 21(3): 239–260. Haralick, R. & Schapiro, L. (1992). Computer and Robot Vision, AddisonWesley. Hay, G. & Castilla, G. (2008). Geographic objectbased image analysis (GEOBIA): A new name for a new discipline, Objectbased Image Analysis: Spatial Concepts for KnowledgeDriven Remote Sensing Applications, Springer. Herold, M., Gardner, M. & Roberts, D. (2003). Spectral resolution requirements for mapping urban areas, IEEE Transactions on Geoscience and Remote Sensing 41: 1907–1919. Herold, M., Schiefer, S., Hostert, P. & Roberts, D. A. (2007). Applying Imaging Spectrometry in Urban Areas, CRC Press, chapter Chapter 7, pp. 137–161. Jantz, P., Goetz, S. & Jantz, C. (2005). Urbanization and the loss of resource lands in the chesapeake bay watershed, Environmental Management 36(6): 808–825. Jensen, J. (2005). Introductory Image Processing: a Remote Sensing perspective, Prentice Hall. Kimes, D. S., Knyazikhin, Y., Privette, J. L., Abuelgasim, A. A. & Gao, F. (2000). Inversion ˇ Remote Sensing of Environment 18: 381 – 439. methods for physicallybased modelsT,
Segmentation of Remotely Sensed Imagery: Moving from Sharp Objects to Fuzzy Regions
271
Lambin, E. (2000). Landcover categories versus biophysical attributes to monitor landcover change by remote sensing, Observing Land from Space: Science, Customers and Technology, Kluwer Academic Publishers. Lang, S. (2008). Objectbased image analysis for remote sensing applications: modeling reality – dealing with complexity, Objectbased Image Analysis: Spatial Concepts for KnowledgeDriven Remote Sensing Applications, Springer, chapter 1, pp. 3–27. Lang, S., Albretch, F. & Blaschke, T. (2006). Tutorial: Introduction to objectbased image analysis, Centre for Geoinformatics  ZGIS. Liang, S. (2007). Recent developments in estimating land surface biogeophysical variables from optical remote sensing, Progress in Physical Geography 31(5): 501 – 516. Lizarazo, I. (2010). Fuzzy image regions for estimation of impervious surface areas, Remote Sensing Letters 1(1): 19–27. Lizarazo, I. & Elsner, P. (2009). Fuzzy segmentation for objectbased image classiﬁcation, International Journal of Remote Sensing 30(6): 1643–1649. Meinel, M. & Neubert, G. (2004). A comparison of segmentation programs for high resolution remote sensing data, International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, ISPRS. Mesev, V. (2003). Remotely Sensed Cities, Taylor & Francis. Moigne, J. L. & Tilton, C. (1995). Reﬁning image segmentation by integration of edge and ˝ region data, IEEE Transactions on Geoscience and Remote Sensing 33(3): 605U615. Molenaar, M. & Cheng, T. (2000). Fuzzy spatial objects and their dynamics, ISPRS Journal of Photogrammetry and Remote Sensing 55(3): 164–175. Neubert, G., Herold, M. & Meinel, M. (2006). Evaluation of remote sensing image segmentation quality  futher results and concepts, First International Conference on ObjectBased Image Analysis  Proceedings, ISPRS. Neubert, M., Herold, H. & Meinel, G. (2008). Assessing image segmentation quality concepts, methods and application, ObjectBased Image Analysis  Spatial concepts for knowledgedriven remote sensing applications, Springer, Berlin, pp. 769–784. Ortiz, A. & Oliver, G. (2006). On the use of the overlapping area matrix for image segmentation algorithm: a survey and new perfomance measures, Pattern Recognition Letters 27: 1916–1926. Pal, N. R. & Pal, S. K. (1993). A review of image segmentation techniques, Pattern Recognition 26(9): 1277–1294. Platt, R. V. & Rapoza, L. (2008). An evaluation of an objectoriented paradigm for land use/land cover classiﬁcation, The Professional Geographer 60(1): 87–100. Rekik, A., Zribi, M., Hamida, A. B. & Benjellounl, M. (2007). Review of satellite image segmentation for an optimal fusion system based on the edge and region approaches, International Journal of Computer Science and Network Security 7(10): 242–250. Robson, M., Vachon, P. W. & Seeker, J. (2006). Evaluation of ecognition for assisted target detection and recognition in sar imagery, Geoscience and Remote Sensing Symposium, 2006. IGARSS 2006. pp. 145–148. Rosenfeld, A. & Kak, A. (1992). Digital Picture Processing, Academic Press. Ross, T. (2004). Fuzzy Logic with Engineering Applications, Wiley. Schiewe, J., Tufte, L. & Ehlers, M. (2001a). Potential and limitations of multiscale segmentation methods in remote sensing, eCognition website . Schiewe, J., Tufte, L. & Ehlers, M. (2001b). Potential and problems of multiscale segmentation methods in remote sensing, GIS. GeoInformationsSysteme 6(9): 34–39.
272
Image Segmentation
Schneider, W. & Steinwender, J. (1999). Landcover mapping by interrelated segmentation and classiﬁcation of satellite images, International Archives of Photogrammetry & Remote Sensing, Vol. 32 Part 743. Schott, J. R. (1997). Remote Sensing: The Image Chain Approach, Oxford University Press. Shackelford, A. & Davis, C. (2003). A combined fuzzy pixelbased and objectbased approach for classiﬁcation of highresolution multispectral data over urban areas, IEEE Transactions on Geoscience and Remote Sensing 41(10): 2354–2363. Shi, J. & Malik, J. (2000). Normalized Cuts and Image Segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence 22(8): 888–905. Song, M., Civco, D. & Hurd, J. (2005). A competitive pixelobject approach for land cover classiﬁcation, International Journal of Remote Sensing 26(22): 4981–4997. Thomas, N., Hendrix, C. & Congalton, R. G. (2003). A comparison of urban mapping methods using highresolution digital imagery, Photogrammetric Engineering and Remote Sensing 69(9): 963–972. Townshend, J. R. G., Huang, C., Kalluri, S. N., DeFries, R. S., Liang, S. & Yang, K. (2000). Beware of perpixel characterization of land cover, International Journal of Remote Sensing pp. 839–843. Wang, F. (1990). Fuzzy supervised classiﬁcation of remote sensing images, IEEE Transactions on Geoscience and Remote Sensing 28(2): 194–201. Wei, W., Chen, X. & Ma, A. (2005). Objectoriented information extraction and application in high resolution remote sensing image, International Geoscience and Remote Sensing Symposium (IGARSS), Vol. 6, pp. 3803–3806. Woodcock, C. E. (2002). Uncertainty in remote sensing, in G. M. Foody & P. M. Atkinson (eds), Uncertainty in Remote Sensing and GIS, Wiley, chapter 2, pp. 19–24. Yan, G., Mas, J. F., Maathuis, B. H. P., Zhang, X. & Van Dijk, P. M. (2006). Comparison of ˝ a case study in a pixelbased and objectoriented image classiﬁcation approaches U coal ﬁre area, Wuda, Inner Mongolia, China, International Journal of Remote Sensing 18(20): 4039–4055. Zadeh, L. (1965). Fuzzy Sets, Information and control (8): 338–353. Zhan, Q., Molenaar, M., Tempﬂi, K. & Shi, W. (2005). Quality assessment for geospatial objects derived from remotely sensed data, International Journal of Remote Sensing pp. 2953–2974. Zhang, J. & Foody, G. (1998). A fuzzy classiﬁcation of suburban land cover from remotely sensed imagery, International Journal of Remote Sensing 19: 2721–2738. Zhang, J. & Stuart, N. (2001). Fuzzy methods for categorical mapping with imagebased land cover data, International Journal of Remote Sensing 15(2): 175–195. Zhang, Y. (2006). A fuzzy logic approach to supervised segmentation for objectoriented classiﬁcation, ASPRS 2006 Annual Conference. Zhang, Y. J. (1996). A survey on evaluation methods for image segmentation, Pattern Recognition 29(8): 1335–1346. Zhou, C., Wang, Z., Zhang, C., Qi, C. & Wang, Y. (2007). Objectoriented information extraction technology from quickbird pansharpened images, Proceedings of SPIE The International Society for Optical Engineering, Vol. 6279, p. 62793L.
14 Colorbased Texture Image Segmentation for Vehicle Detection Ricardo MejíaIñigo, María E. BarillaPérez and Héctor A. MontesVenegas Universidad Autónoma del Estado de México México 1. Introduction Trafﬁc ﬂow is the study of interactions between vehicles, drivers and related infrastructure with the goal of understanding and developing an efﬁcient road network. A number of devices have been implemented for monitoring trafﬁc ﬂow such as magnetic strips and radar detectors (HaidarianShahri et al. (2007)). All these devices have served for a very speciﬁc purpose. However, they do not provide as much information of the scene as cameras may do. The resulting information acquired from cameras should be properly processed to monitor vehicles’ behavior. Visionbased techniques have been increasingly used to extract useful trafﬁc information from image sequences such as vehicle count, vehicle ﬂow, vehicle speed, vehicle classiﬁcation, and lane changes. All these tasks require an accurate vehicle tracking (Zhu et al. (2009)). Vehicle tracking is a technique that generates the trajectory of the vehicle over time by locating its position in every frame of an image sequence (Yilmaz (2006)). Usually, vehicle tracking requires three initial steps, namely, (a) choosing a suitable object representation, (b) selection of image features used as an input for the tracker and (c) vehicle detection. Vehicle detection is considered a difﬁcult task, as an appropriate feature selection is needed to deal with the visual complexity of the road scene that may include congestions, shadows, weather incidences, headlights, road conditions and other trafﬁc participants (Coifman et al. (1998)). Different approaches can be used to carry out the vehicle detection task, such as point detectors, background subtraction, segmentation or supervised learning methods. The selection of discriminative features plays a critical role because they should be so unique as to be easily distinguished in a feature space (Yilmaz (2006)). Texture is considered to be a rich source of information about the nature and the threedimensional shape of objects. It is deﬁned by Petrou (2006) and Materka & Strzelecki (1998) as complex visual patterns and subpatterns with singular properties such as brightness, color and size. There exist different approaches for texture modeling, such as statistical models, structural models and models based on transforms. Nevertheless, texture by itself is not enough to differentiate among different objects because contrasting surfaces cause false positives detections (Sun et al. (2006)). Color is considered a relevant feature when dealing with the perception of static and moving images. Visual contrast is useful to ﬁlter information present in each color component (Trémeau et al. (2008)) and to distinguish among similar grayscale intensities (BarillaPérez & Spann (2008)).
274 2
Image Segmentation Image Segmentation
The combination of color and texture have been proved to achieve better results and could be exploited more effectively when integrated than when treated in isolation (BarillaPérez & Spann (2008)). In this chapter we propose two segmentation methods based on color and texture to solve the problem of differentiating vehicles from the background. Both features were used under the following assumptions: • Road color is considered homogeneous because the road surface is made of the same material. • An abrupt difference between the color of the road and the color of the vehicles is useful to discriminate them from each other. • Even if vehicles are similar in color to the road, car and road textures are different. Thus, texture may still be used to effectively discriminate between road and cars. In this chapter two methods are described for vehicle detection by segmenting the regions they occupy, both methods use color and texture as main features. The ﬁrst method uses L*u*v* color space and the DualTree Complex Wavelet Transform (DTCWT) for texture modeling and a simple background identiﬁcation to carry out the vehicle detection. The second method is based on a change detection technique that integrates the intensity and texture differences between the current frame and a previously modeled background. L*u*v* color space is also used in this method and texture differencing measure is based on a function between gradient vectors. In both methods, an additional process is also applied using morphological operations to improve the detection of the regions that describe the vehicles. The chapter also includes an overview of related work and a detailed description of how color and texture features can be properly combined with both approaches to achieve accurate vehicle detection results. An experimental comparison of the two vehicle detection methods is performed both qualitatively and quantitatively. The Outex framework have been used in order to compare the accuracy of both methods (Ojala et al. (2002)). Finally, the work is summarized and results are discussed.
2. Overview of previous work This section presents an overview of relevant visionbased vehicle detection systems. Vehicle detection using optical sensors is a very challenging task due to the various conditions found in any road scene. The variability of such conditions may be caused by the weather, by illumination changes, by the presence of shadows and by the numerous objects that are part of the scene. Additionally, vehicle features such as shape, size and color are not always sufﬁcient for a reliable detection as the appearance of a vehicle also depends on its pose and its interaction with other trafﬁc participants. In this section we identify several methods for vehicle detection, namely, background extraction methods, edges detection methods and statisticalbased approaches. We also review previous works that deal with the occlusion problem as well as with the shadow removal problem. 2.1 Background subtraction
Background subtraction for vehicle detection methods compare each video frame against a reference background model. Assuming that all moving objects are vehicles, the elements in the current frame that are signiﬁcantly different from the background model are considered the objects of interest. Methods for building a background model include estimates based on median ﬁlters (Cucchiara et al. (2003); Lo & Velastin (2001); Zhou & Aggarwal (2001)), Kalman ﬁlters (Halevy & Weinshall (1999); Wren et al. (1997)), Gaussian models (Cheng et al. (2007);
Colorbased for Vehicle Detection Texture Image Segmentation for Vehicle Detection Colorbased Texture Image Segmentation
275 3
Lee et al. (2003); Stauffer & Grimson (2000)) or simple histograms and blob formation models (Elgammal et al. (1999); SenChing et al. (2005)). 2.2 Edge detection
Edge detection methods have been used under the assumption that they are robust to illumination changes. In order to increase robustness of vehicle detection, edges have been used in conjunction with color information. Tsai et al. (2007) proposed a color model to ﬁnd candidate vehicles under the assumption that vehicles colors change with lightning conditions. Possible vehicles are then actually found by using local features such as corners, edge maps and wavelet coefﬁcients. Eng et al. (2008) also use color and edge information to detect vehicles. They developed a vehicle detection method by subtracting objects from a background which is previously modeled by color and edges. Techawatcharapaikul et al. (2008) proposed a method for vehicles and shadows segmentation using temporal edge density information from adjacent frames. Mo & Zhang (2010) have detected vehicles by ﬁnding their voting center which is then used to implement a trafﬁc analysis system. The vehicles voting centers are calculated by matching the results of a segmentation process with a previously learned spatial distribution of the objects of interest. 2.3 Statisticalbased approaches
Wang & Lien (2008) proposed an statistical approach for vehicle detection based on the vector projection of three main local features of vehicles, namely the roof and the two headlights. Both Principal Component Analysis (PCA) and Independent Component Analysis (ICA) techniques are combined in order to improve the tolerance of the detection process under illumination variations and for vehicle position. Although the authors show very promising results, it is unclear if the method is robust under signiﬁcant rotations of the main local features of the car. An additional work using a graphbased approach that considers the identity of neighboring vehicles has been proposed by Shahri et al. (2007). Their results showed that vehicle tracking is feasible, even for low quality and low frame rate trafﬁc cameras. 2.4 Handling occlusions
Occlusion is one of the most difﬁcult problems associated with vehicle detection. In order to address this problem Kamijo et al. (2000) proposed a method based on a SpatioTemporal Markov Random Field model that considers the texture correlation between consecutive images and the correlation among neighbors within an image. This algorithm requires only gray scale images and does not assume any physical models of the vehicles. Even though occlusions in the scene happened in very complicated manners (vehicles appear in various shapes and they move in unpredictable ways), the method achieved a success rate as high as 95%. Also to detect and handle vehicle occlusions, Zhang et al. (2008) presented a framework composed of three levels named intraframe, interframe, and tracking. On the intraframe level, occlusions are handled by evaluating the size of vehicles. On the interframe level, occlusions are detected by using the motion vectors of vehicles. On the tracking level, occlusions are handled while tracking the vehicle by performing a bidirectional occlusion reasoning algorithm. Most partial occlusions are handled on the intraframe and interframe levels, and full occlusions are handled on the tracking level. In addition to this work, Zhang & Yuan (2007) used lowresolution images for detecting and tracking vehicles as most of these images were captured far from the vehicles. The authors used borders to build a model to ﬁnd outliers that can differentiate the vehicle from another object in the scene. Kalman Filters in combination with the body’s radius allow handling total and partial occlusions when obstacles blocked the view of the vehicle. This method requires
276 4
Image Segmentation Image Segmentation
very lowresolution images to work correctly. Earlier research efforts for solving vehicle occlusions on monocular trafﬁcimage sequences (Pang et al. (2004)) and for detecting and tracking vehicles by segmenting scenes with severe occlusions (Gentile et al. (2004)) have been published. However, based on their experimental results, Zhang et al. (2008) stated that “quantitative evaluation and comparison demonstrate that the proposed method outperforms stateoftheart methods”. 2.5 Shadows removal
Shadows interfere with moving vehicle detection because they cause erroneous segmentation of objects. For this reason, there is a recent interest on developing algorithms to deal with shadows of the vehicles present in the scene. Chen et al. (2010) have removed shadows from the luminance component of an image while keeping the chrominance components intact. Fang et al. (2008) have used spectral and geometrical properties of shadows to create a model to eliminate them from the scene. This method is considered a previous step before the actual vehicle detection. 2.6 Driver assistance
For trafﬁc monitoring tasks it is crucial to detect the lane and other features of the road and then identify the presence of other vehicles. Kastrinaki et al. (2003) present a survey on videobased systems considering both areas of road trafﬁc monitoring and automatic vehicle guidance. In their paper, the authors survey methods for the two main subtasks involved in trafﬁc applications, i.e. the automatic lane ﬁnding and vehicle detection. Two particular cases are considered in this paper. The ﬁrst one is the case in which a static camera (usually located in a high position) observes a road scene for the purpose of trafﬁc surveillance. In the second case at least one camera is mounted on a vehicle that moves in a dynamic road scene. In a later work, Sun et al. (2006) present an extended review of systems for vehicle detection for onboard automotive driver assistance systems. For the methods summarized in this paper, the camera is mounted on the vehicle. DiazAlonso et al. (2008) have proposed an aid system to help the driver in overtaking and lane changing decisions. These tasks are addressed with an opticﬂowdriven scheme, focusing on the visual ﬁeld in the side mirror by placing a camera on top of it.
3. Colorbased texture segmentation Colorbased texture segmentation is a process that divides an image into homogeneous regions using both color and textures features to distinguish from nonsimilar regions (BarillaPérez (2008)). Using more than one descriptor helps to improve the segmentation process when the descriptors are visually relevant and they can be integrated to obtain a single similarity measure. In this section, a colorbased texture segmentation algorithm originally proposed by MejiaIñigo et al. (2009) is described. The algorithm uses a feature vector based on color and texture to differentiate vehicles from the background. This selection of features is based on the assumptions listed in Section 1. 3.1 Texture and color
Texture is regarded as a rich source of information about the nature and threedimensional shape of objects (Petrou (2006)). It consists of complex visual patterns and subpatterns with singular properties such as brightness, color and size (Materka & Strzelecki (1998)). There exist different approaches for texture modeling, such as statistical models, structural models
277 5
Colorbased for Vehicle Detection Texture Image Segmentation for Vehicle Detection Colorbased Texture Image Segmentation
and models based on transforms. Materka & Strzelecki (1998) suggested that transform methods permit the variation of spatial resolution to represent textures at different scales. In this sense, Wavelet Transform is a viable method to obtain timefrequency localization analysis and to identify stationary and nonstationary textures. As a framework, the DualTree Complex Wavelet Transform (DTCWT) overcomes several difﬁculties present in other Wavelet Transform methods (Selesnick et al. (2005)) and offers the following advantages: • Nearly shift invariance. When the transform is applied, the results seem to be mildly affected by translations in the original input. • Directional selectivity. A resulting set of 6 different positive and negative directions: 15◦ , 45◦ , 75◦ , 15◦ , 45◦ and 75◦ . • Moderate redundancy. The redundancy introduced by Complex Wavelets is 2d :1 for d dimensions (Kingsbury (1999)).
Fig. 1. Directional results of applying the DualTree Complex Wavelet Transform to L* component (b) of a car image (a) for (c) 75◦ , (d) 75◦ , (e) 15◦ , (f) 15◦ , (g) 45◦ and (h) 45◦ directions Color is perceived as a nonlinear psychovisual system too complex to be modeled by a simple function (Bovik & Bovik (2000)). The L*u*v* color space is considered one of the most suitable color spaces to represent human perception, as it is perceptually uniform (Malacara (2001)). Its L* component provides lightness information useful to describe texture by illumination changes on the surface of the object, whereas u* and v* provide chrominance
278 6
Image Segmentation Image Segmentation
information about the pixel values. Compared to other color spaces, L*u*v* has been proved to achieve high success rates for image segmentation when using texture and color features (BarillaPérez (2008)). 3.2 Feature extraction
Let be IL∗ u∗ v∗ ( x, y) a color image in the L*u*v* color space of size M × N where { x = 0, ..., M − 1} and {y = 0, ..., N − 1}. For the color plane IL∗ ( x, y) of the image IL∗ u∗ v∗ ( x, y), we compute (s,b)
WI
L∗
(i, j) = DTCWT { IL∗ ( x, y)}.
(1)
(s,b)
where WI ∗ (i, j) are the wavelet coefﬁcients as complex numbers, {s = 1, ..., S} represents L the levels of resolution from the highest to the lowest and {b = 1, ..., 6} indicates the number of subbands produced by the wavelet transform at scale s. The size of the subbands b at scale N M N s is M 2s × 2s where {i = 0, ..., 2s − 1} and { j = 0, ..., 2s − 1}. Computing the modulus of the wavelet coefﬁcients yields its energy. The feature vectors are constructed from this energy combined with the chrominance components u* and v*. Vehicles traveling on a onedirection road have texture qualities in different directions, but if the vehicles are moving towards the capture device we only consider 75◦ and 75◦ given that these two components provide more information about the vertical displacement of the vehicles (see Fig. 1). Also, DTCWT can be applied successively until scale s = S is achieved. Considering the resolution of the images, a S = 3 feature vector limit is employed. L*u*v* color space components are separated in chrominance and lightness information (Sangwine et al. (1998)). DTCWT is applied on the L* component at different coarseness levels because texture features are considered to be more sensitive to lightness and scale changes. In contrast, color remains almost invariant to different textures variations. By reducing DTCWT components, the dimensional space of the feature vector is also reduced to decrease computational cost. The feature vector extraction process is depicted in Fig. 2.
!
!
!
!
!
!
Fig. 2. Feature vector extraction and wavelet components.
279 7
Colorbased for Vehicle Detection Texture Image Segmentation for Vehicle Detection Colorbased Texture Image Segmentation
3.2.1 Segmentation
Once the feature vectors have been calculated, the coarsest level at scale S is processed to obtain a global initialization that will be used to segment higherresolution vectors. Kmeans clustering was used to provide this initial segmentation. The reason for using Kmeans clustering is that can be easily implemented and provides a good reference to identify the most representative k macrotextures for the coarsest level s = S (BarillaPérez (2008)). There are other segmentation methods that could be used to provide this initialization, such as statistical or graphbased methods (Felzenszwalb & Huttenlocher (2004)), but they require several parameters usually deﬁned heuristically. In contrast, Kmeans clustering only requires a k parameter to indicate the most signiﬁcant regions to be identiﬁed within the image. Feature vectors are thought to have 2 representative regions: cars and road, so k = 2 seems to be a reasonable parameter value. The resulting classiﬁcation at scale s = S is used as an initialization for the ExpectationMaximization method at scale s = S − 1, and the result is used to process the feature vector at scale s = S − 2. This process is repeated until the same action is performed to the feature vector at scale s = 1 (See Fig. 3).
Fig. 3. Segmentation process of feature vectors at different scales. Classiﬁcation combines Multi Resolution Analysis (MRA) with a Gaussian Mixture Model to segment images at scale s based on the segmentation provided by scale (s + 1). Segmentation results are in the form of a binary image as depicted in Fig. 4. 3.3 Vehicle motion detection
The analyzed scene is considered static, where only the vehicles are in constant change and the background always preserves the same texture and color. Modeling of the background is needed to separate vehicles from the road. 3.3.1 Background modeling
Considering a scene where images are taken at a time t and an interval between frames Δt, it is possible to build the background from the scene using binary images. Let Frame(t) be a frame taken at a time t; a pixel is part of the background if it keeps the same value from
280 8
Image Segmentation Image Segmentation
Fig. 4. Binary image obtained by a 2cluster segmentation. The white blob is the vehicle in motion and the white vertical stripes are divisions of the lanes.
(t − 1) to (t − m) at interval Δt, where m can be as big as needed to classify the frame. For the experiments discussed in this chapter, the size of m is set to 9. Equation (2) represents the function that classiﬁes pixels on an image at a given time. 0 Frame(t − i ) = Frame(t). Background(t, m) = (2) 1 otherwise. ∀i ∈ {1, ..., m}, An example of a classiﬁed background is shown in Fig.5.d. 3.3.2 Using logic operations for vehicle detection
The background model is represented by a binary value of 1 for all possible objects that can be described as a moving vehicle, and 0 for all objects that do not change in the images from Frame(t − m) to Frame(t). A possible vehicle, PosVehicle, is extracted by applying the logical AND operation between the modeled background Background(t,m) and frame Frame(t). The resultant image contains the pixels that represent the vehicles (see Fig.5.e).
Fig. 5. Possible vehicle detection process. (a) Frame(t2), (b) Frame(t1), (c) current scene Frame(t), (d) modeled background Background(t) and (e) possible vehicle. A number of morphological operations are then used to reduce undesired elements present in the image where the possible vehicles have been detected.
Colorbased for Vehicle Detection Texture Image Segmentation for Vehicle Detection Colorbased Texture Image Segmentation
281 9
3.3.3 Morphological processing
The resultant binary image contains the area of the vehicles along with several undesired elements as well as small holes inside those vehicles caused by illumination changes, background irregularities and vibrations of the capture device (Fig. 6 (a)). In order to remove these unwanted elements while distorting the shape of the objects as little as possible, erosion, opening, dilation and closing operations are applied (Fig. 6 (b)). These morphological operations are useful as describe above, but there are still elements that do not belong to the vehicle, seen as holes inside the objects of interest. Using region ﬁlling (González & Woods (2002)), these holes can be removed to produce a more precise shape of the vehicles detected (Fig. 6.c).
Fig. 6. Morphological processing. (a) Resultant binary image, (b) unwanted elements removal and (c) region ﬁlling.
4. Segmentation based on lightness and texture differences Vehicle detection can be understood as a segmentation process, which allows us to separate the two most signiﬁcative homogeneous regions in a trafﬁc scene: (a) vehicles and (b) road. The method described in this Section attempts to deal with timevariant objects in the scene by differencing those regions in the current frame with a previously modeled background. The background and current frame difference is computed using features of texture and L* component intensities of the L*u*v* color space. 4.1 Background modeling
Vehicle detection can be achieved by building a background model and then identifying the pixels of every frame of the image sequence that vary signiﬁcatively from that model (Yilmaz (2006)). Those pixels are assumed to be a moving vehicle. Several techniques have been proposed to model background as they overcome different drawbacks usually related to the scene being examined (Cheung & Kamath (2004)). These techniques can be classiﬁed as follows: • Recursive techniques: a single background model is updated on each input image. These techniques required less storage, but errors in the background model can persist over the time. • Nonrecursive methods: require an image buffer to estimate a background and variation is estimated for each pixel. Adaptability level is very high as they do not depend on previous background processing, but a large buffer is required to detect movement in congested roads. In order to deal with noise and other objects that may appear in the scene, it is preferable to use nonrecursive techniques to model the background, for instance, a Median Filter.
282 10
Image Segmentation Image Segmentation
Let us consider an image buffer B( x, y, t) that consist on a set of β images, where x and y are deﬁned as spatial image coordinates at a given time t. Median ﬁlter background is deﬁned as Background( x, y, t) = Median( B( x, y, t − β), ..., B( x, y, t − 1) ) ∀ x, ∀ y .
(3)
The median ﬁlter is very consistent even in the presence of outliers caused by noise or sudden object appearance. Correct background model is achieved only when the background is visible at least on 50% of the image buffer for each pixel (Gutchess et al. (2001)). The background is modeled with a buffer using the L* component of the L*u*v* color space. This background model is updated every β frames and it is used as a reference to identify the most persistent pixels within an image. Vehicles are formed by the pixels of the current frame that have a signiﬁcant deviation from the background. 4.2 Image difference
Image difference commonly use absolute value of twoframes intensities subtraction to identify moving objects. However, intensity subtraction is prone to errors because two different objects can have similar intensity. Adding an additional feature, such as texture, may help to effectively discriminate between different objects. In line with Li and Leung’s work (Li & Leung (2002)), texture and intensity information are used to detect changes between two images by using a gradient vector relationship that supports noise and illumination changes. Integrating intensity and texture differences for change detection works under the assumption that background texture remains relatively stable with respect to noise and illumination variations unless it is covered by moving objects or when abrupt lightness changes occur. When the background is covered by a moving object, even if intensities of foreground and background are similar, textures in both images are different. If texture difference is not sufﬁcient to identify homogeneous regions, then intensities are considered more relevant to discriminate regions. 4.2.1 Texture difference
Li & Leung (2002) suggested gradients as a texture measure because neighborhood information is included and it is not easily affected by lightness conditions. Let I be an image, there exists an image I = ( I x , I y ), where I x and I y are results of Sobel gradient operators for its corresponding direction. Image crosscorrelation Ca,b of gradient vectors for two frames a and b is deﬁned as:
Ca,b = Ia Ib cosθ
(4)
where θ is the angle between both vectors. Added to this measure, autocorrelation for a gradient vector frame is
Ca,a = Ia 2
(5)
Vectorial correlation components satisﬁes triangle inequality Ca,a + Cb,b ≥ 2Ca,b
(6)
Ia 2 + Ib 2 ≥ 2 Ia Ib cosθ
2
(7)
2 ,
≈ Ib it means that a From equation 7, we can see that when θ ≈ 0 and when Ia signiﬁcative change inside the image does not exist. If there is a change in gradient image
283 11
Colorbased for Vehicle Detection Texture Image Segmentation for Vehicle Detection Colorbased Texture Image Segmentation
vectors directions of images a and b, the difference of both values will grow according to their magnitude and direction. As mentioned above, autocorrelation and crosscorrelation is applied to differentiate vector direction changes between the current frame L* component and the background model of the same component. Let Mx,y be a 5x5 neighborhood centered at pixel ( x, y), a gradient texture difference measure to discriminate objects between the background and the input frame is deﬁned as follows:
∑
R( x, y) = 1 −
u,v∈ Mx,y
∑
u,v∈ Mx,y
2CB,F (u, v)
(CB,B (u, v) + CF,F (u, v))
(8)
where B is the modeled background and F is the L* component of the current frame. The region Mx,y is used to integrate gradient vector intensities of surrounding pixels that represent local texture features. A change is deemed signiﬁcant when variations in direction are bigger than 90◦ in their orientation. Values of R( x, y) are in the [0,1] interval; where lower values represent non signiﬁcant changes in texture and if values approximate to 1 indicates a higher difference in gradient texture directions. This measure is valid as long as there exist noise and intensity variations in the pixels of the Mx,y region; otherwise, texture becomes less relevant as there is not signiﬁcant gradient change. R( x, y) is then adjusted through a series of validity weights wtexture ( x, y) using the deﬁnition given by Li & Leung (2002). To compute wtexture ( x, y), Li & Leung (2002) produced an image g (equation 9) that combines autocorrelations of two images to validate the pixel values of R. Information of autocorrelation is considered relevant as it contains signiﬁcative intensity changes in both R and g images. Equation 9 uses a maximum function because texture information may exist in any images that have been compared. 1 g( x, y) = maxi∈{ B,F} Ci,i (u, v) (9)  Mx,y  u,v∑ ∈M x,y
The g( x, y) values from equation 9 are coefﬁcients used to detect all pixels with relevant gradient values whether those values belong to the background image or to the current frame. These coefﬁcients are also used to estimate the validity weights wtexture ( x, y) in the [0, 1] interval. For the results reported in this chapter, it is not necessary to detect every change in texture because the images involved can have undesired textured regions not relevant in the current frame; i.e., when background is not modeled properly due to the existence of cluttered regions inside the image buffer, it then causes that the moving objects are considered part of the background. If so, texture detection is not precise because cluttered regions will cause high autocorrelation values, and consequently, high g values that promote a vehicle detection for as long as the same background model is used. In order to address this issue, equation 9 has been modiﬁed as follows: 1 g( x, y) = CF,F (u, v) ∀ x, y ∈ F (10)  Mx,y  u,v∑ ∈M x,y
g( x, y) is used to estimate the validity weights wtexture ( x, y) as follows:
284 12
Image Segmentation Image Segmentation
wtexture ( x, y) =
1, i f g( x, y) > 2Tw g( x, y)/(2Tw ) otherwise
∀ x, y
(11)
From equation 11, Tw is a noise distribution parameter calculated from the image intensity √ difference. It is required that Tw > 2 2σd , however, a very large Tw would deem texture information less signiﬁcant in R. √ For the experiments reported in this chapter, Tw has been set to a value of 50% larger than 2 2σd ; in turn, σd is deﬁned in equation 16. Finally, the texture difference measure is computed as follows: dtexture ( x, y) = wtexture ( x, y) · R( x, y) , ∀ x, ∀y
(12)
4.2.2 Lightness difference
To compute lightness difference, ﬁrst we perform an initial subtraction d( x, y) deﬁned as follows: d( x, y) = L∗F ( x, y) − L∗B ( x, y)
(13)
where B is the modeled background and F is the current frame. d( x, y) measures lightness variations that might be caused by the moving vehicles and other elements in the scene. These elements are removed by smoothing d( x, y) using the following procedure: d( x, y) =
1  Mx,y 
∑
u,v∈ Mx,y
d(u, v)
(14)
Now we compute a difference image d that contains only the lightness variations:
d ( x, y) = d( x, y) − d( x, y)
(15)
It is assumed that the noise contained in image d( x, y) (equation 13) follows a Gaussian distribution N (0, σd ). Consequently, σd is estimated from d ( x, y) as it contains all the small lightness variations. Outlier pixels with high lightness variation can cause a poor noise estimation when deviation needs to be calculated. To tackle this problem, the Least Median of Squares method (LMedS) (Rousseeuw & Leroy (1986)) is used on d ( x, y) as it supports corruption of up to 50% of the input data. Following the deﬁnition given by Rosin (1988), the standard deviation of the noise is calculated as follows: LMedS 0.33724 Finally, d( x, y) is used to estimate the lightness difference dlightness ( x, y) as follows: 1, i f d( x, y) > 2T dlightness ( x, y) = d( x, y)/(2T ) otherwise σd =
(16)
(17)
The T parameter needs to be properly calculated to deal with Gaussian noise. Li & Leung (2002) sets the T parameter as: T = ds  + 3σd .
(18)
Colorbased for Vehicle Detection Texture Image Segmentation for Vehicle Detection Colorbased Texture Image Segmentation
285 13
ds in equation 18 describes lightness changes. It grows when global illumination changes occur, but shadows and local changes diminish this value. ds is calculated as: ds =
1  Ns 
∑
u,v∈ Ns
d(u, v)
(19)
Ns is the set of pixels d( x, y) that satisfy equation 20.
Ns = { x, y : (d ( x, y) < 2σd ) ∧ (d( x, y) < T50% )}
(20)
where T50% is the median value of d( x, y). 4.2.3 Texture and lightness differences integration
Assuming that texture and lightness differences complement each other in a [0,1] interval, wlightness ( x, y) = 1 − wtexture ( x, y), the integration of texture and lightness intensity differences with their corresponding validation weights is carried out by equation 21. dtexture,lightness ( x, y) = wlightness ( x, y) · dlightness ( x, y) + wtexture ( x, y) · dtexture ( x, y) ∀ x, ∀y (21) The ﬁnal segmentation for vehicle detection is produced by thresholding the dtexture,lightness ( x, y) image with the median value of [0, 1] = 0.5. 4.2.4 Morphological processing
The resultant binary image contains a number of the same unwanted elements that are present in the images produced by the Colorbased texture segmentation method (Section 3). As a result, the morphological operations used in that method are also applied here in the same fashion.
5. Experimental results In this section we present the results of applying both segmentation methods using natural scenes images. The image database used in our experiments is publicly available and it is described in MartelBrisson & Zaccarin (2007). We have proceded according to the supervised texture segmentation testing protocol of the Outex database (Ojala et al. (2002)). This framework involves the following components: • Image segmentation test suite. The test sequence is called Highway III and it includes trafﬁc ﬂow scenes and ground truth images. Ground truth images show individual pixel group of vehicles and shadows. However, we have modiﬁed these images so both the vehicle and its shadow are within the same object. • Input data in an individual problem. I is the image to be segmented with NI pixels. R I is the number of regions in I, each one with a distinct class label L R . • Required output in an individual problem. O is the output labeled image equal in size to I. • Performance metric in an individual problem. Let Iij and Oij denote the class labels of pixel (i, j) in I and O, respectively. The score of an individual problem S p is the proportion of all pixels that are correctly labelled: Sp =
100 NI
NI
∑ δ( Iij , Oij )% ij
(22)
286 14
Image Segmentation Image Segmentation
Fig. 7. First column shows the ground truth images. Second and third columns show the results of the methods described in Section 3 and Section 4, respectively.
Colorbased for Vehicle Detection Texture Image Segmentation for Vehicle Detection Colorbased Texture Image Segmentation
287 15
where δ denotes the Kronecker delta function. • Test suite score. Let P be the number of problems to be considered. If test suite contains P > 1 problems, then the scores from equations 23, 24, 25 and 26 are computed. Smean =
1 P
P
∑ Sp
(23)
p
Smin = min(S p ) ∀ p ∈ {1, ..., P}
(24)
Smax = max(S p ) ∀ p ∈ {1, ..., P}
(25)
Sdev
1 = P
P
∑(S p − Smean )2
(26)
p
We have measured the performance of the above values in both segmentation methods. Qualitative image results are shown in Figure 7 and quantitative measures over the image test suite are listed in Table 1. Segmentation based on Colorbased texture lightness and texture Image ground truth segmentation (Section 3) differences (Section 4) Smin 78.5091% 97.2174% Smax 99.1536% 99.4817% Smean 89.7853% 98.5516% Sdev 8.0121% 0.6786% Table 1. Performance scores of the two segmentation methods described in this chapter over the image test suite.
6. Discussion Given the estimators shown in table 1, particularly the mean and standard deviation, it is evident, without any further analysis or evidence beyond themselves, that the Segmentation based on lightness and texture differences method (Section 4) has achieved a better performance than the Colorbased texture segmentation method (Section 3). The Smean score measures how successful the segmentation rate is. This score is the average of the performance values of all the individual problems considered in our experiments expressed as a percentage. The Segmentation based on lightness and texture differences method achieved a Smean = 98.5516, whereas the Colorbased texture segmentation method achieved a Smean = 89.7853. Also, the Sdev score shows that the performance of the method is consistent under similar experimental conditions, as its value tends to be smaller when the dispersion from the mean (successful segmentation rate) is also small. For this score, the Segmentation based on lightness and texture differences method achieved a Sdev = 0.6786, whereas the Colorbased texture segmentation method achieved a Sdev = 8.0121. In the Segmentation based on lightness and texture differences method, the computed gradient vector’s directions have a high tolerance to noisy images and illumination changes. This
288 16
Image Segmentation Image Segmentation
method builds a background model to detect the changes of the objects of interest with respect to that reference model. The method achieves a higher accuracy rate but the borders slightly exceed the region of interest because a number of ﬁltering processes are used to make homogeneous the gradient textures. On the other hand, the Colorbased texture segmentation method is very discriminative when abrupt changes in color are present in the input images. Even though only two components are used in this method, namely 75◦ and −75◦ , calculating the DTCWT coefﬁcients still involves multiple ﬁltering of images at different coarseness levels, thus increasing the computational time cost. The results derived from the experiments of the methods presented in this chapter can be useful in the preliminary phases of a vehicle tracking system. We will include these results in our future research efforts.
7. Acknowledgements This work has been partially supported by the Secretaría de Comunicaciones y Transportes del Gobierno del Estado de Tabasco within the project "UAEM 110171: Plan Integrado de Movilidad de Villahermosa y su zona metropolitana Fase II" and the research grant PROMEP/103.5/08/3016.
8. References BarillaPérez, M. E. (2008). Colourbased Texture Image Segmentation, PhD thesis, University of Birmingham. BarillaPérez, M. E. & Spann, M. (2008). Colourbased texture image classiﬁcation using the complex wavelet transform, 5th International Conference on Electrical Engineering, Computing Science and Automatic Control. Bovik, A. & Bovik, A. (2000). Handbook of Image and Video Processing, Academic Press. Chen, C.T., Su, C.Y. & Kao, W.C. (2010). An enhanced segmentation on visionbased shadow removal for vehicle detection, pp. 679 –682. Cheng, S., Luo, X. & Bhandarkar, S. M. (2007). A multiscale parametric background model for stationary foreground object detection, IEEE Workshop on Motion and Video Computing. Cheung, S.C. S. & Kamath, C. (2004). Robust techniques for background subtraction in urban trafﬁc video, Visual Communications and Image Processing 5308(1): 881–892. Coifman, B., Beymer, D., McLauchlan, P., Malik, J. & Malik, H. (1998). A realtime computer vision system for vehicle tracking and trafﬁc surveillance. Cucchiara, R., Grana, C., Piccardi, M. & Prati, A. (2003). Detecting moving objects, ghosts, and shadows in video streams, IEEE Transactions on Pattern Analisys and Machine Intelligence 25(10): 1337–1342. DiazAlonso, J., RosVidal, E., Rotter, A. & Muhlenberg, M. (2008). Lanechange decision aid system based on motiondriven vehicle tracking, IEEE Transactions on Vehicular Technology 57(5): 2736–2746. Elgammal, A., Harwood, D. & Davis, L. (1999). Nonparametric model for background subtraction, Proc. IEEE ICCV ’99 Framerate workshop, Corfu, Greece. Eng, H.L., Thida, M., Chew, B.F., Leman, K. & Anggrelly, S. (2008). Modelbased detection and segmentation of vehicles for intelligent transportation system, pp. 2127 –2132. Fang, L. Z., Qiong, W. Y. & Sheng, Y. Z. (2008). A method to segment moving vehicle cast shadow based on wavelet transform, Pattern Recognition Letters 29(16): 2182–2188. Felzenszwalb, P. F. & Huttenlocher, D. P. (2004). Efﬁcient graphbased image segmentation, International Journal of Computer Vision 59(2): 167–181.
Colorbased for Vehicle Detection Texture Image Segmentation for Vehicle Detection Colorbased Texture Image Segmentation
289 17
Gentile, C., Camps, O. & Sznaier, M. (2004). Segmentation for robust tracking in the presence ˝ of severe occlusion, IEEE Transactions on Image Processing 13(2): 166U178. González, R. C. & Woods, R. E. (2002). Digital Image Processing, 2nd edition edn, Prentice Hall. Gutchess, D., Trajkovic, M., CohenSolal, E., Lyons, D. & Jain, A. K. (2001). A background model initialization algorithm for video surveillance, in proc. International Conference on Computer Vision, pp. 733–740. HaidarianShahri, H., Namata, G., Navlakha, S., Deshpande, A. & Roussopoulos, N. (2007). A graphbased approach to vehicle tracking in trafﬁc camera video streams, Proceedings of the 4th International Workshop on Data Management for Sensor Networks. Halevy, G. & Weinshall, D. (1999). Motion of disturbances: detection and tracking of ˝ multibody nonrigid motion, Machine Vision and Applications 11(3): 122U137. Kamijo, S., Matsuchita, Y., Ikeuchi, K. & Sakauchi, M. (2000). Occlusion robust vehicle tracking utilizing spatiotemporal markov random ﬁeld model, Proc. IEEE Intelligent Trasportation Systems, Dearborn, MI. Kastrinaki, V., Zervakis, M. & Kalaitzakis, K. (2003). A survey of video processing techniques ˝ for trafﬁc applications, Image and Vision Computing 21: 359U381. Kingsbury, N. G. (1999). Shift invariant properties of the dualtree complex wavelet transform, International Conference on Acoustics, Speech and Signal Processing ICASSP99, Phoenix, AZ. Paper SPTM 3.6. URL: citeseer.ist.psu.edu/kingsbury99shift.html Lee, D. S., Hull, J. & Erol, B. (2003). A bayesian framework for gaussian mixture background modeling, Proc. IEEE International Confererence on Image Processing (ICIP Š03). Li, L. & Leung, M. (2002). Integrating intensity and texture differences for robust change detection, IEEE Transactions on Image Processing 11: 105–112. Lo, B. & Velastin, S. (2001). Automatic congestion detection system for underground platform, Proc. International Symposium on Intelligent Multimedia. Malacara, D. (2001). Color Vision and Colorimetry, Theory and Applications, SPIE International Society for Optical Engineering, Bellingham, Washington USA. MartelBrisson, N. & Zaccarin, A. (2007). Learning and removing cast shadows through a multidistribution approach, Pattern Analysis and Machine Intelligence, IEEE Transactions on 29(7): 1133 –1146. Materka, A. & Strzelecki, M. (1998). Texture analysis methods  a review, Technical report, University of Lodz. MejiaIñigo, R., BarillaPerez, M. E. & MontesVenegas, H. (2009). Colorbased texture image segmentation for vehicle detection, 6th International Conference on Electrical Engineering, Computing Science and Automatic Control, pp. 1–6. Mo, G. & Zhang, S. (2010). Vehicles detection in trafﬁc ﬂow, Vol. 2, pp. 751 –754. Ojala, T., Mäenpää, T., Pietikäinen, M., Viertola, J., Kyllönen, J. & Huovinen, S. (2002). Outex  new framework for empirical evaluation of texture analysis algorithms. Proc. 16th International Conference on Pattern Recognition, Quebec, Canada, 1:701  706. Pang, C. C. C., Lam, W. W. L. & Yung, N. H. C. (2004). A novel method for resolving vehicle occlusion in a monocular trafﬁcimage sequence, IEEE Transactions on Intelligent ˝ Transportation Systems 5(33): 129U141. Petrou, M. (2006). Image Processing, Dealing With Texture, John Wiley & Sons Ltd. Rosin, P. (1988). Thresholding for Change Detection, Brunel University. Rousseeuw, P. & Leroy, A. (1986). Least median of squares: A robust method for outlier and model error detection in regression and calibration, Analytica Chimica Acta 187: 171–179.
290 18
Image Segmentation Image Segmentation
Sangwine, S. J. & Horne, R. E. N. (1998). The Colour Image Processing Handbook, Chapman and Hall. Selesnick, I. W., Baraniuk, R. G. & Kingsbury, N. G. (2005). The dualtree complex wavelet transform, IEEE Signal Processing Magazine pp. 123–151. SenChing, Cheung, S. & Kamath, C. (2005). Robust background subtraction with foreground validation for urban trafﬁc video, EURASIP Journal on Applied Signal Processing 14: 2330–2340. Shahri, H. H., Namata, G., Navlakha, S., Deshpande, A. & Roussopoulos, N. (2007). A graphbased approach to vehicle tracking in trafﬁc camera video streams, Proceedings of the 4th International Workshop on Data Management for Sensor Networks. Stauffer, C. & Grimson, W. (2000). Learning patterns of activity using realtime tracking, IEEE ˝ Transactions on Pattern Analisys and Machine Intelligence 22(8): 747U757. Sun, Z., Bebis, G. & Miller, R. (2006). Onroad vehicle detection: a review, Pattern Analysis and Machine Intelligence, IEEE Transactions on 28(5): 694 –711. Techawatcharapaikul, C., Kaewtrakulpong, P. & Siddhichai, S. (2008). Outdoor vehicle and shadow segmentation by temporal edge density information of adjacent frames, Vol. 1, pp. 433 –436. Trémeau, A., Tominaga, S. & Plataniotis, K. N. (2008). Color in image and video processing: most recent trends and future research directions, J. Image Video Process. 2008: 1–26. Tsai, L.W., Hsieh, J.W. & Fan, K.C. (2007). Vehicle detection using normalized color and edge map, Image Processing, IEEE Transactions on 16(3): 850 –864. Wang, C.C. R. & Lien, J.J. (2008). Automatic vehicle detection using local features  a statistical approach, Intelligent Transportation Systems, IEEE Transactions on 9(1): 83 –96. Wren, C., Azarbayejani, A., Darrell, T., & Pentland, A. (1997). Pﬁnder: realtime tracking of the human body, IEEE Transactions on Pattern Analisys and Machine Intelligence ˝ 19(7): 780U785. Yilmaz, A. (2006). Object tracking: A survey, ACM Computing Surveys 38: 1–45. Zhang, H. & Yuan, F. (2007). Vehicle tracking based on image alignment in aerial videos, Energy Minimization Methods in Computer Vision and Pattern Recognition. Zhang, W., Wu, Q. M. J., Yang, X. & Fang, X. (2008). Multilevel framework to detect and handle vehicle occlusion, IEEE Transactions On Intelligent Transportation Systems 9(1): 161–174. Zhou, Q. & Aggarwal, J. (2001). Tracking and classifying moving objects from videos, Proc. 2nd IEEE Workshop on Performance Evaluation of Tracking and Surveillance (PETS Š01). Zhu, H., Li, M., Zhu, Y. & Ni, L. M. (2009). Hero: Online realtime vehicle tracking, IEEE Transactions on Parallel and Distributed Systems 20: 740–752.
Part 3 Image Segmentation Applications
15 An Enhanced Level Set Algorithm for Wrist Bone Segmentation Donatello Conte, Pasquale Foggia, Francesco Tufano and Mario Vento Università degli Studi di Salerno Italy 1. Introduction In the last decades there was a growing interest in designing CAD devices for medical imaging: the main role of these system is to acquire the images, generally TAC or RMI, and to display the parts of interest of human body on suitable visual devices, after some preelaboration steps aimed to improve the quality of the obtained images. A TAC or a MRI sequence, obtained as a result of a scan process of the interested parts of the human body is a generally wide set of 2D graylevel images, seen as the projection of the body into three different coordinate planes. Starting from these sequences it is rather difﬁcult, for the radiologist, to imagine the whole appearance of the body parts, since without a 3D model of each part it is only possible to browse the images in the three planes independently. In this framework, the most challenging task remains that of extracting, from the whole images, a 3D model of the different parts; such a model would be important not only for visualization purposes, but also for obtaining quantitative measurements that could be used as an input to the diagnostic process. Even if the preprocessing step of the acquired images plays a key role in the achievement of a good visualization quality, the literature is today so rich of papers describing procedures aimed to increase the signal/noise ratio that this problem can be now considered as deﬁnitely solved. So, the attention of researchers is nowadays concentrated on the deﬁnition of robust methods for the 3D segmentation. In the case of Magnetic Resonance Images (MRI), the segmentation is made complex by the unavoidable presence of inhomogeneity in the images, as well as the presence of image distortions. Despite the research efforts and the signiﬁcant advances achieved in recent years, the image segmentation problem still remains a notoriously known challenging problem, especially in the case of poor quality images. In particular, the segmentation of MR images is made even more complex, by the complexity of the shapes of the parts to be segmented, and by the lack of suitable anatomical models able to fully capture all the possible shape variations for each of them. These models can provide, if suitably exploited, important information: for bone tissues it is relevant the knowledge about the shape and the size of the synovial parts, devoted to connecting bones: their characterization allows the scientist to choose the most appropriate technique for a correct segmentation. Namely, synovia appears in the images as a darker surface surrounding the bones; its presence is fundamental for the correct segmentation, since often the bone tissues and the adjacent cartilagineous tissues have similar intensity levels, and would be indistinguishable without the synovia.
294 2
ImageWillbesetbyINTECH Segmentation
a)
b)
Fig. 1. The effect of the similarity threshold on region growing segmentation. a) A threshold too low produces an oversegmentation. b) A threshold too high produces an undersegmentation. The simplest segmentation approaches are those based on a thresholding applied to each pixel (or voxel) on its intensity values: they result to be generally uneffective in the case of an MR image: the segmentation results are in fact very poor, as the intensity levels of the pixels of bones and the synovia are quite similar, so causing the undersegmention of some regions contemporarily to oversegmented areas. A threshold, appropriate for reaching the solution all over the image is practically impossible to be determined. More complex solutions, based on partitioning of the image in different parts, on which different values of the thresholds, in the experience of the authors, can perform better but the results are far from satisfying the radiologist. An other class of segmentation algorithms are those based on the well known region growing paradigm. A point, surely belonging to the area to be segmented, is given as input by the user, and considered as a seed: the pixels adjacents to the seed are considered as belonging to the region to be segmented, if their intensity values are similar to that of the seed: the similarity is suitably deﬁned on the application domain and generally a threshold is applied to determine wether two pixels are similar or not. The process is iterated for the last added pixels until, at the given step, no further pixel is added to the region. Although different variants of this class of algorithms have been developed along the years, their rationale is that of expanding the regions on the basis of their homogeneity. Their application in all the cases in which foreground and background have little gray level differences can results in oversegmentation problems. Figure 1 highlights these effect on a wrist bone, with two different similarity thresholds, so demonstrating the difﬁculty of obtaining effective results in a practical case. More recently, some approaches, based on the attempt of facing the segmentation approach by a classiﬁcation system, have been proposed. The rationale of these methods is aimed to obtaining algorithms able to work without the interaction with the radiologist: they perform the training of the classiﬁer on a suitably built training set of pixels and, once adequately trained, classify the pixels of the image as belonging to a foreground area or to the background. In this way, the interaction with the radiologist, if any, is required in the training phase. The simplest implementations of this class of methods is based on the knearestneighbor, as
An Enhanced Level Set Algorithm for Wrist Bone Segmentation for Wrist Bone Segmentation An Enhanced Level Set Algorithm
295 3
in Vrooman et al. (2007), where brain tissues are segmented; the kNN classiﬁer is trained automatically using an a priori model of the brain provided by a brain atlas. Another approach of this kind is based on Bayes classiﬁers Banga et al. (1992); in particular in the cited article the segmentation of the retina is performed using an unsupervised Bayesian classiﬁer whose parameters have been estimated using the ExpectationMaximization algorithm. In spite of their simplicity and their low computational cost, their intrinsic nature does not allows them to take into account spatial information, so making them unproﬁtable in all those cases in which the latter information is crucial for the ﬁnal result, as in the case of bone tissues. A further class of segmentation algorithms are those based on (unsupervised) clustering techniques. The three most used clustering techniques are the Kmeans, the Fuzzy Cmeans and the ExpectationMaximization algorithm. An example of the use of Kmeans is Vemuri et al. (1995), that performs the segmentation of brain MR images by clustering the voxels on the basis of waveletderived features. Two papers using the Fuzzy Cmeans clustering are Ardizzone et al. (2007), that is also applied to brain MR images, and Foggia et al. (2006), that is applied to mammographic images. Finally, in Wang et al. (2005) a clustering method based on the ExpectationMaximization algorithm is used for segmenting brain images showing a greater robustness with respect to the noise due to ﬁeld inhomogeneity. The algorithms discussed so far assume that the intensities of each voxel class are stationary: this assumption does apply only on limited sets of images, due to the intrinsic heterogeneity of a class, the nonuniform illumination, or other imaging artifacts. So, to take into account spatial information, recently some approaches based on the use of the Markov Random Field (MRF) Models have been used, as in Ruan & Bloyet (2000) and Krause et al. (1997). The idea behind them is that, in the case of biomedical images, the probability of a pixel to belong to a class is strongly related to the values of the surrounding pixels, as rarely the anatomical parts are composed by just one pixel. Two critical points of MRF approach are the computational burden (due to the required iterative optimization schemes) and the sensitivity of the results to the model parameters. The most used approach in segmentation of medical images is the level set (Cremers et al. (2005)), based on an optimization apprach. A segmentation of the image plane Ω is computed by locally minimizing an appropriate energy functional E(C ) by evolving the contour C of the region to be segmented starting from an initial contour. In general, method based on this approach may use either an explicit (parametric) or implicit representation of the contours. In explicit representations (Leitner & Cinquin (1991), McInerney & Terzopoulos (1995)) – such as splines or polygons – a contour is deﬁned as a mapping from an interval to the image domain: C : [0, 1] → Ω. In implicit contour representations (Dervieux & Thomasset (1979), Osher & Sethian (1988)), contours are represented as the (zero) level line of some embedding function φ : Ω → : C = { x ∈ Ω  φ ( x ) = 0}. In the original level set algorithm, only gradient information is taken into account in the energy term E(C ). Some authors (Osher & Santosa (2001), Chan & Vese (2001), Russon & Paragios (2002)) have proposed improvements of the classical algorithm by introducing some priors information (e.g. shape, color or motion information). Level set algorithms are widely used in medical images segmentation because they are very effective. However they present some drawbacks: • The segmentations obtained by a local optimization method are strongly dependent to the initialization. For many realistic images, the segmentation algorithm tends to get stuck in
296 4
ImageWillbesetbyINTECH Segmentation
a)
b)
Fig. 2. The effect of the seed point on Level Set segmentation. Between a) and b), a slight change of the seed point determines a different segmented region shape (the seed point in the image is indicated by the starlike cursor). undesired local minima (especially in the presence of noise) forcing the user to try with several seed points before obtaining a satisfactory solution. • This approach lacks a meaningful probabilistic interpretation. Extensions to other ˝ such as color, texture or motion U ˝ are not straightforward. segmentation criteria U • This algorithm has a problem in ﬁnding correct contours of the regions when the region boundaries have corners or other singularities. In a recent paper (Conte et al. (2009)) we presented a new algorithm that overcomes the ﬁrst of the considered problems. In this paper we propose a signiﬁcant improvement, especially with respect to the last problem (that is still an open problem in the literature). The paper is organized as follows: in section 2 a review of the most used approaches for segmenting MR images is shown; the proposed algorithm is presented in section 3 while in section 4 the experimental phase together with the analysis of the results are described. Section 5 summarizes the conclusions obtained from our work.
2. Important Manuscript must contain clear answers to following questions: What is the problem / What has been done by other researchers and where you can contribute / What have you done / Which method or tools you used / What are your results / What is new and good, what is not good / Future research
3. The proposed method As we have seen, every segmentation approach has its strenght and its weak points. Our proposal is to base the segmentation on the integration of two complementary approaches: region growing and level set segmentation. Region growing has problems with local noise, especially on the boundary of the region to be segmented, and has a strong dependency on a similarity threshold, leading to either oversegmentation (if the threshold is chosen conservatively) or undersegmentation (if the threshold is chosen to capture as much as possible the shape of the region). But neither of those problems affect the level set algorithm.
An Enhanced Level Set Algorithm for Wrist Bone Segmentation for Wrist Bone Segmentation An Enhanced Level Set Algorithm
297 5
On the other hand, level set has a strong dependency on the choice of the seed point, as shown in ﬁg. 2, and also may have problems where the region boundary has some singularity (e.g. a sharp corner). Region growing instead is fairly immune to both those problems. So, region growing and level set segmentation appear to complement each other with respect to their strenght, and this is the reason why we have chosen to combine them into a technique that should overcome the limitations of both. Basically, our method is composed of the following steps: • ﬁrst, a smoothing of the image is performed using a low pass ﬁlter; this step is related to the use of the Laplacian Level Set variant of the level set technique, as we will discuss later; • then a presegmentation is realized using region growing, to obtain a rough, conservative estimate of the region; • the result of the presegmentation is used to initialize the proper segmentation, performed by means of a level set algorithm; in this way the result of the level set algorithm is not dependent on the choice of the seed point; • ﬁnally, a local contour reﬁnement, based again on region growing, is applied in order to better ﬁt the contour to sharp corners and other singularities. Each of these steps will be described with more detail in the following subsections. As an illustration of the different steps, we will present their effect on an example image, shown in ﬁgure 3. 3.1 Smoothing ﬁlter
The region growing technique used for the presegmentation step is highly sensitive to pixellevel noise. So it is important to remove this kind of noise before the presegmentation. Moreover, for the proper segmentation step, we have used a variant of the level set technique called Laplacian Level Set (LLS), introduced in Conte et al. (2009). The LLS algorithm performs a Laplacian ﬁlter on the image to enhance the boundaries of the regions; but a side effect of the Laplacian ﬁlter is a magniﬁcation of the highfrequency noise components. Hence, the denoising is important also for the LLS segmentation. In order to remove the noise we have used a Gaussian smoothing ﬁlter, which is a well known lowpass ﬁlter widely used in the image processing ﬁeld. The use of a lowpass ﬁlter may seem contradictory with the goals of a segmentation algorithm: if the algorithm has to determine the sharp edges that form the boundary of the regions, it may be thought that by smoothing those very edges should make the task of the algorithm more complicated. However, the following factors should be considered: • by carefully choosing the ﬁlter cutoff frequency, the ﬁlter can cancel out only the intensity variations that are due to noise, while the ones due to the boundaries between regions will only be a little bit blurred • the presegmentation process needs only to ﬁnd a reasonable approximation of the region, so it can easily be tuned to be unaffected by the blurring of the region boundary; on the other hand it greatly beneﬁts from the reduction of the pixel level noise achieved by the lowpass ﬁlter • the proper segmentation process will apply a laplacian ﬁlter to the image; the net effect of the combination of the lowpass and laplacian ﬁlter is that of a band pass ﬁlter that, by virtue of the choosen cutoff frequency, will preserve exactly the variations whose spatial frequency correspond to the boundaries between the regions.
298 6
ImageWillbesetbyINTECH Segmentation
a)
b) Fig. 3. An example image, that will be used to illustrate the different steps of the proposed algorithm. What is actually shown is a 2D slice of the 3D MR image. a) The original image. b) A zoomed image of the bone that will be the target for the segmentation.
An Enhanced Level Set Algorithm for Wrist Bone Segmentation for Wrist Bone Segmentation An Enhanced Level Set Algorithm
299 7
Fig. 4. The result of the application of the Gaussian ﬁlter to the image of ﬁg. 3b. The Gaussian ﬁlter introduces a parameter σ, related to the cutoff frequency, that needs to be tuned for obtaining an adequate performance. However the optimal value of σ depends only on the resolution of the image and the size of the smallest features of interest in the segmented region. Hence, for a given MRI machine and anatomical district, the tuning of σ has to be performed only once. Figure 4 shows the effect of the gaussian ﬁlter on our example image. 3.2 Image presegmentation
The level set technique starts with a tentative contour of the region to be segmented, and makes this contour evolve so as to reach a (local) minimum of a suitably deﬁned energy function. The usual approach for initializing the contour is to choose a small sphere around the user selected seed point. However, especially if the shape of the target region is complex, starting with a contour that is so different from the desired one may easily lead the algorithm to a local minimum that does not correspond to the ideal segmentation. Furthermore, this local minimum strongly depends on the choice of the seed point, making it difﬁcult to have a repeatable result for the segmentation process. On the other hand, if the level set algorithm starts from a tentative contour that is reasonably close to the true boundary of the region, it usually converges without problems to the desired minimum of the energy function. In order to provide such an initial contour, our method performs a presegmentation step. In this step, our system attempts to segment the region of interest using a region growing technique Adams & Bischof (1994). In region growing, basically, the algorithms starts with a
300 8
ImageWillbesetbyINTECH Segmentation
Fig. 5. The result of the presegmentation on the image of ﬁgure 3b tentative region formed by the seed point alone; and then repeatedly add adjacent voxels as long as their intensity is within a threshold θ from the average intensity of the region built so far. The tuning of θ is one of the most delicate aspects of region growing, since a value too tight will not make the algorithm cover the whole region (oversegmentation), while a value too loose would cause extra parts to be included in the region (undersegmentation). However, since we are using region growing only as a presegmentation step, we do not need to ﬁnd the optimal value for θ. We just need to be sure to “err on the safe side”, in the sense that the algorithm should not produce an undersegmentation. This is necessary because the level set algorithm can expand the contour, but cannot contract it. So, also the tuning of θ can be done once for a given MRI machine/anatomical district combination, instead of adjusting this parameter for each different image. As an alternative to region growing we have also tried the fast marching technique Zhang et al. (2007) for presegmentation. The results of both algorithms are similar, but fast marching is slower than region growing, and has more parameters to be tuned. Hence, we have decided to adopt region growing. The presegmentation of our example image is shown in ﬁgure 5. 3.3 Laplacian Level Set
The current trend in MR imaging is towards the reduction of the intensity of the magnetic ﬁeld to which the patient is exposed, in order to obtain a reduction in the costs but also in the weight and and space occupied by the MRI machines. At the same time, the acquisition time
An Enhanced Level Set Algorithm for Wrist Bone Segmentation for Wrist Bone Segmentation An Enhanced Level Set Algorithm
Fig. 6. The result of the Laplacian ﬁlter applied to the example image of ﬁgure 3b.
Fig. 7. The result of the Laplacian Level Set segmentation applied to the example image of ﬁgure 3b.
301 9
302 10
ImageWillbesetbyINTECH Segmentation
has to be kept short, since during the scan the patient has to remain still, and so the MRI exam would be too uncomfortable if it were too long. As a result, the contrast between different tissues is often quite low, and this can cause problems to any segmentation algorithm. In order to overcome this issue, in Conte et al. (2009) the so called Laplacian Level Set (LLS) algorithm has been proposed. The LLS algorithm is based on the use of the Laplacian ﬁlter, deﬁned as:
∇2 f ( x, y, z) =
δ2 f ( x, y, z) δ2 f ( x, y, z) δ2 f ( x, y, z) + + 2 2 δx δy δz2
where f ( x, y, z) is the intensity of the voxel at position ( x, y, z). Actually, a discrete approximation of the ﬁlter is used. The ﬁltered image enhances the contours of the regions, as exempliﬁed (on a 2D slice of the image) in Fig. 6. The algorithm operates on the ﬁltered image, starting with a contour surface C that is initialized as the contour of the region found in the presegmentation step, and evolving it in order to minimize an energy function E(C ) which is deﬁned as: E(C ) =
+
+k
Vin (C )
(u( x, y, z) − μin (C ))2 dxdydz +
Vout (C )
(u( x, y, z) − μout (C ))2 dxdydz +
C  Vin (C )
(1)
where • Vin (C ) is the region inside C • Vout (C ) is the region outside C • u( x, y, z) is the intensity of the ﬁltered image • μin (C ) and μout (C ) are the average value of u( x, y, z) over Vin (C ) and Vout (C ) respectively • C /Vin (C ) is the ratio between the area of C and the volume of Vin (C ), that acts as a regularization factor for the contour; this factor is weighted by the parameter k: a larger value for k makes the algorithm oriented towards smoother contours and more robust to noise, but also less able to follow sharp corners on the region boundary The result of the LLS algorithm on our example image is presented in ﬁgure 7. 3.4 Local contour reﬁnement
The last step of the algorithm starts with the contour C found by the Laplacian level set and tries to reﬁne it to better ﬁt the sharp corners that are usually smoothed out by the level set. Namely, this reﬁnement is performed using a limited form of region growing, that has no problem in following sharp corners. Since region growing is prone to undersegmentation, especially in low contrast images such as the ones produced by MRI, special care is taken in the deﬁnition of the stopping criterion to ensure that only small corrections to the contour C are performed. More formally, the voxels adjacent to C are examined and are added to the contour iff:
 f ( x, y, z) − f (n( x, y, z, C )) < θ /d( x, y, z, C ) where:
(2)
An Enhanced Level Set Algorithm for Wrist Bone Segmentation for Wrist Bone Segmentation An Enhanced Level Set Algorithm
303 11
Fig. 8. The result of the local contour reﬁnement applied to the segmentation shown in ﬁgure 7. • f ( x, y, z) is the intensity value of the considered voxel of coordinates ( x, y, z) • n( x, y, z, C ) is the position of the voxel in C that is the nearest to ( x, y, z); so f (n( x, y, z, C )) is the intensity of this voxel • d( x, y, z, C ) is the Euclidean distance between ( x, y, z) and n( x, y, z, C ) • θ is a suitably deﬁned threshold The division of the threshold θ by d( x, y, z, C ) ensures that this reﬁnement step will never extend too much the initial contour C. In particular, this extension is performed on a local basis, only where the adjacent pixels are very similar to the ones already in C. Figure 8 shows the effect of the local contour reﬁnement algorithm on our example image, starting from the segmentation presented in ﬁgure 7.
4. Experimental results The algorithm has been tested on 11 MRI sequences of wrist bones acquired at low magnetic ﬁeld, for a total of 762 bidimensional slices. The ground truth has been manually traced by medical experts. We compare the proposed enhanced laplacian level set algorithm (ELLS) with the following algorithms: • our previous algorithm (Laplacian Level Set, LLS); • the basic level set (BLS) • the basic level set with presegmentation module (PLS) • Geodesic Active Contours (see Caselles et al. (1997) and Yan & Kassim (2006)); Geodesic Active Contours (GAC) algorithms are similar to Level Set algorithms, but the ﬁrst are motivated by a curve evolution approach and not by an energy minimization one;
304 12
ImageWillbesetbyINTECH Segmentation
• Geodesic Active Contours with presegmentation module (PGAC). The choice of Geodesic Active Contours for the comparison is motivated by the fact that also this family of algorithms, like our method, is fairly robust with respect to the choice of the seed point. To evaluate the results of the proposed algorithm we used the precision, recall and findex indices so deﬁned: TP TP + FP TP recall = TP + FN 2 · precision · recall findex = precision + recall precision =
where TP is the number of correctly detected objects of interest, FP is the number of wrongly detected objects of interest and FN is the number of missed objects of interest. The most commonly used deﬁnition of these indexes is directly usable for applications where the objects of interest are either completely detected or completely missed. In our application, however, the objects of interest are not atomic regions, so we need to consider also partial recognition of the tissue of interest. For this reason we have redeﬁned TP, FP and FN in a fuzzy sense as follows: TP = FP =
 g ∩ d  g ∪ d
d − d ∩ g d
 g − d ∩ g  g where g is the set of voxels actually belonging to the region of interest (ground truth), d is the set of voxels detected by the algorithm and · denotes the cardinality of a set. It is simple to show that when the object of interest is perfectly detected (in the sense that all the voxels in the ground truth are detected, and no voxel outside of the ground truth is detected), then precision = 1 and recall = 1; on the other hand, if the algorithm detects voxels that do not belong to the ground truth, it will have precision < 1, and if the algorithm misses some of the voxels in the ground truth, it will have recall < 1. In the following table we report the results, averaged over the 11 MRI sequences: FN =
Precision Recall findex BLS PLS GAC PGAC LLS ELLS Table 1. Experimental Results
0.81 0.92 0.95 0.99 0.99 0.98
0.89 0.94 0.89 0.90 0.94 0.97
0.85 0.93 0.92 0.94 0.96 0.97
An Enhanced Level Set Algorithm for Wrist Bone Segmentation for Wrist Bone Segmentation An Enhanced Level Set Algorithm
305 13
Notice that for the BLS algorithm we had to perform the test changing both the seed point and the value of the parameter k of the Level Set energy function, since the algorithm did not provide adequate results for all the images with a single choice of these parameters. The other algorithms did not have this problem. It is important to remark that the idea of the presegmentation phase allows also the level set algorithm to overcome this problem. Furthermore, the results presented in table 1 show that the Laplacian operator provides an important contribution to the improvement of the performance. Indeed, the two algorithms based on this operator (LLS and ELLS) achieve the best overall performance. Algorithms that exhibit an undersegmenting behavior can be expected to obtain a low value of the precision index. This is indeed the case of the BLS algorithm, as it is evident from table 1. On the other hand, algorithms with a tendence to oversegmentation attain a low value of the recall index. As shown in table 1, this happens for BLS, GAC and PGAC. Notice that the BLS algorithm can yield (depending on the input image) both an undersegmentation and an oversegmentation. Table 1 shows that Geodesic Active Contours based approaches have a relatively low recall value. From a detailed analysis of the images it can be concluded that the while the boundary of the region is usually well approximated, the low recall is due to the fact that often these algorithms miss voxels that are internal to the region. In conclusion, Table 1 shows that our approaches are more effective than all the others. In particular, the ELLS algorithm shows a signiﬁcant improvement in the recall index. A good recall index (together with a good precision) is important for applications that use the segmentation as the basis for quantitative measurements, e.g. for diagnostic purposes. To have a visual idea of the effectiveness of our proposed algorithm, in Fig. 9 the results of each segmentation algorithm are shown. Note that the result of the PLS algorithm, even after a difﬁcult calibration phase, is not able to avoid the undersegmentation problem. Also notice that the result of the PGAC algorithm presents some holes within the tissue. The comparison between ﬁg. 9f and ﬁg. 9g, and between ﬁg. 9f and ﬁg. 9g, demonstrates how the new ELLS algorithm is able to segment correctly the sharp corners of the tissue, overcoming the problems of our previous method.
5. Conclusion In this paper we propose a novel segmentation method for MRI images, that is based on the integration of two complementary techniques: region growing and level set segmentation. Each technique is used at a different stage of the segmentation process, and the results are combined in such a way as to obtain a ﬁnal segmentation that is not affected by the problems and limitations of both techniques when used alone. The new method is robust with respect to the choice of the initial seed and to the setup of the (few) parameters, yielding repeatable results; furthermore, its performance is high in terms of both the precision and recall indices, as we have demonstrated experimentally, resulting appropriate for Computer Aided Diagnosis applications that need accurate quantitative measurements.
306 14
ImageWillbesetbyINTECH Segmentation
a)
b)
c)
d)
e)
f)
g)
Fig. 9. An example of the segmentation obtained by the tested algorithms (only a 2D slice is presented). a) Original image. b) A zoomed image of the region of interest. c) Basic Level Set segmentation (BLS). d) Level Set with Presegmentation (PLS). e) Geodesic Active Contours (GAC). f) Laplacian Level Set (LLS). g) Extended Laplacian Level Set (ELLS).
An Enhanced Level Set Algorithm for Wrist Bone Segmentation for Wrist Bone Segmentation An Enhanced Level Set Algorithm
307 15
6. References Adams, R. & Bischof, L. (1994). Seeded region growing, IEEE Transaction on Pattern Analysis and Machine Learning 16(6): 641–647. Ardizzone, E., Pirrone, R. & Gambino, O. (2007). Fuzzy CMeans Segmentation on Brain MR Slices Corrupted by RFInhomogeneity, SpringerVerlag, chapter 48, pp. 378–384. Banga, C., Ghorbel, F. & Pieczynski, W. (1992). Unsupervised bayesian classiﬁer applied to the segmentation of retina image, IEEE Proceedings of the Annual International Conference of the Engineering in Medicine and Biology Society 5: 1847–1848. Caselles, V., Kimmel, R. & Sapiro, G. (1997). Geodesic active contours, International Journal of Computer Vision 221: 61–79. Chan, T. & Vese, L. (2001). Active contours without edge, IEEE Transaction on Image Processing 10(2): 266–277. Conte, D., Foggia, P., Tufano, F. & Vento., M. (2009). Evaluation and improvement of the level set method for m images segmentation, International conference on Computer Vision Theory and Application. Cremers, D., Rousson, M. & Deriche, R. (2005). Review of statistical approaches to level set segmentation: Integrating color, texture, motion and shape, International Journal on Computer Vision 19(10): 1–35. Dervieux, A. & Thomasset, F. (1979). A ﬁnite element method for the simulation of raleightaylor instability, Springer Lecture Notes in Mathematics 771: 145–158. Foggia, P., Guerriero, M., Percannella, G., Sansone, C., Tufano, F. & Vento., M. (2006). A graphbased method for detecting and classifying clusters in mammographic images, in D.Y. Y. et al (ed.), Lecture Notes in Computer Science, Vol. 4109, SpringerVerlag, Berlin, pp. 484–493. Krause, B., Wells, W., Kikinis, R., Held, K. & Kops, E. (1997). Markov random ﬁeld segmentation of brain mr images, IEEE Transaction on Medical Imaging 16(6): 878–886. Leitner, F. & Cinquin, P. (1991). Complex topology 3d objects segmentation, SPIE Conference on Advances in Intelligent Robotics Systems. McInerney, T. & Terzopoulos, D. (1995). Topologically adaptable snakes, 5th International ˝ Conference on Computer Vision, IEEE Comp. Soc. Press, pp. 840U–845. Osher, S. J. & Sethian, J. A. (1988). Fronts propagation with curvature dependent speed: ˝ Algorithms based on hamiltonUjacobi formulations, Journal of Computational Physics ˝ 79: 12U–49. Osher, S. & Santosa, F. (2001). Level set medhods for optimization problems involving geomertry and constraints i. frequencies of a twodensity inhomogeneous drum, Journal of Computational Physics 171: 272–288. Ruan, S. & Bloyet, D. (2000). Mrf models and multifractal analysis for mri segmentation, IEEE International Conferece on Signal Processing. Russon, M. & Paragios, N. (2002). Shape priors for level set representation, Lecture Notes in Computer Science 2351: 78–92. Vemuri, B. C., Rahman, S. M. & Li, J. (1995). Multiresolution adaptive kmeans algorithm for segmentation of brain mri, ICSC ’95: Proceedings of the Third International Computer Science Conference on Image Analysis Applications and Computer Graphics, SpringerVerlag, pp. 347–354. Vrooman, H. A., Cocosco, C. A., van der Lijn, F., Stokking, R., Ikram, M. A., Vernooij, M. W., Breteler, M. M. & Niessen, W. J. (2007). Multispectral brain tissue segmentation using automatically trained knearestneighbor classiﬁcation, NeuroImage 37(1): 71–81.
308 16
ImageWillbesetbyINTECH Segmentation
Wang, J., Qiu, M., Papademetris, X. & Constable, R. T. (2005). Brain tissue segmentation based on corrected grayscale analysis, Proceedings on Engineering in Medicine and Biology. Yan, P. & Kassim, A. A. (2006). Segmentation of volumetric mra images by using capillary active contour, Medical Image Analysis 10: 317–329. Zhang, H., Zhang, J., Cao, J., Wang, W., Gong, J. & Wang, X. (2007). Robust fast marching method based on anisotropic diffusion, Third International Conference on Natural Computation, Vol. 3, pp. 159–162.
16 Mineral Grain Boundary Detection With Image Processing Method: From Edge Detection Operation To Level Set Technique Bibo Lu , Weixing Wang School of Computer Science and Technology, Henan Polytechnic University, Jiaozuo China 1. Introduction The recognition of grain boundaries in deformed rocks from images of thinsections or polished slabs is an essential step in describing and quantifying various features and analysis, which is usually achieved by image processing procedures. Recently, researchers from geoscience and computer science focused on this issue and many methods have been proposed. We begin with an application of mineral grain boundary detection of thin section in geoscience: strain analysis[1]. Strain analysis plays an important role in the study of structural geology, especially when investigating the tectonic history of a region(see [2] and [3]). The following scheme illustrates main steps of strain analysis. The critical step in the above scheme
is identifying mineral with image segmentation techniques based on edges or regions. High sample intensity is required to provide adequate raw data for precious strain analysis. But in recent publications, the samples used for analysis seem to be insufﬁcient. The primary reason maybe is the laborious and timeconsuming methods to obtain the raw data required for strain analysis[4]. Though many methods based on the technology of image processing have been proposed, there is still much room for improving in efﬁciency and accuracy. In image processing and computer version, segmentation or boundary detection is still a challenging problem and is motivated by some new certain applications. As a interdisciplinary problem, grain boundary detection provides much room to improve well established method to tackle with the special situations in geosciences. Three aspects should be paid more attention when designing the new approaches. The two of them are derived from image capture, and the other is the features of objects to be identiﬁed. • Light phenomenon: under planepolarised light, many mineral grains appear as colorless; under crosspolarised light, mineral grains show varying colors and intensities. In crosspolarized light the interference color displayed depends on the mineral type, the orientation of the indicatrix of the grain with respect to the polarizers and the thickness of the thin section.
310 2
Image Segmentation Intech
• Rotating stage: grains may appear different light with different angels. To get a better visual effect, thin section must be rotated and several images are captured. Each of these image contains the important available for image segmentation. But how many images should be collected and how to fuse the information from these image have been completely answered yet. • Grain features: every kind of grains has its own special shape features. However, in a thin section image, some grains are with a certain orientation, which is determined by the way how it is cut and polished. These features provide important cues for image segmentation and data match. In the previous publications, various techniques have been reported for some certain grains identify. Threshold is used to identify grains of thin section image, while interview is often necessary to select different thresholds to separate several kinds of mineral. Edge detection has a simple control structure and provides more precise location of grain boundaries, but cannot guarantee closure of boundaries. If a boundary is discontinuous, pixels belonging to adjacent grains may be connected and therefore identiﬁed as belonging to a single mineral grain. Thus, grain identiﬁcation will be ﬁnished completely. Several techniques, such as heuristic search and artiﬁcial neural net can be used to correct such edge detection errors. Instead of directly identifying the grain boundary, region based method is proposed to identify a region, i.e. a set of points that purportedly belong to the clast. Regionbased segmentation uses image features to map individual pixels to sets of pixels that correspond to objects. Closed boundaries are always segmented but the positions of the boundaries may not be as accurate as those obtained from edge detection. Seeds are always selected by human interaction, which can be replaced by an automated complex algorithm. Holes may appear in the identiﬁed region, but for extracting features like major and minor axes, orientation, centroid, etc., the presence of these holes has a far diminished role to play than incorrectly identiﬁed edges. Watershed usually produce oversegmentation results for boundary detection. An important tool of postprocessing, it is often used to separate touching grains in binary map created by other arithmetic. In the past two decades, methods based on the partial differential equation (PDE) have been widely used in image segmentation and other image science ﬁelds recently. Impressive effects have been obtained with various PDE models. In [5] and [6] , Chan and Verse proposed a successful segmentation model (CV model) using level set, which derived from the classical active contour(snake) model[7]. The essential idea of the snake model is to evolve an initial contour governed by a PDE until it stops on the edge of the objects. To represent the evolving curve, level set technique is adopted. Level set method is based on the description of the curve as the zero crossing of a higherdimensional function and allows major simpliﬁcations. It offers a nature representation for the contour of the object, which can deal with complicated structures with many advantages, especially when the curve undergoes complex topological change. The key of level set method is to identify grain boundaries and to represent them as closed outlines. A framework of level set for mineral boundary detection was reported recently[8]. Level set method identiﬁes all the objects simultaneity instead of identifying all the interesting objects chosen by clicking them using mouse one by one. In this paper, we present a framework for boundary detection with level set for different kinds of input images. After reviewing related work, we ﬁrst introduce the the level set for boundary detection with a single gray scale image as input. Then level set for color image is presented. For processing two color polarising images as input, a novel energy functional with a curve represented with level set is constructed and a new mathematical model for mineral
Mineral Grain Boundary Detection With Image Processing Method: Edge Detection Operation Level SetSetTechnique Mineral Grain Boundary DetectionFrom With Image Processing Method: From Edge Detection To Operation To Level Technique
311 3
identiﬁcation is obtained. The curve evolves driven by the structures of two color polarising images and stops at the region edge of the grains.
2. Previous work: Brief view Boundary detection of mineral from thin section image is not a trivial task. Various method, including threshold, gradient operator, region grow, watershed, level set, artiﬁcial neural network or a combination of them, have been proposed to address this problem. We give a detailed review of previous work in order of publication. Lumbreras and Serrat considered the segmentation of grains in digital image of thin marble section[10]. An oversegmentation of the image with watershed method was performed and a regionmerging procedure was carried out with some parameters determined by a sequence of images of the same sample with polarized light. In [11], polished rock samples were scanned by a color image scanner and an automatic mineral classiﬁcation approach was presented. Fueten devised a computercontrolled rotating polarizer stage for the petrographic microscope. In this pioneer work, Fueten presented an important system allowed a thin section to remain ﬁxed while the polarizing ﬁlters were rotated by stepper motors. This approach permitted a better integration between the processing software and the microscope and hence better data gathering possibilities. In 1998, Goodchild and Fueten proposed a boundary detection procedure which calculated closed boundary with a series of image processing routines[12]. A color RGB image was converted into a gray intensity image and seven steps were performed to produced accurate and closed edges for mineral grains. Whereas this algorithm was not perfect effective it presented a signiﬁcant improvement over existing routines of that time. In fact, only intensity information was utilized in the original publication and some of color information was missed. The work of Nail and Murthy in [13], in which they constructed a standardization of edge magnitude in color image, may be helpful to improve the algorithm in [12]. To depict the geometrical structures of rock, boundary was deﬁned as pixel with high gradient and fragmentation were rebuilt and reconnected ro form an uninterrupted boundary net[14]. Orientation contrast (OC) images represent a useful starting point to develop an automated technique able to assess grain boundaries in a completely objective and reproducible way. The method in [15] deﬁned boundaries as high brightness gradient features on an OC image of a quartz mylonite through a speciﬁcally designed sequence of detection and ﬁlter algorithms that minimize the effect of local background noise. The initial boundaries produced by edge detection methods were with many imperfections. They employed a detectionﬁltering algorithm to automatically rebuild the real boundary net. When quantifying microstructures of ﬁne grained carbonate mylonites, manually setting the threshold value was adopted to select as much of the dark grainboundary area as possible without selecting the grey values of the grain interiors[16]. Heilbronner presented a simple procedure for creating grain boundary maps in a crystalline aggregate from multiple input images: for each image of a given input set, only the most signiﬁcant grain boundaries were detected with gradient ﬁltering and by combining those, a single grain boundary map was obtained[17]. Thompson, Fueten and Bockus used an artiﬁcial neural network(ANN) for the classiﬁcation of minerals[18]. Based on a set of seven primary images during each sampling, a selected set of parameters were estimated and a threelayer feed forward neural network was trained on manually classiﬁed mineral samples. This is a beginning of ANN for mineral classiﬁcation. Ross, Fueten and Yashkir proposed an automatic mineral identiﬁcation with another important evolutionary computation: genetic algorithm[19]. Touching grains in digital images of thin sections is a hot potato when considering segmentation. Van den Berg and his collaborators tried to deal with it with a
312 4
Image Segmentation Intech
algorithm by separating touching grain sections in binary images of granular material[20]. The algorithm detected characteristic sharp contact wedges in the outline of touching grain sections and created an intersection after checking if the angle of the contact wedge was smaller than a userdeﬁned threshold value. When making analysis of deformation of rock analogues, Fueten et.al applied gradient ﬁltering for different types of movies with frames[21]. Zhou et al. proposed a segmentation method of petrographic images by integrating edge detection and region grow method[22]. They employed a boundary detection method to get the edge information, with which the seeds for region grow were selected automatically. Gradient ﬁltering and threshold were also adopted step by step to pick several minerals in olivine–phyric basalts[23]. A similar method with multithreshold was presented to extract several kinds of objects and to produce the corresponding binary images[24]. A recent method was presented by Roy Choudhury et al. , which they called CASRG, to identify a ’region’, a set of points belonging to the grain[4]. The authors selected the seeds manually and chose the optimal threshold separately for each grain instead of using a single threshold for the whole image. The modiﬁcation was very efﬁcient and the accuracy has been validated for low and high strain samples in their contribution. The imperfections of this method was that all the seeds should be chosen by clicking the mouse. The working of choosing the seeds was also onerous for the large or high stress samples when the clats needed to be clicked are too much or the clasts were deformed badly. In fact, complex optical properties of plagioclase, such as twinning, present a particularly difﬁcult challenge to thin section image processing. Gradientbased boundary detection method are likely to classify optical twin zones as different grains. Barraud shown an example of textural analysis of thin sections of rocks with a Geographic Information System (GIS) program, in which boundary was obtained by watershed segmentation on digital pictures of the thin section[25]. Regiongrow method have also been improved to identify both twinned and untwinned plagioclase areas as seeds[26]. To overcome this problem, a set of plane polarised light images, taken at 51 intervals with 18 polarizer rotations were used to create an average grey level image with high resolution. homogeneous zones were detected and they were classiﬁed manually as seeds to form the basis of further grain boundary recognition. Obara presented a registration algorithm for reversed transformation of rock structure images taken with polarizing microscope stage rotations[27]. The idea behind this algorithm was based on ﬁnding the optimal rotation angle and optimal translation vector, thus ensuring the best correspondence between analysed images. The criteria for optimization was formulated on the basis of the shapes of edges located on images, in which edges were detected with gradient ﬁltering. To identify transcrystalline microcracks in microscope images of a dolomite structure, Obara devised a polarizing system using two nicols: one was ﬁxed and the other could be rotated, while the thin section was kept ﬁxed. 12 colour images were taken: 11 images with two crossed nicols and one with one nicol[28]. Based on the dolomite structure, CIELab color system was used and some image processing techniques, including gradient, threshold and mathematical morphology functions utilizing linear structuring elements, were performed for different components to detect transcrystalline microcracks. The maximum of standard deviation values of 11 α components were fused to be a single image for consequent operations. Obara also used a similar method to detection of the morphological anisotropy of calcite grains in marble[29]. Filtering with a rotated stencil consisting of two linear structuring elements preserved ﬁne structure in thin section image. Fueten and Mason proposed to edit edges with an artiﬁcial neural net assisted approach[30]. The goal of this method was to produce close boundaries and enclose areas were considered to be grains. This method signiﬁcantly improved the speed with which edges could be edited in preparation for other
Mineral Grain Boundary Detection With Image Processing Method: Edge Detection Operation Level SetSetTechnique Mineral Grain Boundary DetectionFrom With Image Processing Method: From Edge Detection To Operation To Level Technique
313 5
studies, although training of the neural network and manual correction of the results were still necessary.. Using GIS, Li, Onasch and Guo presented an automated method to detect grain boundaries and construct a boundary database in which the shape, orientation, and spatial distribution of grains could be quantiﬁed and analyzed in a reproducible manner[31]. In this procedure, they calculated the difference between the maximum and minimum value within a speciﬁed neighborhood, large values were recognized as grain boundaries. In [8], level set method(LSM) was introduced to detect the grain boundary. The major advantage of the LSM is that the grain boundary detected by the LSM is a closed curve, which is preferred for features extraction and data analysis. To quantify microstructures of coarsegrained marbles, Ebert et al. presented a new approach based on the intensity of light reﬂectance in dependence of the crystallographic orientation of calcite grains[32]. As ﬁlters could not distinguish between twin boundaries and different phases (especially in the case of thin micas), and the grain boundaries were compiled from a stack of images (one image for each sample rotation increment), grain boundaries was traced with Photoshop manually. To determine crystal size distributions of olivine in kimberlite, scanned images of polished rock slab were analysed and region of interesting(ROI) was determined by combining texture, colour and grey intensity analysis outputs. Adjoined crystals were separated by adapting and extending the markerbased watershed algorithm. In a study in [33], the application of 2D and 3D textural analysis to the quantiﬁcation of olivine populations in kimberlites was investigated. Olivine grains were segmented with a threshold ﬁlter selecting greyvalues from 50 to 73 connected to greyvalues from 50 to 66 (seeded threshold) after the data set was subjected to a median ﬁlter for noise reduction. Using ANN for image processing seemed to be more popular and is a highly researched area. Baykan proposed an ANN for the classiﬁcation of minerals using color spaces without boundary detection[36]. A microscopic information system(MIS) for petrographic analysis was presented with GIS and applied to transmitted light images[37]. Two region functions were developed and embedded in the GIS environment. GIS software provided optimal management of the MIS database, allowing the cumulative measurement of more than 87,000 grains. The methods mentioned above are mainly based on the traditional image segmentation and edge detection technology in image processing. The initial boundaries produced by edge detection methods have many imperfections. The boundary may be open, discontinuous, which do not coincide with the realistic boundary. Grain boundary, speciﬁcally a close one, provides fundamental information about material properties, such as area, orientation, percent, microstructural analyse [32] and crystal size distributions [33]. Such boundaries can not be used to feature extraction and other image processing tasks and a post processing arithmetic is needed to obtain more realistic boundaries. So postprocessing is necessary to get a closed boundary. Some techniques, as suggested in [30], have been reported to link edges. Concomitantly, the subjectivity and the modiﬁcation on raw data introduced by the postprocessing make them unsuitable for data analysis. Region grow method tries to identify the region of the grain by ’absorbing’ all the points similar with the ’seed’ point. It’s best advantage is that the boundary of the region is closed, which is preferred for measurements and analysis. A critical point in this method is to select the seeds, which often involves human interventions to avoid the seeds failing to grow according to the given rules. In CASGR[4], seed was selected by human interaction and the threshold was set adaptive to data automatically. Zhou et al. tried to overcome this problem by introducing a hybrid method that the results of edgedetection provided clues for automated seed select[22]. Apart from the human intervention in seeding, another defect is the occur of the holes within the region. The size of the hole depends on the noise distribution and the area of
314 6
Image Segmentation Intech
the heterogeneity of color information within the clasts. The small holes can be ﬁlled by employing simple morphological operations, but the larger ones remains almost unchanged. Such postprocessing ineluctably affects the extracted boundaries as well as the extracted features, such as major and minor axes, orientation, centroid, etc[4].
3. Level set method for grain boundary detection Methods based on the partial differential equation (PDE) have been widely used in image segmentation and other image science ﬁelds recently. Impressive effects have been obtained with various PDE models. In [6], Chan and Verse proposed a successful segmentation model (CV model) using level set, which derived from the classical active contour (snake) model [7]. The essential idea of snake model is to evolve an initial contour governed by a PDE until it stops on the edge of the objects. For initial snake model, the edge is deﬁned where have larger magnitude of gradient. As imposed with a second order derivative constraint, the curve have a good smoothing shape, which is contrast to the boundaries produced by edgedetection operator. Level set method is based on the description of the curve as the zero crossing of a higherdimensional function and allows major simpliﬁcations[38]. In level set, a closed curve is seen as the zero level set of a function in high dimension. It offers a nature representation for the contour of the object, which can deal with complicated structures with many advantages, especially when the curve undergoes complex topological change. It is a thriving method in image science for its following advantages: • easy to implement numerically; • the outline of the object is closed; • tackle with the topology change easily, such as merge and split. • some geometric quantities can be expressed directly. With this method, initial curve can be anywhere or with any shape in the image plane. In the following, we will give a framework of level set for grain boundary detection. First we introduce level set for gray scale image. Then, active contour for vector image is given. We end this section by consider a level set model with two polarising images as input. 3.1 Level set for gray scale image
For a garyscale image u, considering the following energy functional: E ( c1 , c2 , C ) =
inside(C )
u0 ( x, y) − c1 2 dxdy +
outside(C )
u0 ( x, y) − c2 2 dxdy + μ · Length(C), (1)
where c1 and c2 are constant unknowns representing the average value of u inside and outside the curve C. For curve evolution, the level set has been used widely. It can deal with cusps, corners and automatic topological changes. Now, we rewrite the original model (1) in the level set formulation. The curve C is deﬁned as the zero level set as follows: C = {( x, y) ∈ Ωφ( x, y) = 0}. Assuming that φ has opposite signs on each side of C, the energy can be rewritten as: E ( c1 , c2 , φ ) =
Ω
((u( x, y) − c1 )2 H (φ) + (u( x, y) − c2 )2 (1 − H (φ)))dxdy + μ
Ω
∇ H (φ)dxdy, (2)
Mineral Grain Boundary Detection With Image Processing Method: Edge Detection Operation Level SetSetTechnique Mineral Grain Boundary DetectionFrom With Image Processing Method: From Edge Detection To Operation To Level Technique
315 7
where φ is level set function, ν is a positive coefﬁcient, H denotes the Heaviside function: 1, z ≥ 0; H (z) = 0, z < 0. In order to compute the associated Euler–Lagrange equation for the unknown function, we consider slightly regularized versions of the function H, denoted here by H as → 0. One example of such approximations is given by ⎧ ⎪ ⎪ ⎨1, z > , H (z) = 0, z < , ⎪ ⎪ ⎩ 1 1 + z + 1 sin( πz ) , z ≤ , 2 π as proposed in [6]. For minimizing the functional deﬁned in (2), ﬁxing c1 and c2 , we obtain the following EulerLagrange equation: δ (φ)[νdiv(
∇φ ) − ( u0 − c1 )2 ) + ( u0 − c2 )2 ] = 0 ∇φ
(3)
where δ is onedimensional Dirac measure and δ = H (z). Using gradient descent method by an artiﬁcial time t, we yield the following evolution equation: ∂φ ∇φ ) − ( u0 − c1 )2 ) + ( u0 − c2 )2 ] = δ (φ)[νdiv( ∂t ∇φ
(4)
An alternative way to improving the above model is to replace δ (φ) by ∇u to extend the evolution to all level set of φ. Keeping φ ﬁxed and minimizing the energy yields the following expressions for c1 and c2 :
u( x, y) H (φ( x, y))dxdy c1 ( φ ) = Ω
, (5) Ω H ( φ ( x, y )) dxdy
u( x, y)(1 − H (φ( x, y)))dxdy . (6) c2 ( φ ) = Ω
Ω (1 − H ( φ ( x, y ))) dxdy To solve this evolution problem, we use a ﬁnite differences scheme, as suggested in [6]. As the thin section image contain abundant structure, the initial contour we obtain is so complex that it is almost impossible to make strain analysis directly, though all the valuable ones have been identiﬁed. The following we have to do is to select the useful grains suitable for analysis. It is a difﬁcult task as the noise disturbs the segmented results, and the small objects are too much, so we should set constrains to discard the useless ones. The select depends on the grain to be identiﬁed and the question tackled with. Here we provide a simple way, which could be improved by the Recalling when the image is processed manually or using other automated methods, the intensity, area and shape of the grain are dominant factors for human inspection and segmentation criterion. Level set provides the boundaries of the objects by utilizing the intensity distribution, so area and shape criterions are adopted for a tough tentative strains to abandon the unsuitable objects. The theory of level set method shows that the contour is a closed curve, so the objects whose area is ranging between two
316 8
Image Segmentation Intech
thresholds k1 and k2 are selected. As for the shape information, the ration of major axis and minor axis could be adopted as another important criterion for high formed minerals. It is a simple attempt to devise such constraints, more precious constraints, mainly decided by the previous knowledge of the the grains, will be proposed in future work. 3.2 Level set for color image
Color information is important factor for image segmentation and edge detection. Though luminance could be extracted from color image and processed as gray scale image, but it is still necessary to consider color components for grain boundary detection. In the following we will discuss the level set for color micrographic of thin section. Let ui,j be the jth channel of an color image u on Ω, with j = 1, 2, 3 channels and C the evolving curve. Each channel would contain the same image with some differences. Let c+ = (c1+ , c2+ , c3+ ) be a unknown constant vector. The extension of the CV model to vector images is E(c+ , c− , C ) =
1 3 + ∑ λ j u j (x, y) − c+j 2 dxdy + inside(C ) 3 j=1
1 3 − ∑ λ j u j (x, y) outside(C ) 3 j=1
2 − c− j  dxdy + μ · Length(C).
(7)
The curve C is deﬁned as the zero level set as follows: C = {( x, y) ∈ Ωφ( x, y) = 0}. In this model, the active contour C is the boundary between two regions deﬁned over all channels. Rewrite it in level set form, we obtain E(c+ , c− , φ) =
+
Ω
1 3 + 2 λ j u j ( x, y) − c+ j  H ( φ ( x, y )) dxdy 3 j∑ =1
Ω
1 3 − 2 λ j u j ( x, y) − c− j  (1 − H ( φ ( x, y ))) dxdy + μ 3 j∑ =1
Ω
∇ H (φ)dxdy, (8)
,− ) for j = 1, 2, 3. The parameters μ is the weight for the length term of the curve. λ+,− = (λ+ j are the weights for the noise error. H denotes the Heaviside function, again. − Minimizing the energy with respect to the constant c+ j , c j , for j = 1, 2, 3, we obtain:
u j H (φ( x, y))dxdy = Ω , average(uj )onŒ ≥ 0, Ω H ( φ ( x, y )) dxdy
u j (1 − H (φ( x, y)))dxdy − cj = Ω
, average(uj )onŒ < 0. Ω H ( φ ( x, y )) dxdy c+ j
Assuming that c+ j are constant vectors, and minimizing the energy functional with respect to φ, we obtain the following Eulerlagrange equation for φ: ⎤ ⎡
3 3 1 1 ∇ Œ 2 − + 2 ⎦ = 0, (u − c+ (9) δ (φ) ⎣− ∑ λ+ j ) + 3 ∑ λ j ( u j − c j ) + μ · div ∇Œ 3 j =1 j j j =1
Mineral Grain Boundary Detection With Image Processing Method: Edge Detection Operation Level SetSetTechnique Mineral Grain Boundary DetectionFrom With Image Processing Method: From Edge Detection To Operation To Level Technique
317 9
where δ is onedimensional Dirac measure and δ = H (z). To solve this PDE numerically, we introduce an artiﬁcial time as gradient descent method ⎤ ⎡
3 3 ∂φ 1 1 ∇ Œ 2 − + 2 ⎦ = 0, = δ (φ) ⎣− ∑ λ+ (u − c+ (10) j ) + 3 ∑ λ j ( u j − c j ) + μ · div ∇Œ ∂t 3 j =1 j j j =1 in Ω with the boundary condition
δ (φ)∂φ =0 ∇u∂n
on ∂Ω, where ∂n denotes the unit normal at the boundary of Ω. 3.3 Level set for polarised images
Polarised images captured under different light contain important clues for mineral boundary detection. For a thin section, two images can be captured from plane or crosspolarised light in microscopy. The main difference between polarizing images and ordinary images is polarised light phenomenon: under planepolarised light, many mineral grain appear as colorless; under crosspolarised light, mineral grains show varying colors and intensities. The colors and intensity rely the mineral, the thickness of the grain and its crystallographic orientation. In the previous publications, gradient ﬁltering has been proposed to deal with every image followed by a fuse procedure. We extend the level set method to segment multi images captured from different light. Figure 1 shows the two images of a thin section. Figure 1(a) is of a less contrast and the edges of Fig. 1(b) seems to be more legible. It is not proper to detect the grain boundary with any image. To segmentation the polarised images, we construct a new active contour model below. The energy as well as the evolving is similar with that of the equation above. Let ui,j be the jth channel of an image ui on Ω, with j = 1, 2, 3 channels, i = 1, 2 images and C the evolving curve. Each channel would contain the same image with some differences. Let + + + − − − , c1,2 , c1,3 ) and c− = (c2,1 , c2,2 , c2,3 ) be two unknown constant vectors. The extension c+ = (c1,1 of the CV model to polarised images is E(c+ , c− , C ) =
+
1 2 3 + ∑ ∑ λi,j ui,j (x, y) − ci,j+ 2 dxdy inside(C ) 6 i =1 j=1
1 2 3 − ∑ ∑ λi,j ui,j (x, y) − ci,j− 2 dxdy + μ · Length(C). outside(C ) 6 i =1 j=1
(11)
In this model, the active contour C is the boundary between two regions deﬁned over all channels. Rewrite it in level set form, we obtain E(c+ , c− , φ) =
+
2
Ω
3
Ω
1 2 3 + ∑ λi,j ui,j (x, y) − cij+ 2 H (φ(x, y))dxdy 6 i∑ =1 j =1
1 ∑ λi,j− ui,j (x, y) − cij− 2 (1 − H (φ(x, y)))dxdy + μ 6 i∑ =1 j =1
Ω
∇ H (φ)dxdy,
(12)
for i = 1, 2 and j = 1, 2, 3. The parameters μ is the weight for the length term of the curve. +,− ) are the weights for the noise error. λ+,− = (λi,j
318 10
Image Segmentation Intech
(a) Taken under planepolised light
(b) Taken under crosspolised light
Fig. 1. Two polarizid thin section images. + − Minimizing the energy with respect to the constant ci,j , ci,j , for i = 1, 2, j = 1, 2, 3, we obtain:
+ ci,j
Ω
=
Ω
− ci,j
=
ui,j H (φ( x, y))dxdy
Ω
H (φ( x, y))dxdy
,
average(ui,j )onŒ ≥ 0,
ui,j (1 − H (φ( x, y)))dxdy
, average(ui,j )onŒ < 0. Ω H ( φ ( x, y )) dxdy
+ are constant vectors, and minimizing the energy functional with respect Assuming that ci,j to φ, we obtain the following Eulerlagrange equation for φ (introducing an artiﬁcial time as gradient descent method): ⎤ ⎡
2 3 2 3 1 1 ∇ Œ ∂φ + + 2 − + 2 ⎦, = δ(φ) ⎣− ∑ ∑ λi,j (ui,j ( x, y)− cij ) + ∑ ∑ λi,j (ui,j ( x, y)− cij ) + μ · div ∂t 6 i =1 j =1 6 i =1 j =1 ∇Œ
(13) in Ω with the boundary condition
δ (φ)∂φ =0 ∇u∂n
on ∂Ω, where ∂n denotes the unit normal at the boundary of Ω. To solve this evolution problem, we use a ﬁnite differences scheme, as suggested in [6].
Mineral Grain Boundary Detection With Image Processing Method: Edge Detection Operation Level SetSetTechnique Mineral Grain Boundary DetectionFrom With Image Processing Method: From Edge Detection To Operation To Level Technique
319 11
The mail difference between the proposed model and the classical model in [6] is the income images. Our model is strongly inspired by the work of [6]. The three methods evolve the curve depending on the information with multichannel. The main difference is the number of the input image and channel. Our model deals with the two polarised images of a thin section of the same view. Under plane and crosspolarised light, different color information is captured. Some grains can be observed in one image while they may have a unconspicuous edges or regions. In the new model, the active contour evolves according to the structure of the two polarised images instead of any single image. More information is concerned to locate a more reasonable and accurate edge position. Similarly, the contour produce by the above model need a postprocessing to remove unmeaning objects.
4. Experimental results We present in this section some numerical results obtained with the models from the previous section. For the details of the numerical schemes and for other numerical results, we refer the reader to [6],[8],[39]. As we will see in this section, these models have the abilities of automatic detection of grain boundaries. Every grain boundary is represented with a closed curve. In Fig.2–Fig.4, we illustrator the whole procedures of low strain grain boundaries detection using level set with gray scale as input. We also compare the grains detected by level set with that of handdrawn method. Comparisons of features and strain analysis results can be found in [8]. In Fig.5, an example of high strain sample boundary detection is given and the result is compared with that of a region growing method: CASGR[4]. A note to this example is that our detection is performed on gray scale image and CASGR is performed with color image. In Fig.6, we show the result of color grain boundary detection with equation (10). We skip the preprocessing of TV ﬂow and the we refer to [40] for more details. In Fig.7 , we demonstrate the segmentation of polarising images of a thin section. The thin section contains some grains and other clasts. The red contours represent the edges of the identiﬁed grains. In Fig.7(a), segmentation of planepolarising image with classical vectorvalued image is shown. Some small clasts are identiﬁed as grains wrongly. Segmentation of crosspolarising image is shown in Fig.7(b). In the left of the image, a large part of ares is recognized as inner of a grain according to the structure of the crosspolarsing image, which is not coincide with the fact of planepolarising image. The edges of grains detected with the proposed model is shown in Fig.7(c). The time space is 0.1 and the total iter number is set 200. The other parameter are i,j
set as follows: μ = 0.2, λ+,− = 1, i, j = 1, 2, 3. Some clasts are excluded from the identiﬁed grains. The reason for a new segmenting result is that the information in six channel is used to determine the regions of the grains. With more clues concerned, the disturb of the noise and false edges is suppressed.
5. Conclusion We have presented in this paper a framework for grain boundary detection using level set with different input images: gray scale image, color image, planepolarising and crosspolarising images. The obtained variational level set models yield closed grain boundary which is preferred for feature extraction and data analysis. Application to grain boundary detection have been illustrated. In future, we will focus on the grain identiﬁcation and classiﬁcation with multiphase level set approach for thin section image of complex topologies.
320 12
Image Segmentation Intech
6. Acknowledgements The authors would like to thank editor for the invitation to contribute to this book. This work has been supported in part by grants NSFC 10926075 and The Doctor Grant of Henan Polytechnic University(B200941).
(a) Original color image.
(b) Gray scale image.
Fig. 2. Thin section image of mylonite.
(a) iter=0
(b) iter=10
(c) iter=15
(d) iter=22
Fig. 3. Curve involving of thin section image.
Mineral Grain Boundary Detection With Image Processing Method: Edge Detection Operation Level SetSetTechnique Mineral Grain Boundary DetectionFrom With Image Processing Method: From Edge Detection To Operation To Level Technique
(a) Grains boundaries by level set in color image.
(b) Contours of grains by level set in corresponding binary image..
(c) Handdrawn boundaries grains in color image.
(d) Handdrawn boundaries of grains in corresponding binary image.
of
Fig. 4. Grain boundaries: level set VS handdrawn.
(a) Color image .
(b) Gray scale image .
(c) Grain boundaries by CASGR.
(d) Grain boundaries by level set.
Fig. 5. Grain boundaries detection of low strain sample.
321 13
322 14
Image Segmentation Intech
(a) Color thin section image
(b) Red channel
(c) Blue channel
(d) Green channel
(e) Contour of segmenting result
(f) Boundary of identiﬁed andalusite
Fig. 6. Grain boundary detection of andalusite with color image.
Mineral Grain Boundary Detection With Image Processing Method: Edge Detection Operation Level SetSetTechnique Mineral Grain Boundary DetectionFrom With Image Processing Method: From Edge Detection To Operation To Level Technique
(a) Segmentation of planepolarising image
(b) Segmentation of crosspolarising image
(c) Segmentation of two palarising images
Fig. 7. Grain boundary detection with different input images
323 15
324 16
Image Segmentation Intech
7. References [1] Ramsay, J. G., 1967. Folding and Fracturing of Rocks. McGraw Hill, New York. [2] Ailleres, L., M. Champenois., 1994. Reﬁnements to the Fry method (1979) using image processing. Journal of Structural Geology 16 (9), 13271330. [3] Yu, H., Zheng, Y., 1984. A statistical analysis applied to the R f / f method. Tectonophysics 110, 151155. [4] Roy Choudhury, K., Patrick A. M., Kieran, F. Mulchrone., 2006. Automated grain boundary detection by CASRG. Journal of Structural Geology 28(3), 363375. [5] T. F. CHAN, B. SANDBERG, and L. VESE, Active contours without edges for vectorvalued images, J. Visual Comm. Image Rep. 11(2) (2000), 130141. [6] T. F., Chan, L. Vese, 2001. Active contours without edges. IEEE Transactions on Image Processing 10, 266277. [7] Kass, M., Witkin, A., Terzopoulos, D., 1988. Snakes: Active contour models. International Journal of Computer Vision 1, 321331. [8] Bibo Lu, Min Cui, Qiang liu, Yangang Wang, 2009, Automated grain boundary detection using the level set method, Computers and Geosciences, 35(2), 267275. [9] Starkey, J., Samantaray, A. K., 1993. Edge detection in petrographic images. Journal of Microscopy 172(3), 263266. [10] Lumbreras, F., Serrat, J., 1996. Segmentation of petrographical images of marbles. Computers and Geosciences 22, 547558. [11] Frank Fueten, 1997, A computercontrolled rotating polarizer stage for the petrographic microscope, Computers and Geosciences, 23(2), 203208. [12] Goodchild, J., Fueten, F., 1998. Edge detection in petrographic images using the rotation polarizer stage. Computers and Geosciences 24, 745751. [13] S.K. Naik and C.A. Murthy, 1999, Standardization of edge magnitude in color images, IEEE Transactions on Image Processing 5 , 1´lC8. [14] Mlynarczuk, M., 1999. Some remarks on the application of image analysis and image processing for the description of the geometrical structures of rock. Physicochemical problems of mineral processing 33, 107116. [15] Bartozzi, M., Boyle, A.P., Prior, D.J., 2000. Automated grain boundary detection and classiﬁcation in orientation contrast images. Journal of Structural Geology 22(11), 15691579. [16] Marco Herwegh, 2000, A new technique to automatically quantify microstructures of ﬁne grained carbonate mylonites: twostep etching combined with SEM imaging and image analysis, Journal of Structural Geology, 22(4) 391400. [17] Heilbronner, R., 2000. Automatic grain boundary detection and grain size analysis using polarization micrographs or orientation images. Journal of Structural Geology 22(15), 969981. [18] Sean Thompson, Frank Fueten, 2001, David Bockus, Mineral identiﬁcation using artiﬁcial neural networks and the rotating polarizer stage, Computers and Geosciences, 27(9), 10811089. [19] Ross, B. J.; Fueten, F.; Yashkir, D. Y. ,2001, Automatic Mineral Identiﬁcation Using Genetic Programming. Machine Vision and Applications 13, 6169. [20] E. H. van den Berg, A. G. C. A. Meesters, J. A. M. Kenter, W. Schlager, 2002, Automated separation of touching grains in digital images of thin sections, Computers and Geosciences, 28(2) 179190.
Mineral Grain Boundary Detection With Image Processing Method: Edge Detection Operation Level SetSetTechnique Mineral Grain Boundary DetectionFrom With Image Processing Method: From Edge Detection To Operation To Level Technique
325 17
[21] Frank Fueten, Kristen Hynes, Ryan L. Van Luttikhuisen, 2002, An experimental setup for the analysis of analogue deformation experiments using the rotating polariser stage, Journal of Structural Geology, 24(1), 241245. [22] Zhou,Y., Starkeyb, J, Mansinha, L, 2004. Segmentation of petrographic images by integrating edge detection and region growing. Computers and Geosciences 30, 817831. [23] C. S. Perring, S. J. Barnes, M. Verrall, R. E. T. Hill, 2004, Using automated digital image analysis to provide quantitative petrographic data on olivinephyric basalts, Computers and Geosciences, 30(2), 183195. [24] Nicoletta Marinoni, Alessandro Pavese, Marco Foi, Luca Trombino, 2005, Characterisation of mortar morphology in thin sections by digital image processing, Cement and Concrete Research, 35(8), 16131619. [25] Joseph Barraud, 2006, The use of watershed segmentation and GIS software for textural analysis of thin sections, Journal of Volcanology and Geothermal Research, Volume 154, Issues 12, Modern Trends in Petrography:  Textural and Microanalysis of Igneous Rocks, 1, 1733. [26] John V. Smith, Eberhard Beermann, 2007, Image analysis of plagioclase crystals in rock thin sections using grey level homogeneity recognition of discrete areas, Computers and Geosciences, 33(3), 335356. [27] Boguslaw Obara, 2007, An image processing algorithm for the reversed transformation of rotated microscope images, Computers and Geosciences, 33(7), 853859. [28] Obara, B., 2007, Identiﬁcation of transcrystalline microcracks observed in microscope images of dolomite structure using image analysis methods based on linear structuring element processing. Comput. Geosci. 33(2), 151´lC158. [29] Obara, B., Kožušníková, A., 2007. Utilisation of the image analysis method for the detection of the morphological anisotropy of calcite grains in marble. Computational Geoscience. 11,275´lC281. [30] F. Fueten and J. Mason, 2007, An artiﬁcial neural net assisted approach to editing edges in petrographic images collected with the rotating polarizer stage, Computers and Geosciences 33, 1176´lC1188. [31] Yingkui Li, Charles M. Onasch, Yonggui Guo, 2008, GISbased detection of grain boundaries, Journal of Structural Geology, 30(4), 431443. [32] A. Ebert, D. RiekeZapp, M. Herwegh, K. Ramseyer, E. Gnos, D. Decrouez, 2009, Microstructures of coarsegrained marbles, analyzed using a new technique based on the bireﬂectance of calcite, Tectonophysics, 463(1), 175184. [33] Dougal A. Jerram, Alex Mock, Graham R. Davis, Matthew Field, Richard J. Brown, 2009, 3D crystal size distributions: A case study on quantifying olivine populations in kimberlites, Lithos, Volume 112, Supplement 1, Proceedings of the 9th International Kimberlite Conference, 9th International Kimberlite Conference, November, 223235. [34] N.Singh, T.N.Singh, A. 2009, Tiwary and Kripamoy Sarkar,2009,Textural identiﬁcation of basaltic rock mass using image processing and neural network,Computational Geosciences. 14(2), 301310. [35] Holden et al., E.J. Holden, S. Moss, J.K. Russell and M.C. Denith, 2009, An image analysis method to determine crystal size distributions of olivine in kimberlite, Computational Geosciences 13(3), 255268. [36] Nurdan Akhan Baykan, Nihat Yilmaz, 2010, Mineral identiﬁcation using color spaces and artiﬁcial neural networks, Computers and Geosciences, 36(1), 9197. [37] Simone Tarquini, Massimiliano Favalli, 2010, A microscopic information system (MIS) for petrographic analysis, Computers and Geosciences, 36(5), 665674.
326 18
Image Segmentation Intech
[38] Osher, S., Sethian,J.A., 1988. Fronts Propagating with Curvature Dependent Speed: Algorithms Based on HamitonJacobi Formulations, J. Computational Physics 79, 1249. [39] Tony F. Chan, B. Yezrielev Sandberg, Luminita A. Vese, 2000, Active Contours without Edges for VectorValued Images, Journal of Visual Communication and Image Representation, 11(2), 130141. [40] Bibo Lu, Chao Ning, 2010, PDEbased Grain Boundary Detection, Procedding of Second IITA Conference on Geoscience and Remote Sensing (IITA GRS 2010), China, 170173.
17 Multiscale Segmentation Techniques for Textile Images Xiqun Lu
College of Computer Science, Zhejiang University Hangzhou, 310027, China 1. Introduction
In this chapter we focus on image segmentation techniques for some very special images — textile images. They are generated from the color halftoning technique in textile and printing production lines. In contrast with natural color images, textile images have some very distinctive features: (1) generally there are a few dominant colors in a textile image, whereas there may exist hundreds of significant colors in a natural color image; (2) in the textile industry designers combine threads of different colors, thickness, and densities to produce the visual impression of other colors (the color halftoning technique in textile printing). The fabric texture structure — texture noise — has a great influence on the colors’ appearance in textile images, which makes the color segmentation of textile images a very difficult problem [1]. A typical textile image is shown in Fig.1. According to human visual observation, there are about six dominant colors in the textile image, but when the image is enlarged to the pixellevel, we will find that there are many different colors in perceived uniform color regions as shown in Fig.1. (a) and (b). Four histograms of the spatial variation magnitudes of the four uniform and edge blocks (as shown in Fig.1 (a), (b), (c) and (d)) are illustrated in Fig.2 (Here the spatial variation magnitude is computed as the added value of the two absolute forward differences along the horizontal and vertical directions on the luminance component of each
Fig. 1. A typical textile image
(a) Uniform red area
(b) Uniform blue area
(c) Edge area I
(d) Edge area II
328
Image Segmentation
(a) Uniform red area
(b) Uniform blue area
(c) Edge area I
(d) Edge area II
Fig. 2. Histograms of the spatial variation magnitudes of uniform areas and edge areas in the textile image block). From these histograms, we can see that it is very difficult to distinguish the variations caused by genuine spatial edges from those caused by texture noise. Some distinctive properties of texture noise are compared with those of white Gaussian noise in [2]. Fig.3. (a) shows a 16×16 texture noise block extracted from the textile image as shown in Fig.1. Fig.3. (d) is a 16×16 white noise block generated by the “randn” function in MatlabTM. Texture noise is clearly nonGaussian distributed, and has highly correlative spatial structure. Any image segmentation algorithm that independently classifies each pixel in the image plane is unlikely to perform well for textile images since there is insufficient information to make a good decision. So many segmentation algorithms divide the image into arbitrary blocks, and classify each block independently. However, if the block size is too small, discriminating among similar textures may be difficult. Alternatively, if the block size is too large, regions of differing textures may be lost. In either case, the resulting boundaries will not be accurate since there is no reason to believe that the actual boundaries occurred along the block boundaries [3]. Multiscale image segmentation approaches [1, 314] have been proven efficient to integrate both image features and contextual information to classify a region in an image differently from its surroundings if there is sufficient statistical evidence to justify a distinct region regardless of size, and refine the segmentation results recursively between different scales. The number of important contributions in this area is so great that just listing all of them would more than exhaust the page budget of this chapter (for example, the bibliography in [4] is 8 pages in a twocolumn format). So in Section 2 we review a few representative multiscale image segmentation techniques which developed over the past decades. In Section 3 we test some of the benchmark multiscale image segmentation techniques on textile images, and compare the experimental results analytically. We finally give some concluding remarks in Section 4. In the following sections, we will follow the convection of using uppercase letters for random variables and lowercase letters for their realizations.
329
Multiscale Segmentation Techniques for Textile Images
(a) Texture noise (b) PDF of texture noise
(c) Autocorrelation function of texture noise
(e) PDF of white noise
(f) Autocorrelation function of white noise
(d) White noise
Fig. 3. Distinctive properties of texture noise
2. Multiscale image segmentation techniques 2.1 Problem statement and notations The image segmentation problem can be considered as the process of inferring the “best” configuration of labelling X from the observed image data Y = y , and both random fields X and Y are defined on a rectangular gird S . Each random variable of X takes its values from a finite set of class labels Λ = {1 " K} , where K is the total number of classes. The standard Bayesian formulation of this inference problem consists in minimizing the expectation of some cost function C , given the data xˆ = arg min E(C ( X , x )Y = y )
(1)
x
Where C ( X , x ) is the cost of estimating the true segmentation X by an approximate segmentation x . The MAP (Maximum a Posteriori) estimator is an solution of (1) if we use the cost functional of C ( X , x ) = 1 whenever any pixel is incorrectly classified. The MAP estimator aims at maximizing the probability that all pixels will be correctly classified. It is well known that the MAP estimator is excessively conservative, and appear even more inappropriate for the estimation of multiscale Markov random fields [6]. An alternative criterion to the MAP estimation is to minimize the probability of classification error, the Maximization of the Posterior Marginals (MPM) estimator, which associates to each cite the most probable value given all the data
330
Image Segmentation
∀s ∈ S ,
xˆ s = arg min P( xs Y = y )
(2)
xs ∈Λ
The MPM estimator results in high computational complexity since each of these functions should be obtained by simultaneously integrating out all xt (t ≠ s ) [5]. Usually in real situations, it is unfeasible to compute these posterior marginals P( xs  y ) exactly since the original joint distribution P( x , y ) is unavailable or it is intractable to specify the joint distribution for each possibility. 2.2 Sequential MAP [6] Although the MPM criterion seems to be more appropriate than the MAP criterion, both the MPM and MAP cost functions do not take into account the location of estimation errors in a hierarchical quadtree structure. Bouman and Shapiro introduced the following cost function in [6] L
L
n=0
i =n
CSMAP ( X , x ) = ∑ 2 n − 1 (1 − ∏ δ ( X ( i ) , x( i ) ))
(3)
At each scale n , the segmentation or labeling is denoted by the random field X ( n ) , and the set of lattice points is denoted by S( n ) . This sequential MAP (SMAP) cost function is to sum up the segmentation errors from multiple scales together. The SMAP estimator aims at minimizing the spatial size of errors. The multiscale image model proposed in [6] is composed of a series of random fields at multiple scales. Each scale has a random field of image feature vectors Y ( n ) , and a random field of class labels X ( n ) . We denote an individual sample at scale n by ys( n ) and xs( n ) , where s is the position in the 2D lattice S( n ) . Markovian dependencies are assumed across scales to capture interscale dependencies of multiscale class labels, and the SMAP recursion can be estimated in a fashion of coarsetofine
{
}
xˆ ( n ) = arg max log py( n )x( n ) ( y ( n ) x( n ) ) + log Px( n )x( n+1) ( x( n ) xˆ ( n + 1) ) x( n )
(4)
The two terms in (4) are the likelihood function of the image feature y ( n ) and the contextbased prior knowledge from the next coarser scale, respectively. Specifically, the quadtree pyramid developed in [6] is to capture interscale dependencies of multiscale class labels regarding the latter part of (4). But a big problem with the quadtree pyramid is that spatially adjacent samples may not have a common parent sample at the next coarser scale which may result in discontinuous boundaries. Therefore a more generalized pyramid graph model was introduced in [6] where each sample has multiple parents at the next coarser scale. However, this pyramid graph also complicates the computation of likelihood functions, and the coarsetofine recursion of (4) has to be solved approximately. Based on the same framework, a trainable context model for multiscale Bayesian segmentation was proposed in [7], where the contextual behavior can be trained offline by providing image data the corresponding ground truth segmentations. Then the segmentation can be accomplished efficiently via a single finetocoarsetofine iteration through the pyramid.
331
Multiscale Segmentation Techniques for Textile Images [8]
2.3 Waveletdomain HMT A distinct contextbased Bayesian segmentation algorithm was proposed in [8] where the context model is characterized by a context vector v( n ) derived from a set of neighboring samples (3×3) at the next coarse scale. It is assumed that, given ys( n ) , its context vector
{
vs( n ) = x(psn )
}
xls( n ) can provide supplement information regarding xs( n ) , where x(psn ) denotes
the class label of its parent sample, and xls( n ) denote the dominant class label of the 3×3 samples at the next coarser scale. Given vs( n ) , xs( n ) is independent with all other class labels. In particular, the contextual prior px( n )v( n ) (cu) is involved in the SMAP estimation which has the same purpose as the second term in (4), and it can be estimated by maximizing the following contextbased mixture model likelihood as
f ( y ( n )  v( n ) = u ) =
K
∏ ∑ px
s∈S
(n)
c =1
(n)
v( n )
( xs( n ) = cvˆ s( n ) = u) f ( ys( n ) c )
(5)
where the likelihood function f ( y ( n ) x( n ) ) is computed by using the waveletdomain
hidden Markov model (HMT) proposed in [9]. An iterative Expectation Maximization (EM) training algorithm was developed in [8] to approach the above problem. The HMT is a treestructure model in the waveletdomain to characterize the joint statistics of wavelet coefficients across scales. In order to perform multiscale segmentation, an image is recursively divided into four subimages of same size J times, and a pyramid is constructed with J scales. A block y ( n ) at scale n is associated with three wavelet subtrees in three subbands as,
{T
(n) LH
}
(n) (n) THL THH . The computation of the model likelihood of
f ( y ( n ) θ ) is a realization of the HMT model θ and is obtained by (n) (n) (n) f ( y ( n ) θ ) = f (TLH θ LH ) f (THL θ HL ) f (THH θ HH )
(6)
Where it is assumed that the three DWT subbands are independent and each one in (6) can be estimated based on a close formula proposed in [9]. An improved waveletdomain hidden Markov model, HMT3S [10], was developed to capture the wavelet coefficients dependencies not just across scales, but across subbands as well, where the three DWT subbands are grouped into one quadtree structure. It is worth noting that twostate GMMs (Gaussian Mixture Model) in HMT [8] are still used to characterize the DWT marginal statistics, and there will be eight states in a node of HMT3S. Thus, HMT3S is parameterized by
{
}
θ HMT − 3S = p J (u), ε uj ,,jv− 1 ,σ b2, j , h b ∈ B, j = 1," J ; u , v = 0," ,7; h = 0,1 Where B = {LH
(7)
HL HH} , and ε uj ,,jv− 1 is the transition probability of the Markov chain
from scale j to scale j − 1 . The EM training algorithm in [8] can be straightforwardly extended to the eightstate HMT3S.
332
Image Segmentation [12]
2.4 Joint multicontext and multiscale HMT3S Most multiscale segmentation algorithms adopt SMAP estimator and mainly consider interscale dependencies of multiscale class labels by assuming Markovian dependencies across scales, and intrascale dependencies are not considered due to the fact that the noncausal structure usually requires extra iterative training process. However, the characterization of intrascale dependencies is useful for boundary localization [11]. Based on three numerical segmentation criteria: 1) Pa the percentage of pixels which are correctly classified; 2) Pb the percentage of boundaries that are coincided with the true ones; 3) Pc the percentage of true boundaries that can be detected, Fan and Xia quantified the segmentation performances of five contextual models in [11]. And show that interscale context models (context1 and context2) may favor Pa by encouraging the homogeneity of texture classification across the scales of the pyramid, and the intrascale context model (context5) may help Pc by being sensitive to texture boundaries within a scale. As hybrid context models (context3 and context4) may provide high Pb by appropriately balancing both interscale dependencies and intrascale dependencies of multiscale class labels into the MAP estimation. It is shown that none of the five context models can work well singly [11]. Since a single context model is unable to provide high accuracy for both texture classification and boundary localization, Fan and Xia proposed a joint multicontext and multiscale (JMCMS) approach to Bayesian segmentation in [12] which reformulates (1) as a multiobject optimization as
xˆ = arg min E(CSMAP ( X , x )Y = y1 ) x
# xˆ = arg min E(CSMAP ( X , x )Y = y z )
(8)
x
Where an image y can be represented as multiple (Z) copies characterized by distinct context models, i.e., {y z z = 1," , Z} . Different context models provide different multiscale modeling, the multiobjective problem in (8) is roughly analogous to the multiple criteria: Pa , Pb and Pc . The above problem can be solved by a heuristic algorithm called the multistage problemsolving technique [11]. Both JMCMS [11] and HMT3S [10] can improve segmentation results in terms of Pa , Pb and Pc by emphasizing the two terms in (4), respectively. It is shown in [12] that combination of them provides the best segmentation result regarding Pa and Pb . 2.5 Multiresolution Gaussian Autoregressive Models (MGAR) based on MMPM [13] A doublestochastic model is proposed in [13] for multiresolution texturedimage segmentation where the observed image is represented as a multiresolution Gaussian autoregressive (MGAR) model and class labels are assumed to be dependent on both the same scale and the adjacent finer and coarser scales as a 3D MRF. The optimization criterion used for segmentation is the minimization of the expected value of the number of misclassified nodes in a multiresolution lattice. The estimator that satisfies this criterion is referred to as the “multiresolution maximization of the posterior marginals” (MMPM) estimator, and is a natural extension of the singleresolution MPM estimate [13]. The cost function in (1) is given by
333
Multiscale Segmentation Techniques for Textile Images J −1
C(X , x ) = ∑
∑ δ (Xs( j ) , xs( j ) )
(9)
j = 0 s∈S( j )
Where J denotes the total number of levels of the multiresolution lattice S , and the multiresolution representation of the observed image Y and the class label pyramid X are defined on the same lattice S . The segmentation x that minimizes the conditional expectation of this cost function will be denoted as x * . Thus E[C ( X , x * )Y = y ] = min E[ ∑ (1 − δ ( Xs , xs ))Y = y ] x
s∈S
= min ∑ E[(1 − δ ( Xs , xs ))Y = y ] x
s∈S
= min ∑ P( Xs ≠ xs Y = y ) x
(10)
s∈S
= min ∑ (1 − P( Xs = xs )Y = y )) x
s∈S
= max ∑ P( Xs = xs Y = y ) x
s∈S
where δ ( Xs , xs ) = 1 , if X s = xs ; δ ( Xs , xs ) = 0 , otherwise. It is assumed that the number of distinct textures in the observed image is known in this approach, but the parameters θ of the MGAR model — the means, prediction coefficients, and prediction error variances of different textures are unknown. A modified version of the EM algorithm is used to estimate the parameters [13]. The Gibbs sampler with constant temperature can be used to generate a Markov chain X (t ) which converges in distribution to a random field with probability mass function pXY ( XY ,θ ) . The marginal conditional probabilities pXsY ( kY ,θ ) , which are to be maximized, are then approximated as the fraction of time the Markov chain spends in state k at node s , for each k and s [13]. If N 0 is the number of iterations (complete passes through the pyramid) of the Gibbs sampler, then the approximations
pXsY ( kY ,θ ) =
1 N0
N0
∑ δ (Xs (t ), k )
∀k , s
(11)
t =1
provide estimates of the value needed to obtain the MMPM estimate of X . Integrating the multiresolution EM with the approximated MMPM algorithm, image segmentation and parameters estimation can be obtained simultaneously [13]. 2.6 Multigrid Belief Propagation [14] Markov random field models provide a robust and unified framework for image segmentation, but the MRF framework yields an optimization problem that is NP hard, and it is usually highly advantageous to use graphs as diagrammatic representations to facilitate analysis and manipulations. One of the feasible approximation solutions is belief propagation [1416]. Belief Propagation (BP) algorithm utilizes the conditional independent properties in the network to derive efficient solutions. Corresponding to the MPM and the MAP estimators, there are two types of BP algorithms. One is belief update (BU) also known
334
Image Segmentation
as the sumproduct algorithm for the MPM inferences, and another is belief revision (BR) a.k.a. the maxproduct algorithm for the MAP inferences [15]. The quality of image segmentation can be described in an energy function as following E( x ) = ∑ Dxs ( xs ) + s∈S
∑
( s , p )∈N
(12)
W ( xs , x p )
Where N are the (undirected) edges in the fourconnected image grid graph. Dxs ( xs ) is the cost of assigning label xs ( xs ∈ {1," , K }) to pixel s in the lattice S , and is referred to as the data cost. W ( xs , x p ) measures the cost of assigning different labels to two neighboring pixels s and p , and is generally referred to as the discontinuity cost. An intuitive measurement of the discontinuity cost is based on the degree of difference between the two neighboring labels, that is, W ( xs , x p ) = V ( xs − x p ) . The BP algorithm works by passing messages around the graph defined by the fourconnected image grid. The method is iterative, with messages from all nodes (pixels) being passed in parallel. At each iteration, in the maxproduct BP algorithm, new messages can be computed with negative log probabilities where the maxproduct becomes a minsum: mst → p ( x p ) = min(V ( xs − x p ) + Dxs ( xs ) + xs
∑
q∈N ( s )\p
mq(t→− 1) s ( xs ))
(13)
In the sumproduct BP algorithm, new messages are updated as mst → p ( x p ) = min(V ( xs − x p ) ⋅ Dxs ( xs ) ⋅ xs
∏
q∈N ( s )\p
mq( t→− 1) s ( xs ))
(14)
The standard implementation of maxproduct message passing algorithm on the grid graph runs in O(nK 2T ) time, where n is the number of pixels in the image, K is the number of possible labels for each pixel and T is the number of iterations. So the computation complexity is very high for images with large sizes. Felzenszwalb and Huttenlocher [14] proposed an efficient multigrid BP technique without changing the graph structure and the energy function: first the data costs are constructed from fine to coarse scales, where the data cost at coarse level can be calculated by summing over four data costs at the next finer level; then the message propagation process is started at the coarsest level, and the belief information is propagated from coarse to fine scales. In contrast, the underlying graph in the example proposed in [16] is changed. First, a multiscale algebraic multigrid technique is used to select nodes that strongly influence others as the coarse nodes at the next coarser level, and an iterated weighted aggregation (IWA) process is followed to calculate the new edge weights among the selected coarse nodes. Second, given the belief propagation result at coarse level is interpolated as a start point for the next finer scale. Since the start point is believed to be close to the true solution, this BP is expected to converge rapidly. 2.7 Multiscale probabilistic reasoning model [1] The quadtree structure induces causal properties that enable design of a noniterative coarsetofine multiscale segmentation algorithm. However, an important disadvantage of the quadtree structure is that blocks that are spatial adjacent may not have common
335
Multiscale Segmentation Techniques for Textile Images
neighbors at the next coarser scale, and this may result in blocky artifacts in the final segmentation [1]. Fan and Xia listed five contextual models in [11], and proved that the context4 model can provide a high percentage of boundaries that coincide with the true ones. A hierarchical probabilistic reasoning model is proposed in [1] to alleviate the problem of blocky artifacts by providing a quadtree structure which combined with a spatial lattice neighborhood at each scale. This hierarchical probabilistic reasoning model [1] is very different from the multiscale models proposed in [613]. The concept of scale in [1] is related to the size of a block in the spatial domain. While the scale described in [613] is related to either the Gaussian pyramid or the wavelet decomposition of the observed image. The Gaussian lowpass filter can alleviate texture noise to some degree, but it can also blur edges at the same time, especially for those lowcontrast edges. The wavelet transform is very useful to represent the singularities (edges and ridges) in an image at multiple scale and three different orientations [8]. However, for textile images, the singularities are corrupted by uniformly distributed texture structure (see Fig.2). So it is more appropriate to perform segmentation in the spatial domain directly than in the waveletdomain, and the experimental results in Section 3 also prove this scheme is inappropriate for textile images. As we know that a large block usually enhances the classification reliability (because the use of many pixels can lessen the disturbance caused by texture noise), but simultaneously risks having pixels of different classes inside the block; while a small block reduces the possibility of having multiple classes in the block, but sacrifices classification reliability due to the paucity of color information and texture noise. So both the large and small scale behavior should be utilized to properly segment both large homogeneous regions and detailed boundary regions. In [1], no special prior distribution assumption is made about the size and shape of regions. At every increasing scale, each block is subdivided into four child blocks, forming a quadtree structure. By adopting the context4 model proposed in [11] the correlations of spatial blocks across different scales and within the same spatial scale are integrated as shown in Fig.4
{
}
xˆ ( n ) = arg max pb( n )c (b( n ) c = c k ) ⋅ pc kvb (c = c k vb ) k
k
(15)
Where pb( n )c (b( n ) c = c k ) is the likelihood of the block b( n ) at scale n given the dominant k
color c k ( k = 1," , K ) . Since textile images are corrupted by nonGaussian texture noise, it is not appropriate to specify this conditional probability pb( n )c (b( n ) c = c k ) as a multivariate k
Gaussian distribution as done in other multiscale models [613]. In [1], we calculated the ratio of how many pixels in the current block b( n ) are within a sphere centered at the dominant color c k ( k = 1," , K ) with a radius of rk in the color space as the likelihood: pb( n )c (b( n ) c = ck ) = k
#(c pij − c k < rk , pij ∈ b( n ) ) size(b( n ) )
(16)
336
Image Segmentation
Where # is the counting operator, and rk is half of the minimum distance between the dominant color c k ( k = 1," , K ) and the other dominant color cl (l ≠ k , l = 1," , K ) . Blocks in the coarser scale The parent block in the coarser scale Neighbor blocks in the coarser scale Blocks in the finer scale
Ω
The current block in the finer scale
v mn  v mp
Neighbor blocks in the finer scale Neighbor blocks in the finer scale with the same parent
Ω
Decision of the dominant color
v mn  v mp
The context model
Fig. 4. Hierarchical probabilistic reasoning model [1] The contextual information vb in (15) contains the color information of its parent block at the next coarser scale and that of the eight neighbor blocks of the current block at the same scale. We assume that the dependence on its parent block’s color information is totally uncorrelated with the dependencies on its eight neighbor blocks’ color information at the same scale. Therefore we can write the contextual probability pc kvb (c = c k vb ) as pc kvb (c = c k vb ) = p(c = c k vbparent ) ⋅ p(c = c k vbneighbor )
(17)
where “parent” denotes the parent block at the next coarser scale, and “neighbor” denotes the eight neighboring blocks at the same scale of the current block. According to Bayes rule, the probability p(c = c k vbparent ) or p(c = c k vbneighbor ) can be computed as follows: p(c = c k vb~ ) =
=
p( v = vb~ c = c k )p(c = c k ) p( v = vb~ )
p( v = vb~ c = c k )p(c = c k )
K
∑ p(c = c j )p( v = j =1
vb~ c
(18)
= cj)
where ~ indicates either “parent” or “neighbor”. The prior distribution of the dominant color p(c = c k ) ( k = 1," , K ) can be computed according to (16), where b( n ) will be the size of
Multiscale Segmentation Techniques for Textile Images
337
the entire image. The likelihood p( v = vbparent c = c k ) of the parent block and the likelihood p( v = vbneighbor c = c k ) of the eight neighbor blocks, given the dominant colors c k ( k = 1," , K ) , can be computed from coarse to fine scales according to (15) respectively.
3. Performances of multiscale image segmentation techniques on textile images In this section we focus on textile image segmentation based on several representative multiscale image segmentation techniques, and test their performance analytically. Texture noise — the texture appearance of the fabric in textile images makes bottomup, solely imagedriven segmentation techniques always prone to errors [1]. Most of textile images are taken directly from the production line in textile industries, there are no ground truth segmentation results for these textile images, and it is difficult for us to manually prepare training data for trainingbased supervised image segmentation [7, 8]. In this paper we mainly consider supervised multiscale image segmentation, that is, it is assumed the number of distinct colors in the input textile image is known, and the dominant colors and their corresponding features can be obtained earlier. The texture statistical characteristics are very helpful for image segmentation when the given image contains different visual texture regions [3, 8, 1013]. However textile images contain uniformly distributed texture structure (as shown in Fig.1), so the color information is the main feature can be handled with during segmentation with during segmentation. Since there are usually only a few dominant colors in textile images, we obtain dominant colors by picking and averaging homogeneous regions from the given image. We extract six dominant colors: blue, red, white, yellow, dark green and shallow green from textile image 1, and eleven dominant colors from textile image 2, and five dominant colors from textile image 3 and textile image 4, respectively. In order to capture the crossscale relationships of texture regions, prior to segmentation, learningbased supervised image segmentation techniques [7, 8] have to pick large enough homogeneous regions as training data from textile images. However, some of the dominant colors only occupied very small areas in textile images, such as black color in textile image 2 and textile image 3, so we only apply the HMTSeg algorithm [8] to textile image 1 and 4 because in these two images each dominant color occupies large enough area. In this paper we only examine supervised multiscale image segmentation techniques on textile images: the multiscale probabilistic reasoning model [1], sequential MAP [6], HMTSeg [8], belief propagation [14]. In the multiscale probabilistic reasoning model [1], no special prior distribution is assumed about the size and shape of regions, but in the SMAP [6] a multivariate Gaussian distribution is applied for that purpose, so in addition to inputting the number of dominant colors and corresponding RGB values, the SMAP algorithm requires inputting the covariance matrices of the three RGB channels of the dominant colors. The SMAP algorithm is good for textile images with fine texture structure and large homogeneous regions, such as textile image 4. Although the computation complexity of the SMAP algorithm is higher than that of the maximum likelihood algorithm, the performance of SMAP is comparable to that of ML for textile images with coarse fabric texture structure and delicate structures. In order to apply the HMTSeg algorithm [8] to textile images, we have to pick several homogeneous blocks from the inputted image as training data. Both textile image 1 and
338
Image Segmentation
textile image 4 have large homogeneous regions, and for each dominant color, we randomly pick ten 16×16 blocks. With this training data and interscale tying in the waveletdomain HMT models [8], the EM training algorithm [9] is used to estimate the parameters of the HMT model for each dominant color. Since each HMT model is trained on 16×16 uniform blocks, the output segmentation results are very blocky (see Fig.5. (f) and Fig.8. (f)). Though the waveletdomain HMT model is very helpful to images with edges and ridges, in textile images, edges and ridges are corrupted by uniformly distributed texture noise and the test image generally is much larger than the training images, so repeat the likelihood computations for image subblocks assuming that the blocks are independent. The waveletdomain HMT model which designed for distinct visual texture statistics is not suitable for textile images, and from the segmentation results, we can see that the misclassified color areas are large. We don’t apply the HMTSeg algorithm to textile image 2 and 3 because some of the dominant colors only occupied very small areas in these two images, and it is difficult for us to pick large enough training blocks from those images. The belief propagation approach [14] can be used to approximate the MAP solutions to MRF problems. The local minima found by BP are minima over “large neighborhoods”, so they can produce high accurate segmentation results in practice. In [14] they proposed three algorithmic schemes to improve the running time of the loopy belief propagation approach: the minconvolution algorithm, belief propagation on grid graphs and a multigrid method which speed up and reduce the memory requirements of belief propagation. In order to allow for large discontinuities in the labelling, the cost function stops growing after the difference becomes large, and they introduced a data term d to V ( x ) in eq. (13) to control the cost to stop increasing when the discontinuities are large. Another parameter λ is applied to balance between the fidelity of the data cost term D( x ) and the smoothness of the discontinuity cost V ( x ) . Both d and λ are selected heuristically. In all the experiments, d is set as 10 4 , and λ is set as 15. The iteration number for each scale is set as 10, and the experimental results do not change too much as the iteration number increases above 10. The BP algorithm is good for large homogeneous regions but will simplify fine structures, and good examples are illustrated in Fig.6 and Fig.8.
4. Conclusions In contrast with natural images, textile images have some very distinctive properties: 1) generally there are a few dominant colors in a textile image, whereas there may exist hundreds of significant colors in a natural image; 2) the fabric texture structure caused by the color halftoning technique in textile printing has a great influence on the colors’ appearance in textile images. The uniform fabric structure (texture noise) makes it a difficult problem for the existing automatic color segmentation methods to extract the dominant colors from textile images. In this chapter, we study various multiscale image segmentation techniques by considering image segmentation as inferring the “best” labeling configuration X from the observed image data Y . The causal properties induced by multiscale structure enable the design of exact, noniterative inference algorithms. In most multiscale image segmentation techniques, the joint probability density of ( X , Y ) and the marginal density are modeled as multivariate Gaussian distributions or Gaussian mixtures. Whereas textile images are corrupted by non
Multiscale Segmentation Techniques for Textile Images
339
Gaussian texture noise, it is difficult to specify a prior distribution about the size and shape of regions, and this is an open problem of our future research work. We analyze the performances of different multiscale image segmentation techniques on some representative textile images, and find that the spatial contextual model proposed in [1] can produce high visual quality segmentation results on average. Since these textile images have no ground truth segmentation results, it is difficult for us to evaluate the performances of different techniques quantitatively. So to evaluate the segmentation results quantitatively is also a problem of our future research works.
Fig. 5. Segmentation results of textile image 1
340
Image Segmentation
(a) Textile image 2
(b) Probabilistic Reasoning [1]
(c) SMAP [6]
(d) BP [14]
(c) SMAP [6]
(d) BP [14]
Fig. 6. Segmentation results of textile image 2
(a) Textile image 3
(b) Probabilistic Reasoning [1]
Fig. 7. Segmentation results of textile image 3
341
Multiscale Segmentation Techniques for Textile Images
(a) Textile image 1
(b) Probabilistic Reasoning [1]
(c) SMAP [6]
(d) BP [14]
(e) ML
(f) HMT [8]
Fig. 8. Segmentation results of textile image 4
5. References [1] Lu, X.Q. (2007). Color textile image segmentation based on multiscale probabilistic reasoning, Optical Engineering, 46(8), Article ID 087002 [2] Lu, X.Q.; Sakaino, H. (2009). A spatial adaptive filter for smoothing of nonGaussian texture noise, Proc. of IEEE ICASSP, pp.841844, Taibei, Apr. 2009, IEEE [3] Bouman, C.; Liu, B. (1991). Multiple resolution segmentation of textured images, IEEE Trans. On Pattern Analysis and Machine Intelligence, vol.13, no.2, pp.99113, 1991 [4] Wilsky, A. S. (2002). Multiresolution Markov models for signal and image processing, Proc. of the IEEE, vol.90, no.8, pp.13961458, 2002 [5] Laferte, J.M.; Heitz, F. (2000). Discrete Markov image modeling and inference on the quadtree, IEEE Trans. On Image Processing, vol.9, no.3, pp.390404, 2000 [6] Bouman, C. A.; Shapiro, M. (1994). A multiscale random field model for Bayesian image segmentation, IEEE Trans. On Image Processing, vol.3, no.2, pp.162177, 1994 [7] Cheng, H.; Bouman, C. A. (2001). Multiscale Bayesian segmentation using a trainable context model, IEEE Trans. On Image Processing, vol.10, no.4, pp.511525, 2001
342
Image Segmentation
[8] Choi, H.; Baraniu, R. G. (2001). Multiscale image segmentation using waveletdomain hidden Markov models, IEEE Trans. On Image Processing, vol.10, no.9, pp.13221331, 2001 [9] Crouse, M. S.; Nowak, R. D., and Baraniu, R. G. (1998). Waveletbased statistical signal processing using hidden Markov models, IEEE Trans. On Signal Processing, vol.46, no.4, pp.886902, 1998 [10] Fan, G.; Xia, X.G. (2000). Maximum likelihood texture analysis and classification using waveletdomain hidden Markov models, Proc. of 34th Asilomar Conference on Signals, Systems and Computers, Pacific Grove, CA, 2000 [11] Fan, G.; Xia, X.G. (2001). A joint multicontext and multiscale approach to Bayesian Image segmentation, IEEE Trans. On Geoscience and Remote Sensing, vol.39, no.12, pp. 26802688, 2001 [12] Fan, G.; Xia, X.G. (2003). Waveletbased texture analysis and synthesis using hidden Markov models, IEEE Trans. On Circuits and Systems — I: Fundamental Theory and Applications, vol.50, no.1, pp. 106120, 2003 [13] Cormer, M. L.; Delp, E. J. (1999). Segmentation of textured images using a multiresolution Gaussian autoregressive model, IEEE Trans. On Image Processing, vol.8, no.3, pp.408420, 1999 [14] Felzenszwalb, P. F.; Huttenlocher, D. P. (2006). Efficient belief propagation for early vision, International Journal of Computer Vision, vol.70, no.1, pp.4154, 2006 [15] Weiss, Y.; Freeman, W. T. (2001). On the optimality of solutions of the maxproduct beliefpropagation algorithm in arbitrary graphs, IEEE Trans. On Information Theory, vol.47, no.2, pp.736744, 2001 [16] Xiong, L; Wang, F. & Zhang, C. (2007). Multilevel belief propagation for fast inference on Markov random fields, Proc. of 7th IEEE International Conference on Data Mining, pp.371380, 2007
18 JSEG Algorithm and Statistical ANN Image Segmentation Techniques for Natural Scenes Luciano Cássio Lulio, Mário Luiz Tronco and Arthur José Vieira Porto Engineering School of Sao Carlos, University of Sao Paulo (EESC/USP) Sao Carlos, Sao Paulo Brazil
1. Introduction Computer vision, when used in open and unstructured environments as in the inspection of crops for natural scenes, demands and requires complex analysis of image processing and segmentation algorithms, since these computational methods evaluate and predict environment physical characteristics, such as color elements, complex objects composition, shadows, brightness and inhomogeneous region colors for texture. Several segmentation algorithms proposed in literature were designed to process images originally characterized by the abovementioned items. Additionally, agricultural automation may take advantage of computer vision resources, which can be applied to a number of different tasks, such as crops inspection, classification of fruits and plants, estimated production, automated collection and guidance of autonomous machines. Bearing the aforenamed in mind, the present chapter aims the use of JSEG unsupervised segmentation algorithm (Deng et al., 1999a), Statistical Pattern Recognition and Artificial Neural Networks (ANN) Multilayer Perceptron (MLP) topology (Haykin, 2008) as merging processing techniques in order to segment and therefore classify images into predetermined classes (e.g. navigable area, planting area, fruits, plants and general crops). The intended approach to segment classification deploys a customized MLP topology to classify and characterize the segments, which deals with a supervised learning by error correction – propagation of pattern inputs with changes in synaptic weights in a cyclic processing, with accurate recognition as well as easy parameter adjustment, as an enhancement of iRPROP algorithm (improved resilient backpropagation) (Igel and Hüsken, 2003) derived from Backpropagation algorithm, which has a faster identification mapping process, that verifies what region maps have similar matches through the explored environment. To carry through this task, a feature vector is necessary for color channels histograms (layers of primary color in a digital image with a counting graph that measures how many pixels are at each level between black and white). After training process, the mean squared error (MSE), denotes the best results achieved by segment classification to create the imageclass map, which represents the segments into distinct feature vectors. Several metrics (vector bundle) can be part of a feature vector, however, a subset of those which describes and evaluates appropriate classes of segments should be chosen.
344
Image Segmentation
2. JSEG image segmentation Color images with homogeneous regions are segmented with an algorithm to generate clusters in the color space/class (different measures classes in spectral distribution, with distinct intensity of visible electromagnetic radiation at many discrete wavelengths) (Cavani, 2007). One way to segment images with textures is to consider the spatial arrangement of pixels using a regiongrowing technique whereby a homogeneity mode is defined with pixels grouped in the segmented region. Furthermore, in order to segment texture images one must consider different scales of images. An unsupervised colortexture regions segmentation algorithm is ideal for this purpose, since it tests the homogeneity of a given colortexture pattern, which is computationally more feasible than model parameter estimation. It deals with the following assumptions for the acquired image: Image containing homogeneous colortexture regions. Color information is represented by quantized colors. Colors between two neighboring regions are distinguishable. The JSEG algorithm segments images of natural scenes properly, without manual parameter adjustment for each image and simplifies texture and color. Segmentation with this algorithm passes through two major stages, namely color space quantization (number reduction process of distinct colors in a given image), and hit rate regions with similar color regions merging, as secondary stage. In the first stage, the color space is quantized with little perceptual degradation by using the quantization algorithm (Deng et al, 1999b; Deng and Manjunath, 2001) with minimum coloring. Each color is associated with a class. The original image pixels are replaced by classes to form the class maps (texture composition) for the next stage. Before performing the hit rate regions, the Jimage  a class map for each windowed color region, whose positive and negative values represent the edges and textures of the processing image  must be created with pixel values used as a similarity algorithm for the hit rate region. These values are called „Jvalues“ and are calculated from a window placed on the quantized image, where the Jvalue belongs. Therefore, the twostage division is justified through the difficult analysis of the colors similarity whital their distributions. The decoupling of these features (color similarity and spatial distribution) allows tractable algorithms development for each of the two processing stages. Map of classes Image Spatial Segmentation Color space quantization
Color regions merging
Segments Jimage
Fig. 1. JSEG image segmentation steps.
Region growing
JSEG Algorithm and Statistical ANN Image Segmentation Techniques for Natural Scenes
345
2.1 Segmentation algorithm evaluation Natural scenes present a 24bit chromatic resolution color image, which is coarsely quantized preserving its major quality. The main idea for a good segmentation criterion is to extract representative colors differentiating neighboring regions in the acquired image, as an unsupervised method. Therewith, the color quantization using peer group filtering (Deng et al., 199a) is applied through perceptual weighting on individual pixels, to smooth the image and remove the existing noise. Then, new values indicating the smoothness of the local areas are obtained, and a weight is assigned to each pixel, prioritizing textured areas to smooth areas. These areas are identified with a quantization vector to the pixel colors, based on General Lloyd Algorithm (GLA) (Gersho and Gray, 1999), which the perceptually uniform L*u*v color space is adopted, presenting the overall distortion D: D = ∑ Di =∑∑ v(n) x(n) − ci i
i
n
2
→ x( n ) ∈ C i
(1)
And it is derived for: ci =
∑ v(n)x(n) → x(n) ∈ C i ∑ v(n)
(2)
The parameters: ci is the centroid of cluster Ci, x(n) and v(n) are the color vector and the perceptual weight for pixel n. Di is the total distortion for Ci. With the centroid value, as denoted by Equation (2)  after the vector quantization and merged clusters, pixels with the same color have two or more clusters, affected by GLA global distortion. For merging close clusters with minimum distance between preset thresholds for two centroids, an agglomerative clustering algorithm is performed on ci (Duda and Hart, 1970), as the quantization parameter needed for spatial distribution. After clustering merging for color quantization, a label is assigned for each quantized color, representing a color class for image pixels quantized to the same color. The image pixel colors are replaced by their corresponding color class labels, creating a classmap. segmented classmap 1 +++++++++ +++++++++ +++++++++                         ######### ######### #########
not segmented classmap 2 +#+#+#+#+  +  +  +  + +#+#+#+#+  +  +  +  + +#+#+#+#+  +  +  +  + +#+#+#+#+  +  +  +  + +#+#+#+#+
Fig. 2. Two different classmap representing three distinct classes of data points. In Figure 2, classmap 1 indicates three regions containing a single class of data points for segmentation process, and classmap 2 is not segmented indicating a color uniformity.
346
Image Segmentation
The symbols (+, , #) denotes the label values (Jvalue) for three distinct data points. All necessary segmentation information, after color quantization, is extracted and relocated to a classmap. A specific region contains pixels from a color class set, which is distributed in image regions. These regions, forming each one, a classmap, has distributed points in all spatial data segments, corresponding a twodimensional plane, and represents the cartesian position vector (x,y). In order to calculate the Jvalue, Z is defined as the set of all points of quantized image, then z = (x, y) with z ∈ Z and being m the average in all Z elements. C is the number of classes obtained in the quantization. Then Z is classified into C classes, Zi are the elements of Z belonging to class i, where i=1,...,C, and mi are the element averages in Zi. m=
1 ∑z N z∈Z
(3)
mi =
1 Ni
(4)
∑z
z∈Z
The Jvalue is as follows: J=
SB (ST − SW ) = SW SW
where: ST =
∑
z−m
(5) 2
(6)
z∈Z
C
SW = ∑ ∑ z − mi
2
(7)
i = 1 z∈Z
The parameter ST represents the sum of quantized image points within the average in all Z elements. Thereby, the relation between SB and SW, denotes the measures of distances of this class relation, for arbitrary nonlinear class distributions. J for higher values indicates an increasing distance between the classes and points for each other, considering images with homogeneous color regions. The distance and consequently, the J value, decrease for images with uniformly color classes. Each segmented region could be recalculated, instead of the entire classmap, with new parameters adjustment for J average. JK represents J calculated over region k, Mk is the number of points in region k, N is the total number of points in the classmap, with all regions in classmap summation. J=
1 ∑ Mk J k N k
(8)
For a fixed number of regions, a criterion for J is intended for lower values. 2.2 Spatial segmentation technique The global minimization of J is not practical, if not applied to a local area of the classmap. Therefore, the idea of Jimage is the generation of a grayscale image whose pixel values are
347
JSEG Algorithm and Statistical ANN Image Segmentation Techniques for Natural Scenes
the J values calculated over local windows centered on these pixels. With a higher value for Jimage, the pixel should be near region boundaries. Expected local windows dimensions determines the size of image regions, for intensity and color edges in smaller sizes, and the opposite occurs detecting texture boundaries. (b)
(a) + + ++ ++ ++ + +
+ + + + + + +
+ + + + + + + + +
+ + + + + + + + +
+ + + + + + + + +
+ + + + + + +
+ + + + + + + + + +
+ + +

+ + + + + + +

+ + + + + + + 

+ + + + + + + + +

+ + + + + + + + +

+ + + + + + + + +

+ + + + + + + 

+ + + + + + +

+ + +
Fig. 3. (a) A 9x9 circular window to avoid rectangular objects bias for local J values; (b) A downsampling scale 2, for a 17x17 window. Using a regiongrowing method to segment the image, this one is considered initially as one single region. The algorithm for spatial segmentation starts segment all the regions in the image at an initial large scale until the minimum specified scale is reached. This final scale is settled manually for the appropriate image size. The initial scale 1 corresponds to 64x64 image size, scale 2 to 128x128 image size, scale 3 to 256x256 image size, with due proportion for increasing scales and the double image size. Below, in Figure 4, the spatial segmentation algorithm is structured in flow steps. 2.2.1 Valley determination A heuristics for the valley determination, presupposes a condition for small initial regions to be determined as the pattern growing. These regions have the lowest J values (valleys). As follows: a. Calculate the standard deviation and the average of the local J values in the region, denoted by σJ and μJ, respectively. b. Threashold for parameters above: TJ = μ J + aσ J
(9)
The condition to consider candidate valley points for pixels with local J values is determined TJ > J. Connect the points based on the 4connectivity and obtain the valleys. a. For candidate valleys smaller than the spatial segmentation relation between scale and image size, they are denoted as valleys.
348 b.
Image Segmentation
A preset parameter values [0.6, 0.4, 0.2, 0, 0.2, 0.4] is given for variable a, which gives the most number of valleys. Original classmap initial scale
each region
calculate local J values
valley determination region growing valley growing
segmented regions final scale (1)
scale < threshold?
no
yes region merging
final results
Fig. 4. Sequence for spatial segmentation algorithm. 2.2.2 Valley growing and region merge After valley determination, the new regions grow from the valleys, which obey the following rules: c. Nondetermined pixels must be removed in the valleys, producing the local J values average in the remaining unsegmented regions. Then, pixels are connected below the average to form growing areas, and if these are adjacent to one and only one valley, they are assigned to that valley.
JSEG Algorithm and Statistical ANN Image Segmentation Techniques for Natural Scenes
349
d.
Calculate local J values for the remaining pixels at the corresponding scale to locate the boundaries regions. e. Grow the remaining pixels at the final scale. Unclassified pixels at valley boundaries are stored in a buffer. The buffer is updated when all pixels are classified. An initial segmentation of the image is obtained, after region growing, providing oversegmented regions, merged on their color similarity. The quantized colors represent color histogram bins and their distance features are calculated through the regions extraction. This distance means the Euclidean distance between two neighboring regions. The pair of regions with minimum distance is merged together. All distances are stored in a database and it is updated when an estimate for color feature vector and the corresponding region is calculated. The process continues until a maximum threshold for the distance is reached. After region merging, the final segmentation results are obtained.
3. Programming (Color quantization and spatial distribution) The sequential images in Figure 5 evince not only the color quantization (spatial distributions forming a map of classes), but also the space segmentation (Jimage representing edges and regions of textured side). Several window sizes are used by Jvalues: the largest detects the region boundaries by referring to texture parameters; the lowest detects changes in color and/or intensity of light. Each window size is associated with a scale image analysis. The concept of Jimage, together with different scales, allows the segmentation of regions by referring to texture parameters. Regions with the lowest values of Jimage are called valleys. The lowest values are applied with a heuristic algorithm. Thus, it is possible to determine the starting point of efficient growth, which depends on the addition of similar valleys. The algorithm ends when there are spare pixels to be added to those regions. Figures 6 to 10 illustrate not only color quantization and spatial distributions of Jimage in others natural scenes, but the flood fill implemented algorithm, for determining the boundaries edges connected on the region growing areas (using queue data structure provided from region valleys). All scenes were submitted to a gradient magnitude, as segmentation function rating (Sobel masks for higher values at the borders of navigation areas and lower values inside planting areas), then image is segmented with a watershed transform directly on the gradient magnitude. JSEG outperforms the evaluation for all images, with an effective spatial distribution on planting lines.
4. Artificial Neural Networks (ANN) and statistical pattern recognition Due to the nature of nonlinear vectors, it is fundamental that an ANNbased classification method associated with a statistical pattern recognition be used. MultiLayer Perceptron (MLP) is suitable for default ANN topology to be implemented through a customized backpropagation algorithm for complex patterns. The most appropriate segment and topology classifications are those using vectors extracted from HSV color space (Hue, Saturation, Value), matching RGB color space (Red, Green, Blue) components. Also, the network with less MSE in the neurons to color space proportion is used to classify the entities. Statistical methods are employed as a combination of results with ANN, showing how accuracy in nonlinear features vectors can be best applied in a MLP algorithm with a
350
Image Segmentation
(a)
(b)
(c)
Fig. 5. (a) Original images; (b) Color quantization (map of classes); (c) Jimage representing edges and regions of textured side (Spatial distributions).
JSEG Algorithm and Statistical ANN Image Segmentation Techniques for Natural Scenes
351
Fig. 6. JSEG segmentation and watershed transform of gradient magnitude in flood fill classmap for scene 1.
352
Image Segmentation
Fig. 7. JSEG segmentation and watershed transform of gradient magnitude in flood fill classmap for scene 2.
JSEG Algorithm and Statistical ANN Image Segmentation Techniques for Natural Scenes
353
Fig. 8. JSEG segmentation and watershed transform of gradient magnitude in flood fill classmap for scene 3.
354
Image Segmentation
Fig. 9. JSEG segmentation and watershed transform of gradient magnitude in flood fill classmap for scene 4.
JSEG Algorithm and Statistical ANN Image Segmentation Techniques for Natural Scenes
Fig. 10. JSEG segmentation and watershed transform of gradient magnitude in flood fill classmap for scene 5.
355
356
Image Segmentation
statistical improvement, which processing speed is essentially important, for pattern classification. Bayes Theorem and Naive Bayes both use a technique for iterations inspection, namely MCA (Main Component Analysis), which uses a linear transformation that minimizes covariance while it maximizes variance. Features found through this transformation are totally uncorrelated, so the redundancy between them is avoided. Thus, the components (features) represent the key information contained in data, reducing the number of dimensions. Therefore, RGB space color is used to compare the total number of dimensions in feature vectors with HSV. With a smaller dimension of iterations, HSV is chosen as the default space color in most applications. Bayes Theorem introduces a modified mathematical equation for the Probability Density Function (PDF), which estimates the training set in a conditional statistics. Equation (4) denotes the solution for p(Ciy) relating the PDF to conditional class i (classes in natural scene), and y is a ndimensional feature vector. Naive Bayes implies independence for vector features, what means that each class assumes the conditional parameter for the PDF, following Equation (5). P(C i  y ) =
∑
p( yC i )P(C i ) K p( yC j )P(C j ) j =1 n
P( yC i ) = ∏ p( y j C i )
(10)
(11)
j =1
Thus, the following items detail these merging techniques for image processing and pattern recognition as far as generated and customized segmenting algorithms are concerned withal. As a result, a modular strategy with JSEG algorithm, ANN and Bayes statistical theorem approach is essential for based applications on agricultural scenes. 4.1 Multilayer perceptron customized algorithm Derived from backpropagation, the iRPROP algorithm (improved resilient backpropagation) (Cavani, 2007) is both fast and accurate, with easy parameter adjustment. It features an Octave (Eaton, 2006) module which was adopted for the purposes of this work
Fig. 11. ANN schematic topology for planting lines with three classes.
JSEG Algorithm and Statistical ANN Image Segmentation Techniques for Natural Scenes
357
and it is classified with HSV (H – hue, S – saturation, V – value) color space channels histograms of 256 categories (32, 64,128 and 256 neurons in a hidden layer training for each color space channel: H, HS, and HSV). The output layer has three neurons, each of them having a predetermined class. All ANNbased topologies are trained with a threshold lower than 0.0001 mean squared errors (MSE), the synaptic neurons weights are initiated with random values and the other algorithm parameters were set with Fast Artificial Neural Network (FANN) library (Nissen, 2006) for Matlab (Mathworks Inc.) platform, and also its Neural Network toolbox. The most appropriate segment and topology classifications are those using vectors extracted from HSV color space. Also, a network with less MSE in the H64 was used so as to classify the planting area; for class navigable area (soil), HSV256 was chosen; as for the class sky, the HS32.
5. Normalization and feature extraction This section tackles how statistical methods were employed as a combination of results with ANN, showing how accuracy in nonlinear features vectors can be best applied in a MLP algorithm with a statistical improvement, which processing speed is essentially important, for patter classification. The MSE results for each topology, shown in Table 1, were partitioned to eliminate the feature vectors that are distant from the class centroids, so the classifier will deal in less dispersed vectors. Upon observing the following table, which shows the vector distribution in five training sets (20%, 30%, 50%, 70% and 100%), this work approached two probabilistic classification methods in order to match final pattern recognition results with ANN: Bayes theorem and Naive Bayes. MSE H
HS
HSV
Neurons 32 64 128 256 32 64 128 256 32 64 128 256
Navigation area 0,079117 0,110642 0,075546 0,086501 0,089143 0,099398 0,049100 0,057136 0,089450 0,059981 0,049677 0,038817
Planting area 0,098437 0,098566 0,079303 0,023520 0,094905 0,045956 0,095064 0,099843 0,022453 0,010384 0,078453 0,079856
Sky 0,098574 0,190555 0,079874 0,079111 0,023409 0,089776 0,097455 0,034532 0,067545 0,082364 0,043493 0,045643
Table 1. MSE results for each topology. RGB space color is used to compare the total number of dimensions in feature vectors with HSV. With a smaller dimension of iterations, HSV was chosen as the default space color. For such iterations inspection, a technique (main component analysis – MCA) uses a linear transformation that minimizes covariance while it maximizes variance. Features found through this transformation are totally uncorrelated, so the redundancy between them is avoided. Thus, the components (features) represent the key information contained in data, reducing the number of dimensions (Costa and Cesar Jr, 2001; Haykin, 1999; Comaniciu and Meer, 1999).
358
Image Segmentation
In HSV space color, the Bayesian classifiers have produced results which are similar to RGB, where there is a hit rate when the number of dimensions increases in an accuracy average ranging from 20% to 50%. A maximum rate accuracy for HSV is 0.38817, which occurs for 30% and 6777 dimensions. In RGB space color, Bayesian conventional classifiers are identical to Naive results, because as the dispersion of classes increases, there is an average hit rate, which goes up to 50%. The classifiers concerning the number of dimensions are different from the previous ones, which range from 20% and 30%, where hit rates fall as the number of dimensions increases.
% 20 30 50 70 100
NA 1029 1345 1390 1409 1503
RGB PA 5486 5768 6094 6298 6300
Sky 34 54 130 149 158
NA 1024 1342 1390 1402 1402
HSV PA 5384 5390 6003 6209 6209
Sky 26 45 103 140 145
Table 2. Vector distribution for RGB and HSV space colors. (Navigation area = NA; Planting are = PA) As a consequence, Bayesian classifiers in HSV space color, outperforms the other classifiers as shown in “Fig. 12”. The average rate of achievement value, together with the number of dimensions draw a linear convergence for all vector distribution in the five training sets.
Fig. 12. Average hit rates for the three major training sets. Although the three methods deliver different performances, yet similar behavior, because the hit rate of the class sky tends to improve owing to the increase in the number of dimensions. Navigation and planting area classes are listed as false feature vectors by the texture similarities in training, which means that ANN and Bayesian must be coupled for improved results.
JSEG Algorithm and Statistical ANN Image Segmentation Techniques for Natural Scenes
359
The following graph about the first components shows that the RGB curves have a higher percentage than most HSV curves. Also, it can be observed that all curves present values lower than 90%.
Fig. 13. RGB and HSV relation with amount of dimensions. After the main component analysis (MCA), the following graph shows the HSV training sets for 100% feature vectors distribution.
Fig. 14. HSV training set for 100% feature vectors distribution.
360
Fig. 15. Classmap 1 – Bayes/HSV and Bayes/RGB.
Fig. 16. Classmap 1 – Naïve Bayes/HSV and Naïve Bayes/RGB.
Fig. 17. Classmap 1 – ANN/HSV and ANN/RGB.
Image Segmentation
JSEG Algorithm and Statistical ANN Image Segmentation Techniques for Natural Scenes
Fig. 18. Classmap 2 – Bayes/HSV and Bayes/RGB.
Fig. 19. Classmap 2 – Naïve Bayes/HSV and Naïve Bayes/RGB.
Fig. 20. Classmap 2 – ANN/HSV and ANN/RGB.
361
362
Fig. 21. Classmap 3 – Bayes/HSV and Bayes/RGB.
Fig. 22. Classmap 3 – Naïve Bayes/HSV and Naïve Bayes/RGB.
Fig. 23. Classmap 3 – ANN/HSV and ANN/RGB.
Image Segmentation
JSEG Algorithm and Statistical ANN Image Segmentation Techniques for Natural Scenes
363
The corresponding class maps, the corresponding classmaps for the three first natural scenes in Figure 4, were shown above, for normalization merging techniques and space colors (Bayes/HSV), (Bayes/RGB), (Naive Bayes/HSV), (Naive Bayes/RGB), (ANN/HSV) and (ANN/RGB).
6. Conclusions This chapter presented merging techniques for segmentation and ANNstatistical classification of navigation agricultural scenes, running multiple segmentation tests with JSEG algorithm possible. As the data provided evince, this generated algorithms fulfils the expectations as far as segmenting is concerned, so that it sorts the appropriate classes (navigation area, planting area and sky). As a result, a modular strategy with ANN and Bayes statistical theorem can be an option for the classification of segments. Moreover, the classification using different types of feature vectors caused the classification metric to be more accurate and sophisticated with ANN, as well as the HSV color space to have lower MSE in test values. Both JSEG and MLP proved suitable for the construction of an image recognition system.
7. References Cavani, F. A. Analysis of natural scenes through image segmentation and pattern recognition, (Master Thesis) University of Sao Paulo (EESC/USP), 2007. Comaniciu, D. and Meer, P. Robust analysis of feature spaces: color image segmentation. In: Conference on Computer Vision and Pattern Recognition, IEEE Computer Society, 1997. Costa, L. F. and Cesar Junior, R. M. Shape analysis and classification – Theory and Practice. 1. ed. Boca Raton, Florida, EUA: CRC Press LLC. ISBN 0849334934, 2001. Deng, Y., Kennedy, C., Moore, M.S., Manjunath, B.S. Peer group filtering and perceptual color image quantization. Proceedings of the 1999 IEEE International Symposium on Circuits and Systems, 1999a. v. 4, p. 2125. Deng, Y., Manjunath, B.S., Shin, H. Color image segmentation. Conference on Computer Vision and Pattern Recognition, IEEE Computer Society, 1999b. p. 446451, v. 2. Deng, Y. and Manjunath, B.S. Unsupervised segmentation of colortexture regions in images and videos. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI '01), vol. 23, no. 8, pp. 800810, Aug. 2001. Duda, R.O. and Hart, P.E. Pattern Classification and Scene Analysis, John Wiley & Sons, New York, 1970. Eaton, J.W. et al. Octave. Avaliable at: http://www.octave.org. (02/12/2006). Gersho, A. and Gray, R.M. Vector quantization and signal compression, Kluwer Academic, Norwell, MA, 1992. Haykin, S. Neural networks: a comprehensive foundation. 2. ed. New Jersey, EUA: PrenticeHall. ISBN 0132733501, 1999. Haykin, S. Neural Networks and Learning Machines. 3.ed. McMaster University, Canada: PrenticeHall. ISBN 0131471392, 2008.
364
Image Segmentation
Igel, C., Hüsken, M. Empirical evaluation of the improved Rprop learning algorithm. Neurocomputing, v. 50, p. 105123, 2003. Nissen, S. et al. Fann: fast artificial neural network library. Avaliable at: http://leenissen.dk/fann/ (02/12/2006).
19 Image Segmentation of ZiehlNeelsen Sputum Slide Images for Tubercle Bacilli Detection R. A. A. Raof1, M. Y. Mashor2, R. B. Ahmad1 and S. S. M. Noor3
1School
of Computer and Communication Engineering, Universiti Malaysia Perlis 2School of Mechatronics Engineering, Universiti Malaysia Perlis 3School of Medical Science, Universiti Sains Malaysia Malaysia
1. Introduction Tuberculosis (TB) remains one of the leading causes of death in developing countries and its recent resurgences in both developed and developing countries warrants global attention. Globally, there were an estimated of 9.27 million incident cases of TB in 2007. This is an increase from 9.24 million cases in 2006, 8.3 million cases in 2000 and 6.6 million cases in 1990. Most of the estimated numbers of cases in 2007 were in Asia (55%) and Africa (31%), with small proportions of cases in the Eastern Mediterranean Region (6%), the European Region (5%) and the Region of the Americas (3%). The five countries that rank first to fifth in terms of total numbers of cases in 2007 are India (2.0 million), China (1.3 million), Indonesia (0.53 million), Nigeria (0.46 million) and South Africa (0.46 million). Of the 9.27 million incident TB cases in 2007, an estimated 1.37 million (15%) were HIVpositive; 79% of these HIVpositive cases were in the African Region and 11% were in the SouthEast Asia Region (WHO, 2009). ZiehlNeelsen stain method is one of the common techniques that are being used to diagnose the TB infection. Smear microscopy with ZiehlNeelsen technique has been the main means of diagnosing TB patients in developing countries. This is because the method is simple, rapid, reproducible, low cost and effective in detecting infectious disease such as TB (Luna, 2004). TB diagnosis is usually being done manually by microbiologist through microscopic examination of sputum specimen of TB patients for pulmonary TB diseases. However, there are some problems that have been reported with manual screening process, such as time consuming and laborintensive, especially for screening of the negative slides. (Veropoulus et al., 1998). When reporting the results of the microscopic examination, the microbiologist should provide the clinician with an estimation of the number of acidfast bacilli detected. If the smear microscopy is clearly positive, very little observation time is needed to confirm the result. The slide is classified as TB positive if at least one tubercle bacilli is found in 300 microscopic fields. The case will then be classified into one of four severity category if the smear on the slide is found to be positive, according to the number of tubercle bacilli found in the slide. For a welltrained microbiologist, it takes 15 to 20 minutes to read and confirm one negative slide, with an average of 25 slides can be read per day. In addition, for some developing countries, there is also a lack of welltrained microbiologist,
366
Image Segmentation
which may results in overload, fatigue and reduces the diagnostic performance (Khutlang et al., 2009). Therefore, an automated TB diagnosis is required so that large number of cases can be handled with the same accuracy and speeding up the process while improving the low sensitivity of manual TB diagnosis. In this research, images of Ziehlneelsen sputum slide are captured using a digital camera attached to a light microscope and displayed on a computer screen. In order to extract the TB bacilli pixels from the TB slide images, several image processing techniques are required. One of the image processing techniques is image segmentation, which is useful to classify the pixels in the image into two region, TB and background. The sputum specimen that has undergone the process of staining using ZiehlNeelsen procedure will make the TB bacilli appear red and other cells and organisms in the sputum smear sample will retained blue background. Image segmentation is a part of image processing technique that will help to discriminate between the TB bacilli and background pixels in the digital image. There are many attempts already been made to enable the image captured by a camera that is attached to the microscope to be viewed through the computer screen. Some image processing algorithms also have been developed in order to carry out automatic TB bacilli detection in the captured image. Forero applied adaptive color thresholding technique to the images that have been captured using fluorescence microscopy (Forero et al., 2003; Forero et al., 2004). Veropoulos used an identification method based on shape descriptors and neural network classifiers (Veropoulus et al., 1998; Veropoulus et al., 1999). Wilkinson proposed a rapid multi resolution segmentation technique based on computing thresholds for different areas in a monochromatic image (Wilkinson, 1996). The studies mentioned above used images captured from fluorescence microscope, which appear different from ZiehlNeelsen sputum slide images captured under light microscope. In this study, a method of grey thresholding technique is reviewed, and it is then being adapted to suit with color images. Thus the color thresholding algorithm is expected to be able to discriminate between the pixels that comprises the mycobacterium and sputum in the ZiehlNeelsen slide images. The outcome of this study should be able to provide a way of getting the suitable threshold values for the images and using the values to achieve the main objective of color thresholding and image segmentation.
2. Image segmentation Segmentation process subdivides an image into its constituent regions or objects. The level of subdivision depends on the problem being solved, where the segmentation should stop when the objects of interest in an application have been isolated. Image segmentation algorithms generally are based on one of the two basic properties of intensity values: discontinuity and similarity. Thresholding is a method of similarity category. It partitions an image into regions that are similar according to a set of predefined criteria. There are various thresholding techniques and it is also a fundamental approach to segmentation that enjoys a significant degree of popularity, especially in applications where speed is an important factor (Gonzalez & Woods, 2002). Traditionally, one simple way to accomplish thresholding is by defining a range of brightness value in the original image, then selects the pixels within the range as belonging to the foreground and rejects all of other pixels to the background. Such an image is then usually displayed as a binary or twolevel image (Sezgin & Sankur, 2004).
Image Segmentation of ZiehlNeelsen Sputum Slide Images for Tubercle Bacilli Detection
367
The general rule for grey level pixel thresholding is as in Equation (1). ⎧0, f ( x , y ) < T , g( x , y ) = ⎨ ⎩ 1, f ( x , y ) ≥ T ,
(1)
where T is the threshold value, f ( x , y ) is the original pixel value, and g( x , y ) is the resulted pixel value after thresholding has been done. Equation (1) specifies 0 and 1 as output values, which will give the result as a true binary image. Equation (1) can be further visualized by Figure 1 as mappings of input grey level to output grey level (Efford, 2000).
g 1
0
T
0
f
Fig. 1. Thresholding for a single threshold There could be more than one thresholding value at a time, which change Equation (1), to ⎧0, f ( x , y ) < T , 1 ⎪⎪ g( x , y ) = ⎨ 1, T1 ≤ f ( x , y ) ≤ T2 , ⎪ ⎪⎩0, f ( x , y ) > T2 .
(2)
where T 1 is the lower threshold value and T 2 is the upper threshold value. Figure 2 shows the visualization of how thresholding with a pair of threshold is being done (Efford, 2000).
g 1
0
0
T1
Fig. 2. Thresholding with a pair of threshold
T2
f
368
Image Segmentation
For color images, more than one variable characterises each pixel in the image, which allows multi spectral thresholding (Otsu, 1979). In color imaging, each pixel is characterised by three red, green and blue (RGB) values. However, with multi spectral or multilayer images such as RGB model, it can be difficult to specify the selection criteria. The logical extension of thresholding is simply to place brightness thresholds on each image, for instance to specify the range of red, blue and green intensities. These multiple criteria are then usually combined with an AND operation (i.e. the pixel is defined as part of the foreground if its three RGB components all lie within the selected range). This logically equivalent to segmenting each image plane individually, creating separate binary images and then combining them with a Boolean AND operator afterward. This color thresholding method is widely used in the image segmentation (Forero et al., 2003; Forero et al., 2004; MancasThillou & Gosselin, 2005).
3. Methodology In this study, a conventional thresholding method has been adopted to suit with color images of ZiehlNeelsen sputum slide specimen for TB detection. The method is used to segment the image into two regions, which are TB and background (consists of sputum and other bacilli). Figure 3 demonstrates the steps involved in the proposed image segmentation process.
Original RGB image
Pixel study
Image enhancement
Color thresholding
Segmented image
Fig. 3. Block diagram of proposed segmentation process 3.1 Image acquisition The sputum specimens consist of TB bacilli were obtained from Department of Microbiology and Parasitology, School of Medical Science, Universiti Sains Malaysia, Kubang Kerian. The sputum specimens have been stained using ZiehlNeelsen staining procedure. The sputum slides were analysed under 40x magnification using LEICA DMLA microscope and the images were captured using Infinity2 digital camera attached directly to the microscope. Figure 4 shows few samples of captured sputum slide images consist of TB bacilli. It can be seen that the TB bacilli appear to be red while the sputum background have bluish color. The original images are images which are directly captured using the digital camera which is attached to the microscope. Since it is manually prepared by the technologist, the thickness of the specimen may vary from one end of the slide to the other end. That is why; it results in the variation in the quality of the images being captured. 3.2 Pixel study A study on the color information on digital sputum slide images that consist of the tubercle bacilli and sputum was carried out to get the most suitable threshold values. The study was
Image Segmentation of ZiehlNeelsen Sputum Slide Images for Tubercle Bacilli Detection
(a)
369
(b)
(c)
(d)
Fig. 4. Samples of sputum slide images consisting of TB bacilli carried out on an amount of 244 TB bacilli with more than 2000 pixel samples together with more than 10000 background pixel samples. Sampling is done within five different category of images. The properties of the RGB pixels are being studied to extract the important features from the image. Based on the color information, the color thresholding algorithm should be able to extract the pixels of tubercle bacilli and reject pixels of other objects. In order to view the important properties of each segment so that necessary features and accurate value of threshold can be obtained from the result, the information is being gathered in a table. Among the features that are noted are the maximum, minimum and average values for each of the RGB components in tubercle bacilli and sputum respectively. It is found that the featured values are different from one category of images to another. Sample of featured values for normal and dark images are shown in Table 1 and Table 2.
RED GREEN BLUE
MIN 14 8 45
TB MAX 255 248 255
AVERAGE 175 157 196
Table 1. RGB information for images in normal category
MIN 14 15 61
Background MAX AVERAGE 255 230 255 239 255 252
370
Image Segmentation
From the data in Table 1, thresholding rules have been constructed based on average values. Rules for images from normal category are shown in Equation (3) – (5). ⎧ f ( x , y ), red( x , y ) < 230, g( x , y ) = ⎨ 255, red( x , y ) ≥ 230. ⎩
(3)
⎧ f ( x , y ),green( x , y ) < 239, g( x , y ) = ⎨ ⎩ 255,green( x , y ) ≥ 239.
(4)
⎧ f ( x , y ), blue( x , y ) < 252, g( x , y ) = ⎨ ⎩ 255, blue( x , y ) ≥ 252.
(5)
where red( x , y ) , green( x , y ) and blue( x , y ) are the pixel values for each of the red, green and blue components respectively.
RED GREEN BLUE
MIN 5 1 38
TB MAX 252 235 255
AVERAGE 155 148 183
MIN 0 0 33
Background MAX AVERAGE 255 160 255 188 255 217
Table 2. RGB information for images in dark category Since the average values between RGB information for images in normal category and dark category are different, separate set of rules has to be constructed for dark images. Rules for images from dark category are shown in Equation (6) – (8). ⎧ f ( x , y ), red( x , y ) < 160, g( x , y ) = ⎨ 255, red( x , y ) ≥ 160. ⎩
(6)
⎧ f ( x , y ),green( x , y ) < 188, g( x , y ) = ⎨ ⎩ 255,green( x , y ) ≥ 188.
(7)
⎧ f ( x , y ), blue( x , y ) < 217, g( x , y ) = ⎨ ⎩ 255, blue( x , y ) ≥ 217.
(8)
However, it is found that rules for one category of images are not universal and it can only be used within that particular image category. For example, rules for images in normal category will not give good result when it is applied to images in dark category. In order to overcome this problem, the images are brought through the image enhancement process so that the all the image from various categories are standardized into one same category. 3.3 Image enhancement Image enhancement process are carried out to overcome the problem risen in the previous stage. This involves the adjustment of brightness, contrast and color in an image so that the pixel values fall into about the same range. Image enhancement technique that is used in
Image Segmentation of ZiehlNeelsen Sputum Slide Images for Tubercle Bacilli Detection
371
this study is partial contrast. Stretching method is done by requantizing each pixel value to a new value using prespecified function. Linear stretching will generally improve the overall contrast of an image. Contrast stretching is a process that applies autoscaling method, which is a linear mapping function. It is usually used to enhance the brightness as well contrast level of the image. The general mapping function is shown in Equation (9) (Weeks, 1996). pk =
(max − min) ( q − fmin ) + min ( f max − f min ) k
(9)
Referring to Equation (9), fmax and fmin are the maximum and minimum color level in an input image. Variable max and min are the desired maximum and minimum color level in the output image. qk is the color level of the input pixel while pk is the color level of the output pixel. The combination of stretching and compressing process is called partial contrast. A part of the intensity level is being stretched to a new range, while other intensity levels left is being compressed to a different new range as well. The stretching and compressing processes are illustrated by Figure 5.
fmin
fmax qk
0 Compressing
Stretching
255 Compressing
pk
0 min
255 max
Fig. 5. Illustration of partial contrast process The process illustrated by Figure 5 can be put into a mathematical function such as in Equation (10).
⎧ min ⎪ (q ) ; for q k < f min ⎪ f min k ⎪ ⎪ ⎪⎪ (max − min) pk = ⎨ (q k − f min ) + min ; for f min ≤ q k ≤ f max ⎪ ( f max − f min ) ⎪ ⎪ ⎪ (255 − max) (q − f max ) + max ; for q k ≥ f max ⎪ (255 − f max ) k ⎩⎪
(10)
3.4 Color thresholding The color thresholding technique was carried out based on the color information of the bacterium to extract TB pixels from the sputum and other objects. This technique specifies
372
Image Segmentation
the range of RGB intensities for thresholding. The objects that lie outside the selection range will be rejected. Therefore, it is very important to determine the selection range because if this threshold cannot acquire a suitable value, the thresholding algorithm will extract pixels other than the expected object. After image enhancement is done, the process of pixel sampling is done once again using enhanced images and the information is gathered in a table to observe its properties. Table 3 reflect the featured values for all the images that have been enhanced.
RED GREEN BLUE
MIN 0 0 24
TB MAX 255 249 255
AVERAGE 185 161 206
MIN 0 0 27
Background MAX AVERAGE 255 235 255 246 255 254
Table 3. RGB information for enhanced images This time a universal rule for segmentation is produced as in Equation (11) – (13). ⎧ g( x , y ), red( x , y ) < 235, g( x , y ) = ⎨ 255, red( x , y ) ≥ 235. ⎩
(11)
⎧ g( x , y ),green( x , y ) < 246, g( x , y ) = ⎨ ⎩ 255,green( x , y ) ≥ 246.
(12)
⎧ 255, blue( x , y ) ≥ 252, g( x , y ) = ⎨ ⎩ g( x , y ), blue( x , y ) < 252.
(13)
From Equations (11), (12), and (13), it can be seen that the original equation that has been mentioned in Equations (1) and (2) have been slightly modified to adopt the method of grey level thresholding to color thresholding. For the new equations, each RGB component is being treated independently. Since there are three components, the thresholding process is being done to one component at a time, and they are then combined into 1 rule using a Boolean AND operator. One more important feature that has been extracted from the information gathered from the study is that in sputum images, the values of green pixels are always greater than the values of red pixels. Therefore, this information also has been adopted as another rule for this thresholding algorithm as in Equation (14). ⎧ 0, green( x , y ) < red( x , y ), g( x , y ) = ⎨ ⎩255, green( x , y ) ≥ red( x , y ).
(14)
Another modification that has been made is that, the output value is not 0 or 1, but either 255 (white pixel) or retaining the old value of the pixel. This means that if the value of that particular pixel falls in the range of the rule whereby the output value is 255, the original pixel value will be automatically changed to 255, which indicates that it is the area of sputum. However, if it is not fall within that range the original value of the pixel is retained to enable it to go through the next filtering algorithm. Note that since Equation (14) is the last rule for the whole algorithm, then the final value is either 0 (red pixel) or 255 (white pixel).
Image Segmentation of ZiehlNeelsen Sputum Slide Images for Tubercle Bacilli Detection
373
4. Results The rules that have been formulated are applied to the original raw image of ZiehlNeelsen sputum slide. Rules for normal images as shown in Equation (3) – (5) have been applied to these images. The result is good for normal images. However, for other types of images, the result is not satisfactorily achieved. Some of the pixels are lost and in some images, too many noises exist. This is reflected in Figure 6.
(a) Original normal image
(b) After threshold
(c) Original dark image
(d) After threshold
(e) Original bright image
(f) After threshold
Fig. 6. Original images from normal, dark and bright category with their respective result after applying rules for normal image
374
Image Segmentation
However, when the process of thresholding is done to the images with the rules formulated from its own category, the result is satisfying. In Figure 7, samples of dark images are thresholded using rules from dark category as shown in Equation (6) – (8). In Figure 8, samples of bright images are thresholded using rules from bright category. This result proves that rules for one category of images are not universal and it can only be used within that particular image category. To overcome the poor result, the images are brought through the image enhancement process. The results of thresholding to the image before the image enhancement process and the results of thresholding to the image after the image enhancement process are shown in Figure 9 to 11 .
(a) Original dark image 1
(b) After threshold
(c) Original dark image 2
(d) After threshold
Fig. 7. Original images from dark category with the result after applying rules for dark image
Image Segmentation of ZiehlNeelsen Sputum Slide Images for Tubercle Bacilli Detection
(a) Original bright image 1
375
(b) After threshold
Fig. 8. Original images from bright category with the result after applying rules for bright image
(a) Original Image
(c) Result of thresholding original image Fig. 9. Segmentation result of normal image
(b) Enhanced image
(d) Result of thresholding enhanced image
376
Image Segmentation
In order to determine whether the thresholding method that has been carried out is successful or not, it relies solely on human intervention. Therefore, the threshold value need to be varied until acceptable results are achieved, based on the human observation. That is why, in carrying the color thresholding procedure, it may be necessary to do a few level of thresholding in order to get the best results. The thresholding procedure must be done to the red, green and blue components. The thresholding on the three colors may be combined into one complete rule using the Boolean AND operator or it may also be separated into two or more rules. If more than one rule is being created, then it is considered to be done on a few level of thresholding. In this study, the process of thresholding is being done automatically based on the predetermined threshold value. In the case of sputum slide images, it can be seen from the original images that most of the pixels appears to be blue in color. The main objective is to filter out those blue pixels and retain the reddish pixels, which are the tubercle bacilli. Therefore, the first level thresholding has been carried out, which involves pixels of RGB components which have been combined using Boolean AND operator, as mentioned earlier in the methodology part. The second level filtering which involves the difference between green and red pixel is carried out, resulting in a binary image, in which the tubercle bacilli finally appears red, while the background which are originally blue, turn out to be white.
(a) Original Image
(c) Result of thresholding original image Fig. 10. Segmentation result of dark image
(b) Enhanced image
(d) Result of thresholding enhanced image
Image Segmentation of ZiehlNeelsen Sputum Slide Images for Tubercle Bacilli Detection
377
From the resulted images that have been presented, it can be said that this technique could be an alternative solution for the image segmentation of TB bacilli, and to further helps the process of TB bacilli identification as well as classification in sputum samples. Most of the research that have been done previously used fluorescence images of sputum (Veropoulos et al., 1998; Veropoulos et al., 1999; Forero et al., 2003; Forero et al., 2004), whereby this research concentrates on the ZiehlNeelsen stained images of sputum. Hence, it provides another option of TB bacilli identification especially for developing countries which are still sticking to this method for TB detection.
(a) Original Image
(c) Result of thresholding original image
(b) Enhanced image
(d) Result of thresholding enhanced image
Fig. 11. Segmentation result of bright image
5. Conclusion A technique of image segmentation by conducting a thresholding method for sputum slide images has been presented. The segmentation allows the elimination of a great amount of unwanted pixels, and retained only those pixels characterised to have similar color to the TB bacilli. The key to this method is to conduct a study on the color attribute of the tubercle bacilli in order to get the basic rules of selecting the most accurate threshold value. The resulted images satisfactorily showed that after the image enhancement process, and by using the selected threshold values, the image segmentation method has been able to filter out the sputum images from the tubercle bacilli images.
378
Image Segmentation
6. References Efford, N. (2000). Digital Image Processing: a practical introduction using Java, Pearson Education, ISBN 9780201596236, USA Forero, M. G.; Cristobal, G. & Borrego, J. A. (2003). Automatic identification techniques of tuberculosis bacteria, SPIE Proceedings Of The Applications Of Digital Image Processing XXVI, Vol. 5203, pp. 7181, ISBN 0819450766, San Diego, CA, Aug. 2003, SPIE, Bellingham WA Forero, M. G.; Sroubek, F. & Cristobal, G. (2004). Identification of tuberculosis bacteria based on shape and color. RealTime Imaging, Vol. 10, No. 4, (August 2004), pp. 251262, ISSN 10772014 Gonzalez, R. C. & Woods, R. E. (2002). Digital Image Processing Second Edition, Pearson Education, ISBN 9780201180756, New Jersey Khutlang, R.; Krishnan, S.; Dendere, R.; Whitelaw, A.; Veropoulos, K.; Learmonth, G. & Douglas T. S. (2010). Classification of Mycobacterium tuberculosis in images of ZNstained sputum smears. IEEE Transactions on Information Technology in Biomedicine, Vol. 14, No. 4, (July 2010), pp. 949957, ISSN 10897771 Luna, J. A. C. (2004). A Tuberculosis Guide for Specialist Physicians, International Union against Tuberculosis and Lung Disease, Paris MancasThillou, C. & Gosselin, B. (2005). Color binarization for complex camerabased images, Proceedings of the Electronic Imaging Conference of the International Society for Optical Imaging (SPIE/IS & T), pp. 301308, ISBN 0819456403 , San Jose (California, USA), Jan. 2005, SPIE, Bellingham WA Otsu, N. (1979). A threshold selection method from gray level histograms. IEEE Transactions on Systems, Man and Cybernetics, Vol. 9, No. 1, (Jan. 1979), pp. 6266, ISSN 00189472 Sezgin, M. & Sankur, B. (2004). Survey over image thresholding techniques and quantitative performance evaluation. Journal of Electronic Imaging, Vol. 13, No. 1, (Jan. 2004), pp. 146168, ISSN 10179909. Veropoulos, K.; Campbell, C. & Learmonth, G. (1998). Image processing and neural computing used in the diagnosis of tuberculosis, Proc. IEE Colloquium on Intelligent Methods in Healthcare and Medical Applications (Digest No. 1998/514), pp. 8/1  8/4, York, UK Veropoulos, K.; Learmonth, G.; Campbell, C. & Knight, B., Simpson, J. (1999). Automatic identification of tubercle bacilli in sputum. A preliminary investigation. Analytical and Quantitative Cytology and Histology, Vol. 21, No. 4, (Aug. 1999), pp. 277–81, ISSN 08846812 Weeks, J. R. (1996). Fundamentals of Electronic Image Processing, WileyIEEE Press, ISBN 9780780334106 WHO Report 2009 (2009). Global tuberculosis control 2009: Epidemiology, Strategy, Financing, WHO Press, World Health Organization, ISBN 978 92 4 156380 2, Geneva, Siwtzerland Wilkinson, M. (1996). Rapid automatic segmentation of fluorescent and phasecontrast images of bacteria, In: Fluorescence Microscopy and Fluorescent Probes. Slavik, J., pp. 261266, NY: Plenum Press, ISBN 0306453924, New York
20 Image Segmentation Based on a TwoDimensional Histogram Masmoudi Lhoussaine1, Zennouhi Rachid1 and Mohamed EL Ansari2 1Mohammed 2Ibn
V University (LETS Laboratory, Faculty of Science) Zohr University (LabSIV Laboratory, Faculty of Science) Morocco
1. Introduction Image segmentation refers to the partitioning of an image into nonoverlapping different regions with similar attributes. For gray level images, the most basic attribute used is the luminance amplitude, and for color or multispectral images, color or information components are used. Various methods are found in the literature and are roughly classified into several categories according to the dominant features they employ. This includes edgebased methods (Zugaj & Lattuati, 1998), region growing methods (Tremeau & Borel, 1998; Schettini, 1993), neural networks methods, physicsbased methods (Maxwell & Shafer, 1996; Bouda et al. 2008) and histogram thresholding methods (Sezgin & Sankur,2004). It is demonstrated that in unsupervised classification cases the histogram threshold method is a good candidate for achieving segmentation for a wide class of gray level images with low computation complexity (Cheng et. al., 2001). This method ignores the spatial relationship information of the pixels that can give improper results. Abutaleb’s work (Abutaleb, 1989) presents another type of 2D gray level histogram. It is formed by the Cartesian product of the original 1D gray level histogram and 1D local average gray level histogram generated by applying a local window to each pixel of the image and then calculating the average of the gray level within the window. Zhang and al. (Zhang & Zhang, 2006) proposed using a minimum gray value in the 4neighbor and the maximum gray value in the 3×3 neighbor except pixels of the 4neighbor. This method’s main advantage is that it does not require prior knowledge regarding the number of objects in the image, and classical and fast gray level image processing algorithms can be used to cluster the 2D histogram (Clement, 2002). For color or multispectral images, the onedimensional (1D) histogram method detracts from the fact that a color cluster is not always present in each component and the combination of the different segmentations cannot catch this spatial property of colors (Clément & Vigouroux, 2001). It also does not take into account the correlation between components (Uchiyama & Arbib, 1994). Therefore multiple histogrambased thresholding is required. However, in a full multidimensional manner, the threedimensional histogram (3Dhistogram) method is handicapped by data sparseness, the complexity of the search algorithm (Lezoray & Cardot,2003) and a huge memory space (Clément & Vigouroux, 2001). An interesting alternative method lies with the use a partial histogram (2Dhistogram)(
380
Image Segmentation
Kurugollu et al.,2001), obtained by projecting a 3Dhistogram onto two color planes (Clément & Vigouroux, 2003). This has several advantages, including a lack of data encountered in the 3D case, such as RGB image color, that is partially overcome and the search complexity is drastically reduced ( Lezoray & Cardot,2003). Another advantage is the fact that a 2Dhistogram is nothing more than a gray level image. Therefore classical and fast gray level image processing algorithms can be used to cluster the 2Dhistogram ( Clément, 2002). It is noted that the HSV color space is fundamentally different from the widely known RGB color space since it separates the intensity from the color information (chromaticity). HSV space was demonstrated to be a perceptual color space that consists of the three components H (hue), S (saturation) and V (value) and corresponds to the color attributes closely associated with the way human eyes perceive the colors. Many works related to the HSV color image have been developed and used (Qi et al., 2007; Sural et al, 2002 ; Zennouhi & Masmoudi, 2009). The organization of this chapter is as follows: in section 2, the 2Dhistogram strategy is presented. Section 3 details the segmentation algorithm based on a 2Dhistogram using HSV space. The experimental results are presented and discussed in section 4. Section 5 concludes the chapter.
2. Twodimensional histogram The histogram threshold method is a good candidate for gray level image segmentation (Cheng et. al., 2001). It is based on the shape of the histogram properties, such as the peaks, valleys and curvatures of the smoothed histogram (Sezgin et Sankur, 2001). Abutaleb’s work (Abutaleb, 1989) presents another type of 2D gray level histogram. It is formed by the Cartesian product of the original 1D gray level histogram and 1D local average gray level histogram generated by applying a local window to each pixel of the image and then calculating the average of the grey levels within the window. The change in the pixel value in the horizontal or vertical directions appears slow and the gradation change continuity appears strong compared to the change in the diagonal direction. Zhang and al. (Zhang & Zhang, 2006) proposed using a minimum gray value in the 4neighbor and the maximum gray value in the 3×3 neighbor except pixels of the 4neighbor. This method’s main advantage is that it does not require prior knowledge about the number of objects in the image. For RGB color or multispectral image, the onedimensional (1D) histogram method detracts from the fact that a color cluster is not always present in each component and the combination of the different segmentations cannot catch this spatial property of colors (Clément & Vigouroux, 2001). It also does not take into account the correlation between components (Uchiyama & Arbib, 1994). Therefore multiple histogrambased thresholding is required. However, in a full multidimensional manner, the 3Dhistogram method is handicapped by data sparseness, the complexity of the search algorithm (Lezoray & Cardot,2003) and a huge memory space (Clément & Vigouroux, 2001). An interesting alternative method lies with the use of 2Dhistogram (Kurugollu et al.,2001), which selects two color bands together, namely RG, RB or GB in the RGB space color, obtained by projecting a 3Dhistogram onto two color planes which can be constructed as follows. A 2Dhistogram pn of a RGB color image I maps p(x1 , x 2 ) , the number of pixels in image I presenting the colorimetric components (x1 , x2 ) . Since each colorimetric axis of image I is
Image Segmentation Based on a TwoDimensional Histogram
381
quantified on 256 levels, the 2Dhistogram pn can be represented by an image J whose spatial resolution is equal to 256x256. The value pn (x1 , x 2 ) of the pixel of coordinates (x1 , x 2 ) in J is obtained by a linear dynamic contraction of the histogram between 1 and M = min(pmax , 255) (Clément & Vigouroux, 2003):
⎡ (M − 1)p(x1 , x 2 ) − Mpmin + pmax ⎤ pn (x1 , x 2 ) = round ⎢ ⎥ pmax − pmin ⎢⎣ ⎥⎦
(1)
where pmin and pmax are respectively the minimum and maximum values of p.
3. Image segmentation algorithm This section presents the segmentation algorithm based on 2Ddimensional histogram analysis using HSV space. 3.1 Twodimensional histogram using HSV space A three dimensional representation of the HSV color space is a hexacone that consists of three components: Hue, Saturation, and value. Hue is a color attribute that describes what a pure color (pure yellow, orange, or red) is. Hue refers to the perceived color (technically, the dominant wavelength). As hue varies from 0 to 1.0, the corresponding colors vary from red, through yellow, green, cyan, blue, and magenta, back to red, so that there are actually red values both at 0 and 1.0. Saturation is a measure of the degree to which a pure color is diluted by white light, giving rise to ‘light purple’, ‘dark purple’, etc. It can be loosely thought of as how pure the color is. Greater values in the saturation channel make the color appear stronger. Lower values (tending to black) make the color appear washed out. As saturation varies from 0 to 1.0, the corresponding colors (hues) vary from unsaturated (shades of gray) to fully saturated. As value, or brightness, varies from 0 to 1.0, the corresponding colors become increasingly brighter (Chen & Wu, 2005). It is noted that the HSV color space is fundamentally different from the widely known RGB color space since it separates the intensity from the color information (chromaticity). And it was demonstrated that the HSV components correspond to the color attributes closely associated with the way human eyes perceive the colors. Many works related to the color image have been developed using this color space (Qi et al., 2007; Sural et al, 2002). Sural et al. (Sural et al., 2002) analyzed the properties of the HSV color space with emphasis on the visual perception of the variation in hue, saturation and intensity values of an image pixel. For a given intensity and hue if the saturation is changed from 0 to 1, the perceived color changes from a shade of gray to the most pure form of the color represented by its hue. Looked at from a different angle, any color in the HSV space can be transformed to a shade of gray by sufficiently lowering the saturation. The saturation threshold that determines the transition, between the low and the higher values of the saturation, is once again dependent on the intensity. It is illustrated by Sural et al.(Sural et al., 2002) that for higher values of intensity, a saturation of 0.2 differentiates between hue and intensity dominance. Assuming the maximum intensity value to be 255, the threshold function to determine if a pixel should be represented by its hue or its intensity as its dominant feature is given by
382
Image Segmentation
Th s (V) = 1 −
0.8V 255
(2)
That can lead to a feature vector of two parts: the hue values between 0 and 2π quantized after a transformation and a quantized set of intensity values. In this way we use the HSV color space to build the histogram where each pixel contributes to either its hue or its intensity. Based on the threshold function equation (2), we determine an intermediate image (Fig. 1a): for low values of saturation, a color can be approximated by a gray value specified by the intensity level, while for higher saturation; the color can be approximated by its hue value.
Fig. 1. 3x3 block : (a) and (c) intermediate image; (b) and (d) their component in the original image Subsequently we can construct the 2D color histogram for the intermediate image as follow: for each block of 3x3 pixels of the intermediate image, we consider the central pixel which can be an intensity Component or a hue Component (Fig. 1a and c), according to the equation (2), and we calculate the maximum (Max) and the minimum (Min) in its corresponding component (H or V) in the original image (Fig. 1b and d) (zennouhi & Masmoudi, 2009). Where the Max is the maximum hue or intensity in the 3x3 neighbor except pixels of the fourneighbor and Min is the minimum hue or intensity in the fourneighbor
Fig. 2. 2D histogram
383
Image Segmentation Based on a TwoDimensional Histogram
Max = maximum [(v4, v2, v6, v8) or ( h4, h2, h6, h8)] and Min = minimum[(v0,v1,v3,v5,v7) or (h0,h1,h3,h5, h7)] Then we build the 2Dhistogram by mapping the number of pixels presenting the minimum and the maximum (Min, Max) in all 3x3 block of the image according to Ph(Min; Max) and Pv(Min; Max) for central pixel represented by H or V component respectively (Fig.2). 3.2 Segmentation algorithm The segmentation algorithm is achieved in two main steps: one, building the 2Dhistogram using HSV space according the approach developed before. Two, detecting the peaks representing classes using the classification algorithm proposed in Ref. (Clément & Vigouroux, 2003). A peak is labeled as significant if it represents a population greater than or equal to a threshold d0 (expressed in per cent of the total population in image). The classification algorithm is performed by reclassification of the pixels not classified in the determined classes according to Euclidian distance. In order to evaluate the segmentation algorithm, we use the Q function (Zhang et al., 2008; Borsotti et al. , 1998)
2⎞ ⎛ R Ni ⎟ R ⎜ ei2 1 Q(Im) = R× ∑ ⎜ + ⎟ 10000 N Ni2 ⎟ i = 1 ⎜ 1 + logNi ⎝ ⎠
( )
(3)
Im is the segmented image, N is the image size, R is the number of regions of the segmented image, Ni is the area of the ith region, R(Ni) is the number of regions having an area equal to Ni, and ei is the average color error of the ith region, which is defined as the sum of the Euclidean distances between the RGB color vectors of the pixels of the ith region and the color vector attributed to the ith region in the segmented image. The smaller the Q value, the better the image segmentation method.
4. Experimentation results This section presents the experimentation results obtained on synthetic and real images. Two color images are selected: the 465x386 synthetic Squares, which is comprised of four colors, the 709x608 real Mandrill image. Figure 3 shows the original images and the results of the proposed method. It can be seen that the performance is acceptable both for synthetic and real images. In order to compare the performance of the segmentation method with other existing ones, two different images are used. First, we consider a synthetic image (Gillet et al., 2002, Macaire et al., 2006) that contains four patterns with different shapes and different colors. The circular pattern contains two shapes that differ only by the variation of the saturation component. The image contains then, if we consider the background, six classes. The difference between two regions due to the variation of saturation is still a difficult problem. Often, two distinct colors are merged together. The 2Dhistogram segmentation method using RGB space fails to separate the two shapes in the circular pattern. The same problem arises when we applied the 1Dhistogram method using HSV space. It can be seen from experimentation results (Fig. 4) that the proposed
384
Image Segmentation
(a)
(b)
(c)
(d)
Fig. 3. (a,c) original images; (b,d) segmented images
(a)
(b)
(c )
(d)
Fig. 4. (a) synthetic image; (b) RGhistogram; (c) 1D histogram using HSV space; (d) 2D histogram using HSV space. approach gives better clustering and the problem of missing to separate the two shapes is alleviated. From theses results, it can be deduced that RGB space is not able to separate the two shapes in the circular pattern. This is due to the high correlation among the R, G and B components and that the measurement of a color in RGB space does not represent color differences in a uniform scale; hence it is impossible to evaluate the similarity of two colors from their distance in RGB space. However, in the HSV space where the color and intensity information are separated, the segmentation of the two shapes can be achieved by the proposed method. Second, we are interested in some agricultural applications. Plants are exposed to a multitude of natural biotic and abiotic stresses (Lichtenthaler,1996). Water availability is one
385
Image Segmentation Based on a TwoDimensional Histogram
of the most important limitations to photosynthesis and plant productivity (Tezara et al., 1999). The proper monitoring of plant water stress is essential for the development of appropriate, sustainable irrigation programs for crop production in semiarid areas (Penuelas & Filella, 1998). The use of nondestructive imaging methods, such as fluorescence imaging, thermal imaging and imaging using near infrared, holds great promise for early, efficient and objective detection of plant responses to various stresses (Govindjee and Nedbal, 2000; Chaerle & van der Straeten, 2001 ). However, these techniques provide less human intuition, are more difficult to assess during system integration and are the most costly and time consuming. So, the use of the imaging based on the electromagnetic radiation in the visible range would be of great interest.
(a)
(b)
(c)
(d)
Fig. 5. (a) menthe image; (b) RGhistogram; (c) 1D histogram using HSV space; (d) 2D histogram using HSV space In this study we have considered a medicinal plant. The first step in our procedure to detect early stress is to segment each image into two classes: vegetation and soil. The color RGB images of the plant are provided by a digital camera. Each color plane is quantized on 256 levels with a resolution of 640x480. It was noted that the 2D histogram method using RGB space fails to determine the color and intensity variation presented in the image plant. It can be seen from the Fig. 5b that the RGB method could not separate the plant and pot. Figure 5c shows the obtained result of 1D histogram using HSV space. It is clear that the plant and pot are separated; however, this method could not separate the variation intensity presented in soil class which can be useful information for the study of the plant environment. In contrast, it can be clearly seen from Fig. 5d that the performances of the proposed method are higher than those presented for comparison. Finally, to evaluate the proposed technique, we have used the ‘Q’ evaluation function equation (3). From Table 1, it can be seen that the proposed method performs better than the analysis of 2Dhistogram using RGB space.
386
Image Segmentation
Square image
Mandrill image
Synthetic image (Fig 4)
Reel image (Fig.5)
2Dhistogram using RGB space
35 ,5067
52128
8080,9
8765,8
2Dhistogram using HSV space
33 ,0656
17367
721 ,7661
6466,8
Table 1. Values of evaluation function ‘Q’ for various images
5. Conclusion In this chapter, we have developed an approach of color image segmentation which is based on the analysis of 2Dhistogram using HSV space. The method was applied to various synthetic and real images to prove the performance of segmentation algorithm. Additionally, the method was applied to a particular agricultural application to separate the vegetation and soil. The obtained results have been compared to the methods of others, and shown to be more satisfactory than those obtained either by 1Dhistogram using HSV or by 2Dhistogram using RGB space.
6. References Abutaleb, A. S.(1989). Automatic thresholding of graylevel pictures using twodimensional entropy. Journal of Computer Vision, Graphic and Image Process, Vol. 47, (1989) 22–32 Borsotti, M.; Campadelli, P. & Schettini, R.(1998). Quantitative evaluation of color image segmentation results. Patt. Recogn. Lett., Vol.19, (1998) 741–747 Bouda, B.; Masmoudi, L. & Aboutajdine, D (2008) . Cvvefm : Cubical voxels and virtual electric field model for detection in color images. Signal Processing, Elsevier, Vol. 88, No. 4, (2008) 905915 Chaerle, L. & van der Straeten, D.(2001). Seeing is believing: imaging techniques to monitor plant health. Biochim. Biophys. Acta, Vol. 1519,(2001) 153–166 Chen, C. & Wu, W. (2005). Color pattern recognition with the multichannel nonzeroorder joint transform correlator based on the HSV color space. Optics Communications, Vol.244 (2005) 51–59 Cheng, H.D. ; Jiang, X.H.; Sun, Y.& Wang, J.(2001). Color image segmentation: advances and prospects. Pattern Recognition, Vol.34, No.6, (2001) 22592281 Clément, A. & Vigouroux, B. ( 2001). Un histogramme compact pour l’analyse d’images multicomposantes, Actes du 18e Colloque sur le Traitement du Signal et des Images: GRETSI’01, pp.305–307, Centre de congres Pierre Baudis,2001 Vol. 1, Toulouse, France Clément, A. & Vigouroux, B.(2003). Unsupervised segmentation of scenes containing vegetation (Forsythia) and soil by hierarchical analysis of bidimensional histograms. Patt. Recogn. Lett., Vol. 24,(2003) 1951–1957
Image Segmentation Based on a TwoDimensional Histogram
387
Clément,A.(2002). Algorithmes et outils informatiques pour l’analyse d’images couleur. Application à l’étude de coupes histologiques de baies de raisin en microscopie optique, PhD thesis, Université d’Angers. Gillet, A.; Macaire, L.; BotteLecocq, C. & Postaire, J. G.(2002). Color image segmentation by analysis of 3D histogram with fuzzy morphological filters, in: SpringerVerlag Editor, Fuzzy Filters for Image ProcessingStudies in Fuzziness and Soft Computing, pp. 154177, New York Govindjee & Nedbal, G. L. (2000). Seeing is believing. Photosynthetica, Vol. 38, (2000) 481– 482 Kurugollu, F.; Sankur, B. & Harmanci, A.(2001). Color image segmentation using histogram multithresholding and fusion. Image and Vision Comput., Vol. 19, (2001) 915–928. Lezoray, O. & Cardot, H. (2003). Hybrid color image segmentation using 2D histogram clustering and region merging, Proc. Int. Conf. on Image and signal processing: ICISP’03, pp. 22–29, Agadir, 2003, Maroc Lichtenthaler, H. K.(1996). Vegetation stress: an introduction to the stress concept in plants. J. Plant Physiol., Vol. 148,(1996) 4–14 Macaire, L.; Vandenbroucke, N. & Postaire, J. G.(2006). Color image segmentation by analysis of subset connectedness and color homogeneity properties. Comput. Vision Image Understand., Vol.102, (2006) 105–116 Maxwell, B.A. & Shafer, S.A. (1996). Physicsbased segmentation: looking beyond color, Proceedings of Image Understanding Workshop, pp. 867–878, Palm Springs, CA February 1996, ARPA, USA Penuelas, J. & Filella, I.(1998). Visible and nearinfrared reflectance techniques for diagnosing plant physiological status. Trends Plant Sci., Vol.3, (1998)151 –156 Qi, X. J.; Qi, J. & Wu, Y. J. (2007). RootLM: a simple color image analysis program for length measurement of primary roots in Arabidopsis. Plant Root, Vol. 1, (2007) 10–16 Schettini, R. (1993). A segmentation algorithm for color images. Pattern Recognition Letters, Vol.14, (1993) 499–506 Sezgin, M. & Sankur, B. (2004). Survey over image thresholding techniques and quantitative performance evaluation. Journal of Electronic Imaging, Vol.13, No.1,(2004) 146165 Sural, S.; Qian, G. & Pramanik, S.(2002). Segmentation and histogram eneration using the HSV color space for image retrieval, Proc. Int. Conf. on Image processing: ICIP’02, Rochester, NY, USA, IEEE, Vol. 2, pp. II589–II592 Tezara, W.; Mitchell, V. J.; Driscoll, S. D. & Lawlor, D. W. (1999). Water stress inhibits plant photosynthesis by decreasing coupling factor and ATP. Nature, Vol. 401, (1999) 914–917 Tremeau, A. & Borel, N. (1998). A region growing and merging algorithm to color segmentation, Pattern Recognition, Vol.30, No.7, (1998)11911203 Uchiyama, T. & Arbib, M.(1994). Color image segmentation using competitive learning. IEEE Trans. Pattern Anal. Mach. Intell., 16, (1994) 1197–1206
388
Image Segmentation
Zennouhi, R. & Masmoudi, LH. (2009). A new 2D histogram scheme for color image segmentation. The Imaging Science Journal, Vol. 57, (2009) 260265 Zhang, H.; Fritts, J. E. & Goldman, S. A.(2008). Image segmentation evaluation: A survey of unsupervised methods. Computer Vision and Image Understanding, Vol. 110, (2008) 260–280 Zhang, Y. F. & Zhang, Y. (2006). Another Method of Building 2D Entropy to Realize Automatic Segmentation. Journal of Physics Conference Series, Vol. 48, (2006) 303– 307 Zugaj, D. & Lattuati, V. (1998). A new approach of color images segmentation based on fusing region and edge segmentation outputs. Pattern Recognition, Vol. 31, No.2,(1998) 105–113
21 Segmentation Methods for Biomedical Images Roberto Rodríguez Morales
Institute of Cybernetics, Mathematics & Physics (ICIMAF), Digital Signal Processing Group Cuba 1. Introduction Nowadays biomedical imaging provides major aid in many branches of medicine; it enables and facilitates the capture, transmission and analysis of biomedical images as well as providing assistance towards medical diagnoses. Medical imaging is still on the rise with new imaging modalities being added and continuous improvements of devices’ capabilities. Segmentation and contour extraction are important steps towards image analysis. Segmented images are now used routinely in a multitude of different applications, such as, diagnosis, treatment planning, localization of pathology, study of anatomical structure, computerintegrated surgery, among others. However, image segmentation remains a difficult task due to both the variability of object shapes and the variation in image quality. Particularly, biomedical images are often corrupted by noise and sampling artifacts, which can cause considerable difficulties when applying rigid methods. In many occasions, segmentation is considered as a process of classification of objects in a scene, and also in certain measure, it is equivalent to its recognition (objects) since, as a consequence of segmentation the different objects (physical realizations of classes or abstract patterns), these stay perfectly located inside the digital image. In a graphic way in Figure 1, it is represented the segmentation process of a simple image. It is appreciated in this Figure that as result of segmentation the image was transformed from a gross digital image with all their information in form of graylevels to an image much more simplified, where the different objects are well distinguished.
1 Segmentation
3
2
4
Fig. 1. Simple outline that represents the segmentation process Many segmentation methods have been proposed for biomedicalimage data [ChinHsing et al., 1998; Kenong & Levine, 1995; Koss et al., 1999; Rodríguez et al., 2002; Sijbers et al., 1997;
390
Image Segmentation
Schmid, 1999]. Unfortunately, segmentation using traditional lowlevel image processing techniques, such as thresholding, histogram and other classical operations, requires a considerable amount of interactive guidance in order to get satisfactory results. Automating these modelfree approaches is difficult because of shape complexity, shadows, and variability within and across individual objects. Furthermore, noise and other image artifacts can cause incorrect regions or boundary discontinuities in objects recovered from these methods. However, it is accepted that despite the more complex algorithms developed so far, the segmentation of images remains highly dependent on the application and a single method has not yet been found that can solve all the problems in the universe. In mathematical morphology (MM) important methods have been developed for image segmentation [Vicent & Soille, 1991; Vincent, 1993]. One of the most powerful tools developed in MM is the Watersheds transformation, which is classic in the field of topography and it has been used in many problems of image segmentation. However, the Watersheds transformation has the disadvantage of producing oversegmentation [Fuh et al., 1991; Najman & Schmitt, 1996]. For that reason, the correct way to use watersheds for grayscale image segmentation is to mark the regions we want to segment; that is, the objects, but also the background. One and only one marker must correspond to each region. The design of robust marker detection techniques involves the use of specific knowledge of the series of images under study. Other technique very used in the last years for image segmentation is the mean shift, which is a nonparametric procedure for multimodal data analysis and has shown great superiority in many applications of highlevel tasks [Comaniciu, 2000]. The mean shift was proposed in 1975 by Fukunaga et. al. [Fukunaga & Hostetler, 1975] and was largely forgotten until Cheng in 1995 rekindled interests in it [Cheng, 1995]. A computer model based on the mean shift procedure is an extremely versatile tool for feature analysis and can give a reliable solution for many tasks of computer vision [Comaniciu & Meer, 2002]. Segmentation by using the mean shift carries out a smoothing filter as a first step before segmentation is performed [Comaniciu, 2000]. The aim of this chapter is to present the advances that the author and their collaborators have obtained in the biomedical image segmentation. Also, some strategies that constitute appropriate tools are presented, which it can be used in many system of image analysis where methods of segmentation are required. This chapter is structured as follows: In Section 2 the most significant theoretical aspects are detailed of each of the used methods in biomedical image segmentation. In Section 3 some of the characteristics of the studied images are described. In Section 4 the steps carried out in the experimentation are explained, and also an analysis and discussion of the proposed strategies is carried out. In this section two algorithms are presented to carry out the segmentation. Finally, in Section 5 the most important conclusions of this chapter are given.
2. Theoretical aspects In mathematical morphology, it is usual to consider that grayscale images are often considered as topographic reliefs. In the topographic representation of a given image I, the numerical value (that is, the gray tone) of each pixel stands for the elevation at this point. Such a representation is extremely useful, since it allows one to better appreciate the effect of the different transformations on the image under study.
391
Segmentation Methods for Biomedical Images
2.1 Morphological grayscale reconstruction Now, it is given a group of definitions and concepts which are important for a better understanding of the proposed strategies. Let us consider a twodimensional grayscale picture I whose definition domain is denoted DI ⊂ Z2. I is supposed to take discrete (gray) values in a given range [0, L1], L being an arbitrary positive integer
⎛ D ⊂ Z 2 → { 0, 1, 2, .........., L  1} ⎞ I ⎟ ⎜p ⎟ → I ( p) ⎝ ⎠
I⎜
(1)
Let J and I be two grayscale images defined on the same domain DI, taking their values in the discrete set {0, 1, ........, L1} and such that J ≤ I (i.e., for each pixel p Є DI , J(p) ≤ I(p)). In this way, it is useful to introduce the geodesic dilations according to the following definition [Vincent, 1993]: Definition 2.1.1 (Geodesic dilation): The elementary geodesic dilation of δ I(1) ( J ) of grayscale image J ≤ I, J “under” I (J is called the marker image and I is the mask) is defined as,
δ I(1) ( J ) = ( J ⊕ B) ∧ I
(2)
where the symbol Λ stands for the pointwise minimum and J ⊕ B is the dilation of J by flat structuring element B. The grayscale geodesic dilation of size n ≥ 0 is obtained by,
δ I( n ) ( J ) = δ I(1) δ I(1) ⋅ ⋅ ⋅ ⋅ ⋅ δ I(1) ( J ), n times
(3)
Starting from the definition 2.1.1 and the expression (3) it can be derived an efficient definition for the morphological reconstruction of images in graylevels. Definition 2.1.2 (Grayscale reconstruction): The grayscale reconstruction ρI( J ) of I from J is obtained by iterating grayscale dilations of J “under” I until stability is reached [Vincent, 1993], that is,
ρI ( J ) =
∨δ
n ≥1
(n) I
(J)
(4)
Definition 2.1.3 (Geodesic erosion): Similarly, the elementary geodesic erosion ε I(1) ( J ) of grayscale image J ≥ I, J “above” I is given by,
ε I(1) ( J ) = ( J θ B) ∨ I
(5)
where ∨ stands for the pointwise maximum and J θ B is the erosion of J by flat structuring element B. The grayscale geodesic erosion of size n ≥ 0 is then given by,
ε I( n ) ( J ) = ε I(1) ε I(1) ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ⋅ ε I(1) ( J ), n times
(6)
Definition 2.1.4 (Dual reconstruction): The dual grayscale reconstruction ρ ∗f ( J ) of mask I from marker J is obtained by iterating grayscale geodesic erosions of J “above” I until stability is reached; that is,
392
Image Segmentation
ρ I∗ ( J ) = ∧ ε I( n ) ( J )
(7)
n≥1
Reconstruction turns out to provide a very efficient method to extract regional maxima and minima from grayscale images. Furthermore, the technique extends to the determination of maximal structures, which will be call hdomes and hbasins. The hdomes transformation extracts light structure without involving any size or shape criteria. The only parameter (h) is related to the height of these structures. The mathematical background and other definitions can be found in [Vincent, 1993]. 2.1.1 Dome extraction and regional maxima We consider the successive thresholds Th (I) of I, for h = 0 to L1 [Vincent, 1993],
Th ( I ) = { p ∈ DI / I ( p) ≤ h }
(8)
They are said to constitute the threshold decomposition of I, where these sets satisfy the following inclusion relationship: Th (I) ⊆ Th1 (I)
∀
h ∈ [0, L1]
(9)
Definition 2.1.1.1 (Regional Minimum): A regional minimum M at altitude h of grayscale image I is a connected component C of Th (I) such that C ∩ Th1 (I) = φ , Th (I) being a threshold of I at level h and the symbol φ stands for empty set. Definition 2.1.1.2 (Regional maximum): A regional maximum at altitude h of grayscale image I is a connected component C de Th (I) such that C ∩ Th+1(f) = φ . In Figure 2 one can observe a representation of regional maxima in a grayscale image. Regional maxima
Fig. 2. Regional maxima in a grayscale image Regional maxima should not be mistaken with local maxima. A pixel p of I is a local maximum for grid H if and only if its value I (p) is greater or equal to that of any of its neighbours. All the pixels belonging to a regional maximum are local maxima, but the converse is not true. For example, a pixel p belonging to the inside of a plateau is a local maximum, but the plateau may have neighbouring pixels of higher altitude and thus not be a regional maximum. One of the most efficient methods makes use of grayscale reconstruction and is based on the following proposition. Proposition 2.1.1.1: The (binary) image M (I) of the regional maxima of I is given by,
393
Segmentation Methods for Biomedical Images
M( I ) = f − ρ I ( I  1) where ρ I ( I  1) , it is a morphological reconstruction (see definition 2.1.2, in this case J = I1). The proof of this proposition can be seen in [Vincent, 1993]. For a better understanding of this proposition the same is illustrated in Figure 3. Regional maxima
⇒
I
Grayscale reconstruction I1
Fig. 3. Extracting the regional maxima of I by reconstruction of I from I1 The proposition 2.1.1.1 can be generalized and instead of subtracting value 1, an arbitrary graylevel constant h can be subtracted from I. This provides a useful technique for extracting domes of a given height, which is called hdomes. Then, be the following definition: Definition 2.1.1.3: The hdome image D h (I) of the hdomes of a grayscale image I is given by Dh ( I ) = I ρ I ( I  h ) The hdome transformation is illustrated on Figure 4. Unlike the classical transformation TopHat, the hdome transformation extracts light structures without involving any size or shape criterion. The only parameter (h) is related to the height of these structures [Vincent, 1993].
h I Reconstructio Ih
Subtraction
Fig. 4. Determination of the hdomes of grayscale image I
394
Image Segmentation
2.1.2 Watershed segmentation method The Watershed segmentation consists in extracting objects from a graylevel image as precisely as possible [Vicent & Soille, 1991]. The advantage of this technique is that totally closed contours are obtained, and a complete partition of the domain of image is achieved. With this method is not necessary to carry out an additional connection of the edges. The disadvantage of this method resides in that it produces oversegmentation when the objects (to extract) are not appropriately marked. The following definitions and concepts are important for understanding better this method. In the following, this work equally considers grayscale images as numerical functions or as topographic reliefs. Let H denote the underlying digital grid, which can be of any type: a square grid in four or eight connectivity, or a hexagonal grid in six connectivity. H is a subset of Z2 x Z2. Definition 2.1.2.1: A path P of length l between two pixels p and q in image I is a (l+1)tuple of pixels (p0, p1, ……., pl1, pl) such that p0 = p, pl = q, and ∀i ∈ [1, l ],( pi − 1 , pi ) ∈ H
Given the previous definition, it will be denoted l (P) as the length of a given path P. Also, it will be denoted NH (p) as the set of the neighbours of a pixel p, with respect to H; NH (p) = { p´∈ Z2, (p, p´) ∈ H }. Definition 2.1.2.2 (Catchment basin, first definition): Let I be a grayscale image. The catchment basin C (M) associated with a minimum M is the set of pixels p of DI such that a water drop falling at p flows down along the relief, following a certain descending path called the downstream of p, and eventually reaches M [Vicent & Soille, 1991]. The lines which separate different catchment basins build what is called the watersheds (or dividing lines) of I. (see Figure 5).
“Catchment basins”
Dividing lines
Mínima Fig. 5. Minima, catchment basins, and dividing lines Note that the catchment basins of an image I correspond to the influence zones of its minima, also called areas of attraction of the drops of water. In this sense, it there is a close
395
Segmentation Methods for Biomedical Images
relation between the binary skeleton by influence zones and the watersheds. Therefore, this intuitive approach to watershed in not well suited to practical implementations [Vicent & Soille, 1991]. Then, the given definitions in the next section are more suited to the formalization of catchment basins and watersheds in digital space. Definition 2.1.2.3 (Catchment basin by immersion): Suppose, that in each regional minimum of I is pierced a hole, this image being regarded as a topographic surface. Then, this image is slowly immersed into a lake. Starting from the minima of lowest altitude, the water will progressively fill up the different catchment basins of I. Now, at each pixel where the water coming from two different minima would merge, a “dam” is built (see Figure 6). At the end of this immersion procedure, each minimum is completely surrounded by dams, which delimit its associated catchment basin. The whole set of dams which has been built thus provides a tessellation of I in its different catchment basins. These correspond to the dividing lines of I. It is possible to express the immersion procedure more formally [Vicent & Soille, 1991]: I being the grayscale image under study, denote hmin the smallest values taken by I on its domain DI. Similarly, denote hmax the large value by I on DI. In the following, Th (I) stands for the threshold of I at level h (see equation (8)). Denote C (M) the catchment basin associated with a minimum M and Ch (M) the subset of this catchment basin made of the points having an altitude smaller or equal to h: Ch (M) = {p ∈ C (M), f (p) ≤ h} = C (M) ∩ Th ( f )
(10)
As concerns the minima of I, minh (I) refers to the set of points belonging to the minima at altitude h.
Catchment basins
Dams
Minima
Fig. 6. Building dams at the places where the water coming from two different minima would merge
396
Image Segmentation
It is necessary for further understanding in what continues to introduce the definitions of geodesic distance and of the geodesic influence zones. Let X be a set which is first supposed to be simply connected. Definition 2.1.2.4: The geodesic distance dX (p, q) between two pixels p and q in X is the infimum of the length of the paths which join p and q and are totally included in X (see Figure 7): DX (p, q) = inf {l (P), P path between p and q which is totally included in X}
Fig. 7. Geodesic distance between p and q inside X Suppose that X contains a set B made of several connected components B1, B2, …….., Bk. Definition 2.1.2.5: The geodesic influence zone izX (Bi) of a connected component Bi in X is the locus of the points of X whose geodesic distance to Bi is smaller than their geodesic distance to any other component of B ( see Figure 8): izX(Bi) = {p ∈ X, ∀ j ∈ [1, k]/ {i}, dX (p, Bi) < dX (p, Bj) }
(11)
X
B3 B1 B2 izX (B1) Fig. 8. Geodesic influence zone of connected component B1 inside set X Those points of X which do not belong to any geodesic influence zone constitute the skeleton by influence zones, (SKIZ) of B inside X, denoted SKIZX (B): SKIZX (B) = X / izX (B) with izX (B) =
∪
i ∈ [1; k]
iz x (Bi )
(12)
In [Vicent & Soille, 1991], it is demonstrated that the previous definitions are easily extended to the case where X is not simply connected, nor even connected at all. Then, finally the following definition is presented.
397
Segmentation Methods for Biomedical Images
In order to simulate the immersion procedure, one starts from the set T ( I ), the ponits of hmin which being those first reached by the water. These points constityute the starting set of the recursion. So, of this way, Xh
min
Xh
=T
hmin
(I),
is made of the points of I which belong to the minima of lowest altitude. For a bigger
min
profundización of the theoretical aspects on this issue see [Vicent & Soille, 1991]. Definition 2.1.2.6 (Catchment basins and Watersheds by immersion): The set of the catchment obtained after the following basins of the grayscale image I is equal to the set X hmax recursion: a) X h
=T min
b) ∀ h ∈ [ h
h
min
(I), min
,h
max
 1 ], X
h +1
= min
h +1
∪I Z
(I) T h +1
. (X ) h
The watersheds transformation of I correspond to the complement of this set in DI; that is, to the set of the points of DI which do not belong to any catchment basins. The recursion process between two successive levels is illustrated in Figure 9.
Xh
Xh+1
⇒ Th+1 Fig. 9. Recursion process between Xh and Xh+1 In many practical cases, one of the principal problems is the obtaining of regional minima, due to the fact that, in general, digital images are corrupted by noise. Then, if one directly applies the watershed transformation to an image many dividing lines are obtained, which is known as oversegmentation. Therefore, the first step consists in marking the objects we want to segment, and also the background. By marker of an object, it means a connected component of pixels located inside the object to be extracted. To each object one and only one marker must correspond.
398
Image Segmentation
The design of robust marker detection techniques involves the use of knowledge specific to the series of images under study. A marking function is then constructed, whose different catchment basins correspond to the desired objects. When one works in the other way, then the watershed transformation produces oversegmentation. The oversegmentation mainly comes from the fact that the markers are not perfectly appropriate to the objects to be contoured. In short, the quality of the segmentation is directly linked to the marking function. If this condition is attained, with the watershed method continuous edges are obtained. With the classical methods this is practically impossible. This statement will be verified in the section corresponding to the experimental results. The proposed strategy in this work facilitates to obtain a marking function which is of great utility for the watershed segmentation. On one hand, the oversegmentation is avoided and on the other hand, a correct delimitation of contours of the objects of interest is achieved. 2.2 The mean shift and analysis The iterative procedure to compute the mean shift is introduced as a normalized density estimate of the gradient. By employing a differentiable kernel, an estimate of the density can be defined as the gradient of the kernel density estimate; that is,
n x −xi ⎞ ˆ f ( x ) = ∇fˆ ( x ) = 1 ∑ ∇ ∇K ⎛⎜ d i =1 h ⎟⎠ ⎝ nh
(13)
The function of the kernel K (x) is now a function defined by the vector x, ddimensional, which satisfies
∫Rd K( x )dx = 1
(14)
Conditions on the kernel K(x) and the window radius h are derived in [Fukunaga & Hostetler, 1975] to guarantee asymptotic unbiasedness, meansquare consistency, and uniform consistency of the estimate in the expression (13). For example, for Epanechnikov kernel: ⎧⎪1 / 2 −1 ( d + 2 ) (1  x cd
K E (x) = ⎨
⎪⎩0
2
) if x < 1
otherwise
(15)
where cd is the volume of the unit ddimensional sphere, the density gradient estimate becomes
ˆ f (x) = ∇ E
⎞ nx 1 d+2 d + 2 ⎛⎜ 1 ⎟ ⋅ ( − ) = ⋅ ( − ) x x x x ∑ ∑ i i ⎟ n( h dc d ) h 2 xi ∈Sh ( x ) n ( h dcd ) h 2 ⎜⎝ nx xi ∈Sh ( x ) ⎠
(16)
where the region Sh ( x ) is a hypersphere of radius h having volume h dc d , centered at x, and containing nx data points; that is, the uniform kernel. In addition, in this case d = 3, due to that the vector x has three dimensions, two for the spatial domain and one for the range domain (levels of gray).The last factor in expression (16) is called the sample mean shift,
399
Segmentation Methods for Biomedical Images
Mh , U (x) = 1 ∑ (xi − x) = 1 ∑ xi − x nx x ∈S ( x ) nx x ∈S ( x ) i
The quantity
nx
n ( h dcd )
i
h
(17)
h
is the kernel density estimate fˆU ( x ) (the uniform kernel) computed
with the hypersphere Sh ( x ) , and thus we can write the expression (16) as:
ˆ f (x) = fˆ (x) ⋅ d + 2 M (x) ∇ E U h, U h2
(18)
ˆ f (x ) 2 ∇ E Mh , U ( x ) = h d + 2 fˆ (x)
(19)
which yields,
U
Expression (19) shows that an estimate of the normalized gradient can be obtained by computing the sample mean shift in a uniform kernel centered on x. In addition, the mean shift has the direction of the gradient of the density estimate at x when this estimate is obtained with the Epanechnikov kernel. Since the mean shift vector always points towards the direction of the maximum increase in the density, it can define a path leading to a local density maximum; that is, to a mode of the density (see Figure 10). In addition, expression (19) shows that the mean is shifted towards the region in which the majority of the points reside. Since the mean shift is proportional to the local gradient estimate, it can define a path leading to a stationary point of the estimated density, where these stationary points are the modes. Moreover, the normalized gradient in expression (19) introduces a desirable adaptive behavior, since the mean shift step is large for low density regions corresponding to valleys, and decreases as x approaches a mode. ˆ f (x) ∇ ˆ f ( x ) . Thus, the corresponding Mathematically speaking, this is justified since E >∇ E fˆU ( x ) step size for the same gradient will be greater than the nearer mode. This will allow observations far from the mode or near a local minimum to move towards the mode faster ˆ f ( x ) alone. than using ∇ E
In [Comaniciu, 2000], it was proven that the mean shift procedure obtained by successive: computation of the mean shift vector Mh (x) • translation of the window Sh (x) by Mh (x), • is guaranteed to converge. A digital image can be represented as a twodimensional array of pdimensional vectors (pixels), where p = 1 in the gray level case, p = 3 for color images, and p > 3 in the multispectral case. As was pointed out in [Comaniciu, 2000] when the location and range vectors are concatenated in the joint spatialrange domain of dimension d = p+2, their different nature has to be compensated by proper normalization of parameters hs and hr. Thus, the multivariable kernel is defined as the product of two radially symmetric kernels and the Euclidean metric allows a single bandwidth for each domain, that is:
400
Image Segmentation
f (x)
x1
x2 x3
x4
x5
x6
x7
Fig. 10. Gradient mode clustering
Kh ,h s r
(x) =
C 2
⎛
s 2⎞
x k ⎜ ⎜ hs p
hs hr
⎝
⎛ r ⎟k⎜ x ⎟ ⎜ hr ⎠ ⎝
2⎞
⎟ ⎟ ⎠
where xs is the spatial part, xr is the range part of a feature vector, k(x) the common profile used in both domains, hs and hr the employed kernel bandwidths, and C the corresponding normalization constant. The value hs is associated with spatial resolution of the analysis, while the value hr is associated with the resolution in the domain of range (levels of gray). More details related with the mean shift can be found in [Comaniciu, 2000; Comaniciu & Meer, 2002].
3. Some characteristics of the images under study Studied images are of arteries, which have atherosclerotic lesions and these were obtained from different parts of the human body, from more of 80 autopsies. These arteries were contrasted with a special tint in order to accentuate the different lesions in arteries. Later, the lesions are manually classified by the pathologists according to World Health Organization. They classified the lesions in type I, II, III and IV. For example, the lesions I and II, these are the fatty streaks and fibrous plaques respectively, while the lesions III and IV are respectively the complicated and calcified plaques. The pathologists mark only the lesions; that is, they do not outline to them. For that reason, the edge automatically detection, it is very important, because these images will be subject to an additional morphometrical analysis in order to study the atherosclerosis and its organicconsequences. The arteries were directly digitalized from the working desk by using a video camera and an A/D converter. In Figure 11(a) a typical image with lesions I and II can be seen, while in Figure 11(b) is shown its histogram. These images were captured via the MADIP system
401
Segmentation Methods for Biomedical Images
with a resolution of 512x512x8 bit/pixels. MADIP Morphometrical Analysis by Digital Image Processing is a system created to help physicians in his research work using images. MADIP is a software for the morphometrical characterization of macro and micro lesions in biomedical images. In general, it may be used in any branch of science and technology, where not only quantitative techniques of high precision are required, but it is also necessary to do a big number of measurements with high accuracy and rapidly [Rodríguez et al., 2001]. Image histogram
Type I
Range
F R E Q U E N C Y
Type II
(a)
(b)
Fig. 11. (a) Atherosclerosis image. (b) Histogram. It can be observed low contrast. The arrows show an example of structures that we attempt to segment. Figure 12 shows other examples with other types of lesions.
Lesion IV Lesion II
Placa de Ateroma Lesion III de tipo III
Fig. 12. Atherosclerosis images with different lesions, which are marked with arrows. The arrows show the lesions. There are several remarkable characteristics of these images, which are common to typical images that we encounter in the atherosclerotic lesions: 1. High local variation of intensity is observed both, within the atherosclerotic lesions and the background. However, the local variation of intensities is higher within the lesions than in background regions.
402
Image Segmentation
2. 3.
The histogram of Fig. 11(b) shows that there is a low contrast in the images. The lesions III and IV have better contrast than the lesions I and II (see Fig. 12). In addition, due to variations in the intensity of the background across the image and the low contrast between lesions and background intensities, principally for the lesions I and II, the atherosclerotic lesions in a region of the image may appear lighter than the background in a distant region. 4. It is common of these images the diversity in shape and size of the atherosclerotic lesions. 5. The boundaries of the atherosclerotic lesions, principally for the lesions I and II, may be extremely difficult to define. Due to variations in intensity, both within the lesions and in the background, portions of the atherosclerotic lesion may appear blended into the background, without creating a distinct boundary. While the characteristics presented above testify the difficulty in identifying atherosclerotic lesions, a close examination reveals information that can be used. We observed that two features of the image, local variation of intensity and image intensity level, can be used to identify regions of the image that describe lesions. High local variation of intensity is exhibited by regions within and near the boundaries of lesions. Thus, local variation of intensity can roughly identify regions of the image that contain lesions. Across the entire image, changes in intensity level cannot reliably distinguish atherosclerotic lesions, due to possible nonuniformity of the average background intensity and low contrast between lesions and background; principally, in the lesions I and II. However, within a region of interest, changes in intensity level can effectively distinguish a lesion, since locally a lesion has major variation of intensities than its surrounding background. The exact shape and size of this region are not important, and hence the region is referred to as an approximate region. Other details on this study can be found in [Rodríguez & Pacheco, 2007]. For the study with images of blood vessels were used biopsies, which represent an angiogenesis process in malignant tumors. These were included in paraffin by using the inmunohistoquimic technique with the complex method of avidina biotina. Finally, monoclonal CD34 was contrasted with green methyl to accentuate formation of new blood vessels. These biopsies were obtained from soft parts of human bodies. This analysis was carried out for more than 100 patients. In Figure 13 can be seen typical images, which were captured via MADIP system with a resolution of 512x512x8 bit/pixels.
Blood vessels
Blood vessel
Blood vessel
Fig. 13. These images represent the angiogenesis process. The blood vessels are marked with arrows In Figure 14 a horizontal profile can be observed through the centre of a vessel; that is, a plot of the pixel intensities along a single row. In Figure 15 (b), the histogram of this image is represented.
403
Segmentation Methods for Biomedical Images
Profile
Background Blood vessel
New profile
Fig. 14. An intensity profile through the centre of a vessel. Profile is indicated by a line
Blood vessels
(a)
(b)
Fig. 15. (a ) Original image. (b) Histogram image There are several notable characteristics of this image, which are common to typical images that we encounter in the tissues of biopsies: 1. The intensity is slightly darker within the blood vessel than in the local surrounding background. It is emphasized that this observation holds only within the local surroundings. 2. High local variation of intensity is observed both within the blood vessel and the background. However, the local variation of intensity is higher within the blood vessel than in background regions (see Fig. 14) 3. The variability of blood vessels both, in size and shape, it can be observed (see Fig. 13). 4. The contrast of intensities between the blood vessel (BV) and the background is low. This is evident from the intensity histogram (see Fig. 15 (b)). This histogram is unimodal, reflecting the low separation in average intensity of the BV and background.
404
Image Segmentation
Due to acquisition protocol, the images are corrupted with a lot of noise. It can be observed the ripple of the histogram in Figure 15(b). Other details on this research can be found in [Rodríguez & Pacheco, 2005].
4. Experimental results and discussion One of the most important diseases to study is the atherosclerosis and its organicconsequences, which is one of the principal causes of death in the world [FernándezBritto et al., 1988; FernándezBritto & Carlevaro, 1989]. The atherosclerosis produces as final consequence the loss of elasticity and increase of the wall of arteries. For example, heart attack, cerebral attack and ischemia are some of its principal consequences [Cotran, 2002]. Strategies that can isolate atherosclerotic lesions in an automatic way constitute important tools for pathologists. With the goal of diminishing the noise existing in the atherosclerosis images, we used the Gauss filter according to the parameters described in [Rodríguez & Pacheco, 2007]. Figure 16 shows two examples.
(a)
(b)
(c)
(d)
Fig. 16. (a) and (c) Original images, (b) and (d) Filtered images with Gauss (σ = 3, 5x5 window size). One can observe in Figure 16 (b, d) that the images are smoother. It is possible to see that the atherosclerotic lesions are more uniform, which is indicative that the noise was diminished. In a second step, we carried out a modification of the histogram to these images, according to the algorithm presented in [Rodríguez & Pacheco, 2007]. This procedure was carried out in order to increase the contrast. We show in Figure 17 the obtained results.
(a)
(b)
(c)
(d)
Fig. 17. (a) and (c) Filtered images with Gauss. Figures (b) and (d) contrasted images. It is evident the good obtained result with the enhancement.
405
Segmentation Methods for Biomedical Images
We shall carry out a general description of the proposed method before going into the details of each one of the steps, which will allow having a panoramic vision of the proposed solution. In Figure 18 is depicted all steps of the proposed strategy to obtain robust markers for the atherosclerotic lesions.
Reconstruction by dilation (Lesions I and II)
Filtered image by Gauss
Reconstruction by erosion (Lesions III and IV)
Contrast enhancement
Extraction of region of interest
Obtaining of marks
Fig. 18. Steps to obtain the markers of atherosclerotic lesions It is very important to point out that the proposed strategy was obtained according to experimentation; that is, we carried out firstly a morphological reconstruction by erosion for all lesions (see definition 2.1.4), and secondly, we carried out a morphological reconstruction by dilation for all lesions (see definition 2.1.2). We verified that, in all cases for the lesions I and II, the best results by using a reconstruction by dilation were obtained; while for the lesions III and IV the best results were obtained carrying out a reconstruction by erosion. This affirmation will be appreciated in the next steps. We processed 100 images in total, but for space problem, we only reported 10 cases. In order to cope with problems previously cited and with the goal of extracting the approximate regions of interest; after the histogram modification, we carried out a morphological reconstruction. The region of interest is a region that contains the lesion and its neighboring background, where the exact shape and size of this region are not important. This region of interest is referred to an approximate region. We verified that the reconstruction by erosion (for the lesions III and IV) led to an image where the dark zones correspond to these lesions. For example, in Figure 19 is shown the obtained result for a lesion of type IV.
(a)
(b)
Fig. 19. (a) Resulting image of the histogram modification. (b) Image obtained by a reconstruction by erosion. The dark parts correspond to the lesion IV.
406
Image Segmentation
The result in Figure 19(b) was obtained by using a rhombus as structuring element of 5x5 pixels and a height equal to 60 (see definition 2.1.1.3). Basically, we used the hdomes, where the principle is to subtract an arbitrary constant h from the original image I and to perform a grayscale geodesic reconstruction of I from I+h (J = I+h). This provides a useful technique for extracting “domes” of a given height h. The choice of h turns out not to be a critical operation, since a range of values yield correct results, which we verified in the practice. In addition, the selection of this structuring element and its size was obtained via experimentation. It is important to point out that when one speaks of size of the structuring element its dimension is in pixels. In addition, the structuring element is plane. For an understanding better on this aspect, as an example, the form and size of four structuring elements are shown (see Figure 20). In Figure 21, it is represented the obtained results (in the reconstruction) for other structuring elements. In all cases in order to carry out the geodesic reconstruction, the considered height was equal to 60. As can be appreciated in Figure 21(a), for a structuring element minor than 5x5 pixels, the area of the atherosclerotic lesions decreased. Comparing the image of Figure 19 (b) with those of Figures 21(b) and 21(c), one can see that for structuring elements (rhombus or circle) major than 5x5 pixels, the obtained results were very similar, but the computation time was increased. In Figures 21(d), 21(e) and 21(f), it can be observed that for diagonal segments as structuring elements, the obtained results were not good. The lesions were notably deformed. For these reasons, we considered that the rhombus of 5x5 pixels was the most suitable. This behaviour was similar for the 100 processed images.
1
1
1
1
1
1
1
1
1
1
1
1
1 1
1 1
1
1 0
Rhombus 5x5
Segment of line 45
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
Circle 5x5
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
Hexagon 5x5
Fig. 20. Four structuring elements; superior left: rhombus of size 5x5 pixels, superior right: diagonal segment of 5 pixels, inferior left: circle of 5x5 pixels and inferior right: hexagon of 5x5 pixels.
407
Segmentation Methods for Biomedical Images
With respect to the height for carrying out the geodesic reconstruction, we verified that for all our images the optimal values were in the range from 40 to 60. In fact, in Figure 22 are depicted the obtained results considering a height out of this range. In Figures 22 (b) and 22 (c), one can observe that for a large height the atherosclerotic lesions were very smoothed. Furthermore, the areas of the lesions also were increased. Some of them were fused (see arrows). For a height smaller than 40, according to the criterion of pathologists, the area of the atherosclerotic lesions decreased (see Fig. 22(a)). Then, in these cases, an exact delimitation of the lesions is not obtained and the final results will be poor. This behavior was similar for all the processed images, which is not included here by space problem.
(a)
(d)
(b)
(e)
(c)
(f)
Fig. 21. (a) Reconstruction by a rhombus of 3x3 pixels. (b) Reconstruction by a rhombus of 7x7 pixels. (c) Reconstruction by a circle of 7x7 pixels. (d) Reconstruction by a diagonal segment (450) of 3 pixels. (e) Reconstruction by a diagonal segment of 5 pixels (450). (f) Reconstruction by a diagonal segment of 7 pixels (450). The height was equal to 60. After obtaining both, the size of structuring element and the optimal height, the next step of our strategy was to segment the approximate region of interest. That is, as it was pointed out, a region that contains the atherosclerotic lesion and its neighbouring background. This step was carried out by applying a simple threshold through Otsu method [Otsu, 1978]. The thresholding value does not have much influence on the performance, because the exact shape and size of this region are not important. In this case, the region is referred to as an approximate region. In Figure 23(b) one can see the region of interest.
408
Image Segmentation
Fusion of regions
Fusion of regions
(c)
(b)
(a)
Fig. 22. (a) Reconstruction with a rhombus of 5x5 and height equal to 30. (b) Reconstruction with a rhombus of 5x5 and height equal to 80. (c) Reconstruction with a rhombus of 5x5 and height equal to 120.
Lesions
(a)
(b)
Fig. 23. (a) Image of the reconstruction, where the arrows indicate the lesions. (b) Regions of interest In Figure 23 (b), it is possible to observe that all the indicated lesions in Figure 23 (a) were detected. Furthermore, it can be seen that the regions of interest don't have the exact size of the lesions, since they simply are an approximation. We verified that the regions of interest, in general, were smaller than the atherosclerotic lesions. This result was the same one for all the processed images. Other regions of interest will be able to be observed in the next section. After this result, we developed an algorithm to obtain markers for the atherosclerotic lesion. 4.1 Algorithm to obtain markers The steps of the algorithm are described below: 1. Obtain the regions of interest, which as we pointed out, it are those regions belonging to the lesions and they are obtained through a thresholding procedure, after carrying out the
Segmentation Methods for Biomedical Images
409
geodesic reconstruction. To the resulting image of this procedure we will call IREZI. 2. Label the resulting image of the step 1. This step is as follows. Create an auxiliary image with all pixels in zero: Let IA1 be this image. Scan IREZI at iterative way, then in this image all the background is labeled with a value equal to 1. 3. With the goal of finding connected components, scan IREZI again from the top to the bottom and from the left to the right. If there is a pixel, which belongs to a connected component and in the image IA1 this pixel has zero value, then other iterative method begins to work. This new iterative method marks with a determined value within the image IA1 all pixels belonging to a connected component. In addition, pixels within the image IREZI are also marked with a value, which identifies the connected component to which they belong. This action is carried out in the whole image. As this step is finished, in the image IREZI all the connected components were filled and in the image IA1 all the connected components were labeled. 4. Create other auxiliary image (let IA2 be this image) with the same values of the image IA1. Also create an array, which controls if a connected component was reduced. In the image IA2 is where in each step the reduction of the connected components are obtained, the final result is represented in the image IA1. 5. Scan the labeled image (IA1). When in this image a pixel is found, which belongs to a connected component, through other iterative method, this component is reduced and in the image IA2 all the frontiers of the connected component are marked. If still there is some pixel within the connected component, which is no frontier, in the images IA2 and IA1, the mentioned pixel is eliminated and this function begins again until that all points are frontiers. In this case, the obtained result (reduction) is taken as the mark. In the array (see step 4) it is indicated that the labeled component with this value was processed and it is began to look for other component. 6. Finish when the image IA1 is completely scanned. When this step is concluded, in the image IA1 all marks of the atherosclerotic lesions are. These marks are collocated in the image IREZI. Here, after the step two, the connected components (in IREZI) were filled. The image IREZI is the resulting image. All the functions in C of this algorithm appear in [Rodríguez & Pacheco, 2007]. The result of carrying out this algorithm to the image of Figure 23(b) is shown in Figure 24. In Figure 24(b) one can see that the mark is unique for each of the lesions, which is always inside the atherosclerotic lesion. In this case, we firstly applied this algorithm to the regions of interest of the lesions III and IV. In other words, to the contrasted images of this type of lesions (III and IV) a reconstruction by erosion is carried out, later we applied a thresholding in order to obtain the regions of interest; lastly the proposed algorithm is applied to the image of the regions of interest (see Fig. 24 (b)). This procedure is the same one for all the lesions III and IV. In total 40 lesions of type III and IV, it were processed. Now, we shall explain the steps that we carried out to obtain the marks for the lesions I and II (see Fig. 18). In this case, the first step was to carry out a modification of histogram according to the algorithm that appears in [Rodríguez & Pacheco, 2007]. After this step, we carried out a reconstruction by dilation. This reconstruction improved, even more, the contrast of the lesions I and II. Figure 25 shows the obtained result of the reconstruction.
410
Image Segmentation
M Connected
components Component
(a)
Marker
Markers
(c)
(b)
Fig. 24. (a). Image with regions of interest, (b) Marking image, (c) Marks superimposed on the original image
(a)
(b)
Fig. 25. (a) Initial image. (b) Reconstruction by dilation (lesion II) The result in Figure 25(b) was obtained by using a structuring element type rhombus of 5x5 pixels, similar to the one used for the lesions III and IV. In this case, the used height in order to carry out a geodesic reconstruction by dilatation was equal to 40. Also, for these lesions (I and II), several experiments with distinct structuring elements were carried out. The obtained results were very similar to those depicted in Figures 21 and 22. We verified that the structuring element type rhombus of 5x5 pixels was of the best performance. In addition, with respect to height for the reconstruction, we concluded that the optimal height was in the range from 40 to 60 too. Out of this range, similar results were obtained as in the lesions III and IV. This experiment was carried out for a total of 50 images (lesions I and II). All the obtained results were not shown here because they were very similar. Later, we obtained the approximate region of interest via a thresholding and lastly, the markers were obtained through the algorithm presented in section 4.1. The obtained result is shown in Figure 26. One can observe that the regions of interest are not exactly the atherosclerotic lesions, but simply an approximate region.
411
Segmentation Methods for Biomedical Images
(a)
(b)
(c)
Fig. 26. (a) Regions of interest. The arrows indicate the connected components. (b) Image with marks. (c) The marks superimposed on the original image. In Figure 26(b) it is possible to see that the marks are unique for each of the lesions, which are always inside the atherosclerotic lesion. This aspect is very important. 4.2 Application of the proposed strategy for atherosclerosis image segmentation by using the watershed method As have been pointed out the watershed transformation has the drawback of producing oversegmentation. This is true even if one had taken the precaution of filtering the initial image or its gradient. In fact, Figure 27(b) shows the obtained result as we directly applied the watershed transformation to an atherosclerosis image without good markers for the lesions. It is evident as the contours of the atherosclerotic lesions were not well detected and it is observed a lot of noise. However, in Figure 27(c) is shown the excellent obtained result according to our strategy and the introduced algorithm in this work (see section 4.1). The contours of the atherosclerotic lesions were well defined.
(a)
(b)
(c)
Fig. 27. (a) Original image. (b) The watershed segmentation without marks in the lesions. (c) The watershed segmentation according to our strategy. In Figure 28, we show the contours superimposed on the original image in order to see the exact coincidence of the obtained contours. This result is evident.
412
Image Segmentation
Fig. 28. The contours superimposed on the original image. It is possible to see the exact coincidence of the contours on the lesions. The arrow indicates an object, which does not correspond to an atherosclerotic lesion. In [Rodríguez & Pacheco, 2007], a quantitative evaluation of this algorithm was carried out. In Figure 29, another example of the application of watershed method on other atherosclerosis image is shown. In Figure 29 (b), we applied watershed segmentation on the gradient of image. It is evident that, even though the watershed method was applied on the gradient of image, an oversegmentation was produced. In Figure 29 (c), one can observe the obtained result of carrying out our strategy. In Figure 30 is shown the obtained contours with our strategy superimposed on the original image. It is possible to observe the exact coincidence of the contours on the lesions.
(a)
(b)
(c)
Fig. 29. (a) Original image. (b) Watershed transformation on the gradient of the original image (c) The watershed segmentation according to our strategy. This same strategy was applied for image segmentation of blood vessels [Rodríguez et al., 2005], but here we will expose the obtained results on these images (blood vessels) by using the mean shift.
Segmentation Methods for Biomedical Images
413
Fig. 30. The contours superimposed on the original image. The arrow indicates an object, which does not correspond to an atherosclerotic lesion (see reference [Rodríguez & Pacheco, 2007]) 4.3 Obtained results in image segmentation of blood vessels by using the mean shift The angiogenesis process is an important aspect in the histopathological research of diseases [DíazFlores et al., 1994]. This process is studied by pathological anatomy and it is characterised by a new formation of blood vessels in the tissue, which increases its growing. Angiogenesis is a normal process that occurs in all tissues, but it is known that in pathological cases the increase of blood vessels is abnormal. For example, in mammary, prostate, bladder and brain cancers the greater the number of blood vessels, the more developed the disease is [DíazFlores et al., 1994]. In the study of the angiogenesis process the pathologists analyse all information related with the blood vessels by using a microscope. This work is very tedious and time consuming and obviously, the automation of this analysis is highly desirable. In such a sense, a useful task for digital images processing should be the segmentation of blood vessels. In this work we utilized the mean shift for segmentation of blood vessels. So, we proposed a segmentation algorithm, which will be exposed next. In general, an image captured with a real physical device is contaminated with noise and in most cases a statistical model of white noise is assumed, mean zero and variance σ. For smoothing or elimination of this form of noise many types of filters have been published, the most classic being the low pass filter. This filter indiscriminately replaces the central pixel in a window by the average or the weighted average of pixels contained therein. The end result with this filtering is a blurred image; since this reduces the noise but also important information is taken away from the edges. However, there are low pass filtering techniques that preserve the discontinuities and reduce abrupt changes near local structures. A diverse number of approaches have been published taking into consideration the use of adaptive filtering. These range from an adaptive Wiener filter, local isotropic smoothing, to an anisotropic filtering. The mean shift works in the spatialrange domain, but differs from it (anisotropic filtering) in the use of local information. The algorithm that was proposed in [Comaniciu & Meer, 2002] for filtering through mean shift is as follows:
414
Image Segmentation
{ }
Let xi and i
{zi }i ,
i = 1,… , n be the input and filtered images in the joint spatialrange
domain. For each pixel p ∈ xi , p = ( x , y , z ) ∈ ℜ3 , where
( x , y ) ∈ ℜ2
and z ∈ ⎡0, 2 B − 1⎤ , B ⎣ ⎦ being the quantity of bits/pixel in the image. The filtering algorithm comprises the following steps: For each i = 1,… , n 1. Initialize j =1 and yi,1 = pi. 2. Compute the mean shift in order to obtain the mode where the pixel converges; that is, the calculation of the mean shift is carried out until convergence, y = yi,c. 3. Store at Zi the component of the gray level of calculated
(
)
is value: Zi = xis , yir, c , where x s is the spatial component and y r i i, c the range component. 4.3.1 Proposed segmentation algorithm via mean shift 1. Run the mean shift filtering procedure for the image according to former steps, and store all the information about the ddimensional convergence point in Zi. 2. Define the regions, which are found in the spatial domain (hs) with intensities smaller or equal than hr/2. 3. For each region, which was defined in the step 2, look for all the pixels belonging to the region whose mean intensity values are assigned at Zi. 4. Build the region graph through an adjacent list as follows: for each region, look for all adjacent regions to the right or under the region. 5. While there are nodes in the graph, which have been not visited, run a variant of the “DepthFirst Search” (DFS), which allows concatenating adjacent regions; as parameter one has to provide a node which has been not yet visited. To do this the next steps are carried out: 5.1.Mark the mentioned node as visited. 5.2.While there is children not yet visited: children = current child if Zparent − Zchild< = hr, then, The region of the child is fused with the region of the father, where the same label is assigned and it is marked as visited. 5.3 To the parent is assigned the children of the visited child from the first position and the children which have been not visited are remained. 5.4 Look for the following child and come back to step 2 while stability it is not obtained (i.e., no more pixel values are modified). 6. Eliminate spatial regions containing less than M pixels, since those regions are considered irrelevant. 7. (Binary): To all the pixels belonging to background, assign the white color to the objects and the black color to the background.
The effectiveness of this algorithm was recently proven in [Rodríguez et al., 2008]. In that paper was carried out a comparison of the obtained results with this algorithm and those attained by using the spectral methods.
415
Segmentation Methods for Biomedical Images
Now, some of the obtained results will be exposed, in real images, using the proposed segmentation algorithm of the mean shift. In Figure 31 are presented a first example where one can appreciate the segmentation result.
(a)
(b)
(c)
Fig. 31. (a) Original Image, (b) Image segmented with the proposed algorithm (hs=4, hr=15, M =20), (c) Binarized image. In Figures 32 and 33, other two segmentation examples appear using the proposed algorithm.
(a)
(b)
(c)
Fig. 32. (a) Original Image, (b) Image segmented with the proposed algorithm (hs=4, hr=15, M =20), (c) Binarized image.
(a)
(b)
(c)
Fig. 33. (a) Original Image, (b) Image segmented with the proposed algorithm (hs=4, hr=15, M =20), (c) Binarized image.
416
Image Segmentation
One can see in Figures 31, 32 and 33 the robustness of the obtained results using the proposed segmentation algorithm. According to the opinion of specialists the method was able to discriminate all the objects of interest (blood vessels). This is of supreme importance from the point of view of the diagnosis. It is also observed that the obtained results are free of noise; in addition, for all the segmented images the same parameters hr and hs were used. By the way of examples, two of the obtained results in [Rodriguez et al., 2008] it will be exposed. These results are shown in Figure 34 and 35. In the presented examples, the reader can appreciate that both methods were able to effectively well isolate the blood vessels. However, one also can see that the segmented images via the mean shift algorithm were less noisy than those obtained via the spectral method. Furthermore, the edges were more robust and better defined in the images segmented with the mean shift. In order to obtain less noisy images with the spectral method it would be necessary to carry out an additional step of filtering with the aim of eliminating the noise that arises during the segmentation process. This procedure could be carried out through a morphological technique, which can be an opening or a majority filter. Another way could be to use the mean shift procedure as a preprocessing in order to improve performance of the spectral method. In [Rodriguez et al., 2008], it also can be seen a quantitative method of evaluation of these strategies.
(a)
(b)
(c)
Fig. 34. (a) Original image, (b) Segmentation via the mean shift, (c) Segmentation via the spectral method
(a)
(b)
(c)
Fig. 35. (a) Original image, (b) Segmentation via the mean shift, (c) Segmentation via the spectral method
Segmentation Methods for Biomedical Images
417
5. Conclusions In this chapter was carried out a brief theoretical introduction of the Watershed transformation. Through the experimentation suitable markers were obtained for the atherosclerotic lesions (including to the background). In such a sense, a new algorithm was introduced, which was efficient in order to mark the atherosclerotic lesions and to eliminate all other spurious information. With the proposed strategy the application of the watershed segmentation gave good results, where the exact contours of the objects of interest were obtained (atherosclerotic lesions). Furthermore, oversegmentation never was obtained. A theoretical introduction of the mean shift was carried out too. In such a sense, a new algorithm was proposed for image segmentation of blood vessels. The same was suitable for this application. In order to demonstrate its good performance, the same one was compared with another segmentation algorithm already established (the spectral method). Through several experiments with real images, we proven that the segmented images by using the mean shift were less noisy than those attained by means of the spectral method. In addition, the obtained edges were better defined by using the mean shift than those attained by the spectral method. The strategies introduced in this chapter can be extended to other types of biomedical images and their application is valid for other tasks of image analysis where suitable methods of segmentation are required.
6. References Cheng, Y. (1995). Mean Shift, Mode Seeking, and Clustering, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 17, No. 8, (August 1995), pp.(790799), ISSN: 01628828 ChinHsing, C.; Lee, J.; Wang, J. & Mao, C. W. (1998). Colour image segmentation for bladder cancer diagnosis, Mathematical and Computer Modelling, Vol. 27, No. 2, (January 1998), pp (103120), ISSN: 08957177 Comaniciu, D. I. (2000). Nonparametric Robust Method for Computer Vision, Ph.D. thesis, New Brunswick, Rutgers, The State University of New Jersey, January, 2000. Comaniciu, D. & Meer, P. (2002). Mean Shift: A Robust Approach toward Feature Space Analysis, IEEE Transaction on Pattern Analysis and Machine Intelligence, Vol. 24, No. 5, (May 2002), ISSN: 01628828 Cotran, R. (2002). Patología Estructural y Funcional, Mc. Grow Hill, México, ISBN: 9701027876. DíazFlores, L.; Guitiérrez, R. & Varela, H. (1994). Angiogenesis. An update, Histoogy and Histopathology, Vol. 9, (July 1994), pp (807853), ISSN 02133911 Fukunaga, K. & Hostetler, L. D. (1975). The Estimation of the Gradient of a Density Function, IEEE Transactions on Information Theory, Vol. IT21, No. 1, (January 1975), pp. (3240), ISSN: 00189448 FernándezBritto, J. E.; Carlevaro, P. V.; Bacallao, J.; Koch, A.S.; Guski, H. & Campos, R. (1988): "Coronary atherosclerotic lesion: its study applying an atherometric system using principal components analysis". Z. Klin. Med. 44: 291294, 1988. FernándezBritto, J. E. & Carlevaro, P.V. (1989). Atherometric System: Morphometric standardized methodology to study atherosclerosis and its consequences, Gegenbaurs morphologisches Jahrbuch, Vol. 135, (November 1989), pp (112), ISSN: 00015180 Fuh, CS.; Maragos, P. & Vincent, L. (1991). Region based approaches to visual motion correspondence, Technical Report 9118, (November 1991), pp (129), Harvard
418
Image Segmentation
Robotics Laboratory, Harvard University, Division of Applied Sciences, Pierce Hall, Cambridge NA 02138 Grenier, T.; RevolMuller, C.; Davignon, F. & Gimenez, G. (2006). Hybrid Approach for Multiparametric Mean Shift Filtering, Image Processing, IEEE, International Conference, Atlanta, GA, 811, (Oct. 2006), pp (15411544). Kenong, W.; Gauthier, D. & Levine, M. D. (1995). Live Cell Image Segmentation, IEEE Transactions on Biomedical Engineering, Vol.42, No. 1, (January 1995), pp (111), ISSN: 00189294 Koss, J.; Newman, F.; Johnson, D. & Kirch, D. (1999). Abdominal organ segmentation using texture transforms and a hopfield neural network, IEEE Transactions Medical Imaging, Vol. 18, No. 7, (July, 1999), ISSN: 02780062 Najman, L. & Schmitt, M. (1996). Geodesic Saliency of Watershed Countours and Hierarchical Segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 18, No. 12, (December 1996), pp (11631173), ISSN: 01628828 Otsu, N. (1978). A threshold selection method from grey level histogram, IEEE Transactions on Systems, Man and Cybernetics, Vol. SMC8, No. 1, (January 1978), pp (6266), ISSN: 00189472 Rodríguez, R.; Alarcón, T. & Sánchez, L. (2001). MADIP: Morphometrical Analysis by Digital Image Processing, Proceedings of the IX Spanish Symposium on Pattern Recognition and Image Analysis, Vol. I, pp (291298), ISBN 8480213493, Spain. Rodríguez, R.; Alarcón, T.; Wong, R. & Sanchez, L. (2002). Color segmentation applied to study of the angiogenesis. Part I, Journal of Intelligent and Robotic System, Vol. 34, No. 1, (May 2002), pp (8397), ISSN: 09210296 Rodríguez, R.; Alarcón, T. & Pacheco, O. (2005). A New Strategy to Obtain Robust Markers for Blood Vessels Segmentation by using the Watersheds Method, Journal of Computers in Biology and Medicine, Vol. 35, No. 8, (October 2005), pp (665686), ISSN: 00104825 Rodríguez, R. & Pacheco, O. (2007). A Strategy for Atherosclerosis Image Segmentation by using Robust Markers, Journal Intelligent & Robotic System, Vol. 50, No. 2, (October 2007), pp (121 – 140), ISSN: 09210296. Rodríguez, R.; Castillo, P. J.; Guerra, V., Sossa, J. H.; Suárez A. G. & Izquierdo, E. (2008). A comparison between two robust techniques for segmentation of blood vessels, Journal of Computers in Biology and Medicine, Vol. 38, No. Issue 8, (June 2008). pp (931 – 940), ISSN: 00104825 Sijbers, J.; Scheunders, P.; Verhoye, M.; Van der Linden, A.; Van Dyck, D. & Raman, E. (1997). Watershedbased segmentation of 3D MR data for volume quatization, Magnetic Resonance Imaging, Vol. 15, No. 6, (February 1997), pp (679688), ISSN: 0730725X Schmid, P. (1999). Segmentation of digitized dermatoscopic images by twodimensional color clustering, IEEE Transactions Medical Imaging, Vol. 18, No. 2, February 1999), pp (164171), ISSN: 02780062 Vicent, L. & Soille, P. (1991). Watersheds in digital spaces: An efficient algorithm based on immersion simulations, IEEE Transactins on Pattern Analysis and Machine Intelligence, Vol. 13, No. 6, (June 1991), pp (583593), ISSN: 01628828 Vincent, L. (1993). Morphological grayscale reconstruction in Image Analysis: Applications and Efficient Algorithms, IEEE Transactions on Image Processing, Vol.2, No. 2, (April, 1993), pp (176201), ISSN: 10577149 Weidner N.; Carroll, P. R.; Flax, J.; Blumenfeld, W. & Folkman, J. (1993). Tumour angiogenesis correlates with metastasis in invasive prostate carcinoma, American Journal of Pathology, Vol. 143, No. 2, (October 1993). ISSN: 00029440
22 Algorithm Selection Based on a Region Similarity Metric for Intracellular Image Segmentation Satoko Takemoto and Hideo Yokota
BioResearch Infrastructure Construction Team, RIKEN Japan 1. Introduction Livecell imaging using fluorescence microscopy has become popular in modern biology to analyze complex cellular events such as the dynamics of substances inside cells (Eils & Athale, 2003; Bhaskar & Singh, 2007). The next step in furthering this type of analysis is accumulating useful information from the observed images to quantify the dynamics (Cong & Parvin, 2000; Goldman & Spector, 2004; Harder et al., 2008, Waltera et al., 2010). However, quantification of intracellular images is a difficult process because microscopic images with ultrahigh sensitivity have a low signaltonoise ratio. In addition, the amount of data required for quantification has gradually increased as microscopy has developed. These obstacles make it more difficult for cell biologists to identify regions of interest and accumulate various types of quantitative information, such as the volume, shape, and dynamics of intracellular substances. Hence, it is important to develop computational methods for identifying objective targets, such as organelles labeled with, for example, a fluorescent protein. Image segmentation, the process by which an image is divided into multiple regions corresponding to the components pictured in the image, plays a key role as one of the first steps in the quantification of objective targets from observed images. The use of segmented regions allows us to distinguish substances of interest from irrelevant regions, including background and noise. Numerous segmentation algorithms have been proposed (e.g., Haralick & Shapiro, 1985; Pal & Pal, 1993), but most approaches have been developed for a specific task and cannot be generalized for other segmentation tasks. As a result, researchers have had to face the difficult duty of choosing the most suitable algorithm for a given task while facing increasing numbers of images needing quantification. Moreover, recent notable improvements in livecell imaging require that segmentation algorithms be flexible enough to accommodate timevariable changes in targets. No single algorithm performed with a fixedparameter setting is considered to be sufficient for analyzing all timelapse images, and optimizing algorithms for a variety of images is a tedious task for researchers. Solutions to these problems have been proposed based on the idea of algorithm selection (e.g., Cardoso & CorteReal, 2005; Zhang, 2006; Polak et al., 2009). An appropriate algorithm with an optimized parameter setting for each task is automatically selected according to unique evaluation metrics of algorithm performance. Evaluation can be roughly divided into two types: unsupervised evaluation and supervised evaluation. The former type can
420
Image Segmentation
evaluate different algorithms only by simply computing some chosen evaluation metrics without requiring a prior knowledge about segmentation targets (Cardoso, 2005; Zhang et al., 2006). Statistical features, such as the greylevel standard deviation or homogeneity of pixel intensities in the segmented region, are generally computed. For example, a region contrast (Levine & Nazif, 1985) or region shape (Sahoo et al., 1988) have been proposed (see the comprehensive survey; Zhang et al., 2008). Although the advantage of unsupervised evaluation is that a large number of segmentation algorithms can be evaluated, if there is no guarantee that the predefined range for some statistical features will be satisfied, unsupervised evaluation should not be used. In addition, the range of features of intracellular substances cannot be predefined, and the diversity in features of intracellular substances may destabilize the result of evaluation. The latter type can evaluate different algorithms by using some metrics based on similarity (or error) measurement between two regions: an automatically segmented region and a manually segmented region, called the reference region or the groundtruth (e.g., Zhang & Gerbrands, 1992; Martin et al., 2001; Jiang et al., 2006; Polak et al., 2009). For example, the number of missegmented pixels (Ysnoff et al., 1977), or the number of segmented targets (Zhang, 1996) is commonly used as an error measurement. Although metrics for supervised evaluation have been proposed so as to reflect the human perception, it is not clear whether the evaluation procedure has actually reflected the perception. That is, the region, which is segmented by using the selected algorithm, may not identify the objective targets to be quantified. In this research, we propose a novel evaluation metric composed of similarity measurements of a combination of intensitybased and shapebased image features between a segmented region and the groundtruth. Our evaluation metric adopts the philosophy of supervised evaluation and expands it so as to reflect the human perception. We chose these two kinds of image features because cell biologists usually pay attention to them when identifying objective targets, and our proposed method is able to select an appropriate algorithm with optimal parameter settings so as to satisfy biologists' intentions. The proposed method evaluates the performance of segmentation algorithms by comparing each segmentation result with the groundtruth specified by cell biologists, and it predicts which algorithm will provide the best performance on new images that have similar image features to the original groundtruth. We investigated the performance of two types of segmentation algorithms under our proposed evaluation metric for the identification of fluorescent labeled targets with granular shapes on real intracellular images. In addition to demonstrating the automatic selection of an appropriate algorithm suited to the segmentation task, we showed that our evaluation metric can rank different types of algorithms. We also tested to see whether the selected algorithm showed good segmentation results for other similar images. The rest of the paper is structured as follows. In Section 2, we describe the algorithm selection framework and explain our proposed evaluation metric based on the region similarity. Experimental results and discussion including segmentation quality for intracellular images taken by a confocal microscope are presented in Section 3. Finally, a conclusion is offered in Section 4.
2. Algorithm selection framework Many possible solutions must be considered when establishing a segmentation algorithm for a specific application that satisfies a user's intention. In many cases, a target intracellular
Algorithm Selection Based on a Region Similarity Metric for Intracellular Image Segmentation
421
substance can be represented by a homogeneous unique image feature and can be distinguished from other substances, even from background. Here, we focus on the segmentation techniques implemented by a pattern classification technique (Duda et al., 2007) that can classify image features into classes (or categories) associated with substances. When performing segmentation, the computer first calculates Ndimensional image features that are derived by pixel intensity and classifies them into multiple classes in the Ndimensional feature space. Ideally, each class is associated with one substance pictured in the image, such as an organelle in intracellular images. In the case of supervised classification, the distribution of image features of each class is initially specified by a user who has knowledge of the segmentation target. Then the classifier (i.e. classification rule), such as a discriminant function, is generated based on their distribution so as to assign the image features to a specified class. Manual segmentation is generally conducted for specifying classes. According to the generated classification rule, the computer is able to automatically classify the new inputs that are calculated from the still unsegmented images. As a result, target segmentation can be achieved by detecting only the pixels that have the feature classified as the target class (see Fig. 1).
Fig. 1. A segmentation approach based on pattern classification theory. In this approach, the user specifies the region of the segmentation target. However, the segmentation algorithm implemented by this classification technique is not general enough because there will be large differences in segmentation results depending on the algorithm used. That is, the segmentation results are greatly influenced by the type of features and classification rules adopted, and the optimal algorithm for one segmentation
422
Image Segmentation
task may not be optimal for a different one. To solve these problems, we propose a new framework that can select an optimal algorithm that satisfies the user's intention in each segmentation task. Here, "algorithm" means the set including the feature space constructed by the extracted image feature, the classification rule, and the parameter settings for generating the feature space and the classification rule. Our framework selects the algorithm that can extract the target region with the highest level of accuracy by means of our proposed evaluation metric, as long as the groundtruth is specified. As shown in Fig. 2, the algorithm that can segment the region most similarly to the groundtruth is automatically selected from a given set of algorithms.
Fig. 2. A framework of algorithm selection. 2.1 Selection metric The similarities between the groundtruth and the automatically segmented region produced by each given algorithm are used as an evaluation metric for selecting the proper algorithm in our framework. Previous researchers have used many evaluation metrics based on similarities with the groundtruth. For example, (Haindl & Mikes, 2008; Arbelaez, 2009) considered segmentations as a clustering of pixels, and used the Variation of Information (VI), which is based on the distance between two clusters in terms of their average conditional entropy, to measure similarity between two segmentations. Similarly, (Unnikrishnan et al., 2007) introduced the Rand Index (RI) for measuring the distance of two clusters by comparing the compatibility of assignments between pairs of elements in the clusters. Although it is possible to evaluate the performance of segmentation algorithms using the RI and VI and the number of segmented regions evaluated is not constrained with these indexes, their perceptual meaning (that is, an association between human judgement
Algorithm Selection Based on a Region Similarity Metric for Intracellular Image Segmentation
423
and these indexes) and applicability in the presence of the groundtruth reference (i.e., supervised evaluations) remains unclear. Martin et al. (Martin et al., 2001) proposed the similarity indexes called Global Consistence Error (GCE) and Local Consistent Error (LCE), which are wellknown as evaluation metrics for natural image segmentation. Although there are limitations in terms of the number of segmented regions that can be evaluated and computational cost, a notable advantage of these metrics is that supervised evaluation based on human perception can be conducted only from the viewpoint of region boundaries. Our evaluation metric for intracellular image segmentation is composed of similarity measurements between the groundtruth and automatically segmented regions, not only from the viewpoint of region boundaries but also from the statistical features in the segmented region. The similarity is measured by the distance of the intensitybased and the shapebased image features between the two regions. The algorithm that produces the minimum distance is defined as the optimal one for a given segmentation task. That is, a user can obtain the most accurate segmentation result by using the selected algorithm to segment a target that has similar characteristics to the groundtruth. It is well known that, if a highly accurate identification is achieved for a feature distribution with a certain classification rule (e.g., a discriminant function), the rule is also applicable to a similar feature distribution and can lead to accurate classification results (Duda et al., 2007). People generally focus on specific characteristics of a region when evaluating a segmented region. We consider that image features derived from the pixel intensity and boundary shape of the segmentation target are the most important characteristics. We defined S g as the target region of the groundtruth that is supervised by a user and S = {Sa , a ∈ A} as the automatically segmented regions by given algorithms in a plane (or a space). The similarity R A between those two regions can be calculated as follows:
RA =
1 dist(S g , S )
=
1 dist( X g , X A ) + dist( P g , PA )
,
(1)
where X = ( x1 , x2 ,..., xN ) represents the Ndimensional image features and P = ( p1 , p2 ,..., pn ) , ( p j ∈ C N ) represents the spatially discrete shape features of a region. The symbol dist(⋅) means the distance calculation of two elements. That is, R A is defined as a linear combination of dist( X g , X A ) and dist( P g , PA ) . We can select an optimal algorithm that can segment a similar region with the groundtruth as follows: ai = arg min
0 0 , ∀ω ∈ Ω
(
) (
(19)
)
P l i = ωil j = ω j , Pi ≠ Pj = P l i = ωil j = ω j , Pj ∈ N(Pi ) , ∀Pi ∈ P and ω ∈ Ω
(20)
where P(.) and P(..) are the joint and conditional probability density functions, respectively. The configurations of MRF obey a Gibbs distribution [Hammersley & Clifford, 1971] P ( ω ) = 1 / Z.exp( −U(ω) / T)
(21)
where Z is a normalizing constant and T is a constant called temperature. U(ω) is the energy function, which is a sum of clique potentials Vc (ω) on all possible cliques, i.e. U (ω ) = ∑Vc (ω )
(22)
c ∈C
In this method, the set of cliques is defined as C = {ci ci = {Pi } ∪N ( Pi ) , Pi ∈ P}
(23)
where each pixon in G = {P, E} defines one clique. And the clique potential is defined by ⎛ ⎞ ⎛ ⎞ Vci (ω ) = wc ⎜ P i ⎟ ∑ we bij , bi , b j wp ⎜ Pi , Pj ⎟ηij ⎝ ⎠ Pij ∈N ( Pi ) ⎝ ⎠
(
)
(24)
where ηij is a binary variable which has the value 1 if Pi and Pj have the same label and the value 0 otherwise; w c ⎜⎛ P i ⎟⎞ = n i is the clique weight; w e bij , bi , b j = bij / bi is the ⎝ ⎠ normalized edge weight; and w p ⎛⎜ Pi ,Pj ⎞⎟ = 1/μi − μ j  is the pixon distance weight that ⎝ ⎠ denotes the difference of image characteristics between two pixons.
(
)
505
PixonBased Image Segmentation
In all, the prior probability is defined as P ( S ) = P (ω ) =
bij ηij 1 1 exp( − ∑ ni ∑ ) Z T Pi ∈P Pij ∈N ( Pi ) bi  μi − μ j 
(25)
2.2.4.3 Optimization From (13) and (14), the optimal segmented image can be written as S * = arg min( − ln P ( IS ) − ln P ( S ) ).
(26)
s
Using (16) and (25), the objective function is then obtained, K
F ( S ) = F (ω ) = ∑
∑
k = 1 Pi ⊆ Γ K
[( μi − uK ) + σ i 2 ] 2
[ ni (
2σ K
2
+ ln σ k ) + α ni
∑
bij
ηij
] Pij ∈N ( Pi ) bi  μi − μ j 
(27)
where α = 1 / T is a weight of MRF model, which denotes the tradeoff between the fidelity to the observed image and the smoothness of the segmented image. The constant term has been removed from the objective function. The class number K and the weight α are given before optimization. The initial segmented image is obtained using Fuzzy CMeans (FCM) clustering, and the initial parameters of each class are estimated from the initial segmented image, i.e. the means u k and variances σ k . Then the threshold T is computed, the value of T should not be too large, otherwise the pixon will contain many pixels which actually belong to two different classes. So we using follow empirical function: T=
min
0 < i , k ≤ K , i ≠ j
( ui − u j − σ i − σ j )
(28)
Finally, the segmented image and the parameters are optimized, simultaneously. Let F(ω, I i,new ) denote the objective function value when the ith label of ω is changed into Ii ,new and ΔF(ω, I i,new ) denote F ( ω, I i ,new ) − F(ω) . The optimization is described as follows 1. Initialize the number of classes K ; the total number of iteration NUM ; u 1 , … , u k and σ 1 , … , σ k according to an initial segmentation, which is obtained using FCM method;
compute the threshold T ; and the iteration index j = 0 ; 2.
Extraction of pixonrepresentation, then initialize the pixonbased image model: assign a label λ k to each pixon P , which minimizes the expression μP − uk l 
3.
Find the best label for each pixon, l i,best , 1 ≤ i ≤ N , which minimizes ΔF(ω, I i,new ) .
4.
Find the ΔF(ω, I min ,best ) , satisfying ΔF ( ω, I min ,best ) ≤ ΔF ( ω, Ii,best ) , 1 ≤ i ≤ N
5.
If ΔF ( ω, I min ,best ) < 0 and j < NUM , go to step 4, otherwise stop iteration.
6.
Update the best label of each pixon and reestimate u k , σ k using new ω .
7.
j = j + 1 , Go to step 3.
506
Image Segmentation
In fact, ΔF ( ω, I i,best ) can be calculated using the correlative terms with the ith label in F(x) , i.e.
(
)
2 ⎛ ⎞ 2 ⎛ ni bij n j bij ⎞ ηij ,new ⎜ μi − uli ,new + σ i ⎟ ⎟ + + Fi (ω , I i , new ) = ni ⎜ σ ln li ,new ⎟ + α ∑ ⎜ 2 ⎜ b j ⎠⎟μi − μ j  2σ li ,new Pij ∈N ( Pi ) ⎝ bi ⎜ ⎟ ⎝ ⎠
(29)
2.3 WPB method Pixonbased approach using wavelet thresholding is a recently developed image segmentation method [Hassanpour et al, 2009]. In this method, a wavelet thresholding technique is initially applied on the image to reduce noise and to slightly smooth the image. This technique causes an image not to be oversegmented when the pixonbased method is used. Indeed, the wavelet thresholding, as a preprocessing step, eliminates the unnecessary details of the image and results in a fewer pixon number, faster performance and more robustness against unwanted environmental noises. The image is then considered as a pixonal model with a new structure. The obtained image is segmented using the hierarchical clustering method (Fuzzy CMeans algorithm). 2.3.1 PreProcessing step As mentioned above, the wavelet thresholding technique is used as a preprocessing step in order to smooth the image. For this purpose, by choosing an optimal wavelet level and an appropriate mother wavelet, the image is decomposed into different channels, namely lowlow, lowhigh, highlow and highhigh (LL, LH, HL, HH respectively) channels and their coefficients are extracted in each level. The decomposition process can be recursively applied to the low frequency channel (LL) to generate decomposition at the next level. The suitable threshold is achieved using one of the different thresholding methods and then details coefficients cut with this threshold. Then, inverse wavelet transform is performed and smoothed image is reconstructed.
2.3.1.1 Wavelet thresholding technique Thresholding is a simple nonlinear technique which operates on the wavelet coefficients. In this technique, each coefficient is cut by comparing to a value as the threshold. The coefficients which are smaller than the threshold are set to zero and the others are kept or modified by considering the thresholding method. Whereas the wavelet transform is good for energy compaction, the small coefficients are considered as noise and large coefficients indicate important signal features [Gupta & kaur, 2002]. Therefore, these small coefficients can be cut with no effect on the significant features of the image. Let X = { X i, j , i, j = 1, 2 …M} denotes the M × M matrix of the original image. The two dimensional orthogonal Discrete Wavelet Transform (DWT) matrix and its inverse are implied by W and W −1 , respectively. After applying the wavelet transform to the image matrix X, this matrix is subdivided into four subbands namely LL, HL, LH and HH [Burrus et al., 1998]. Whereas the LL channel possesses the main information of the image signal, we apply the hard or soft thresholding technique to the other three subbands which contain the details coefficients. The outcome matrix which is produced after utilizing the thresholding level is ˆ matrix. Finally, the smoothed image matrix can be obtained as follows: denoted as L
507
PixonBased Image Segmentation
Xˆ = W −1Lˆ
(30)
The brief description of the hard thresholding is as follows: ⎧⎪Y if Y > T ⎩⎪ 0 otherrwise
γ (Y ) = ⎨
(31)
where Y is an arbitrary input matrix, γ ( Y ) is the hard thresholding function which is applied on Y , and T indicates the threshold value. Using this function, all coefficients less than the threshold