Soft Computing in Image Processing: Recent Advances (Studies in Fuzziness and Soft Computing)

Mike Nachtegael, Dietrich Van der Weken, Etienne E. Kerre, Wilfried Philips (Eds.) Soft Computing in Image Processing ...

Author: Mike Nachtegael | Dietrich Van der Weken | Etienne E. Kerre | Wilfried Philips

71 downloads 1407 Views 17MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Mike Nachtegael, Dietrich Van der Weken, Etienne E. Kerre, Wilfried Philips (Eds.) Soft Computing in Image Processing

Studies in Fuzziness and Soft Computing, Volume 210 Editor-in-chief Prof. Janusz Kacprzyk Systems Research Institute Polish Academy of Sciences ul. Newelska 6 01-447 Warsaw Poland E-mail: [email protected] Further volumes of this series can be found on our homepage: springer.com Vol. 194. Dawn E. Holmes, Lakhmi C. Jain (Eds.) Innovations in Machine Learning, 2006 ISBN 3-540-30609-9 Vol. 195. Zongmin Ma Fuzzy Database Modeling of Imprecise and Uncertain Engineering Information, 2006 ISBN 3-540-30675-7 Vol. 196. James J. Buckley Fuzzy Probability and Statistics, 2006 ISBN 3-540-30841-5 Vol. 197. Enrique Herrera-Viedma, Gabriella Pasi, Fabio Crestani (Eds.) Soft Computing in Web Information Retrieval, 2006 ISBN 3-540-31588-8 Vol. 198. Hung T. Nguyen, Berlin Wu Fundamentals of Statistics with Fuzzy Data, 2006 ISBN 3-540-31695-7 Vol. 199. Zhong Li Fuzzy Chaotic Systems, 2006 ISBN 3-540-33220-0 Vol. 200. Kai Michels, Frank Klawonn, Rudolf Kruse, Andreas Nürnberger Fuzzy Control, 2006 ISBN 3-540-31765-1 Vol. 201. Cengiz Kahraman (Ed.) Fuzzy Applications in Industrial Engineering, 2006 ISBN 3-540-33516-1

Vol. 202. Patrick Doherty, Witold Łukaszewicz, Andrzej Skowron, Andrzej Szałas Knowledge Representation Techniques: A Rough Set Approach, 2006 ISBN 3-540-33518-8 Vol. 203. Gloria Bordogna, Giuseppe Psaila (Eds.) Flexible Databases Supporting Imprecision and Uncertainty, 2006 ISBN 3-540-33288-X Vol. 204. Zongmin Ma (Ed.) Soft Computing in Ontologies and Semantic Web, 2006 ISBN 3-540-33472-6 Vol. 205. Mika Sato-Ilic, Lakhmi C. Jain Innovations in Fuzzy Clustering, 2006 ISBN 3-540-34356-3 Vol. 206. A. Sengupta (Ed.) Chaos, Nonlinearity, Complexity, 2006 ISBN 3-540-31756-2 Vol. 207. Isabelle Guyon, Steve Gunn, Masoud Nikravesh, Lotfi A. Zadeh (Eds.) Feature Extraction, 2006 ISBN 3-540-35487-5 Vol. 208. Oscar Castillo, Patricia Melin, Janusz Kacprzyk, Witold Pedrycz (Eds.) Hybrid Intelligent Systems, 2007 ISBN 3-540-34719-1 Vol. 209. Alexander Mehler, Reinhard Köhler Aspects of Automatic Text Analysis, 2007 ISBN 3-540-37520-1 Vol. 210. Mike Nachtegael, Dietrich Van der Weken, Etienne E. Kerre, Wilfried Philips (Eds.) Soft Computing in Image Processing, 2007 ISBN 3-540-38232-1

Mike Nachtegael Dietrich Van der Weken Etienne E. Kerre Wilfried Philips (Eds.)

Soft Computing in Image Processing Recent Advances

ABC

Mike Nachtegael Dietrich Van der Weken Etienne E. Kerre

Wilfried Philips

Department of Applied Mathematics and Computer Science Ghent University Krijgslaan 281 – S9 9000 Gent Belgium

Department of Telecommunications and Information Processing Ghent University Sint-Pietersnieuwstraat 41 9000 Gent Belgium

Library of Congress Control Number: 2006932180

ISSN print edition: 1434-9922 ISSN electronic edition: 1860-0808 ISBN-10 3-540-38232-1 Springer Berlin Heidelberg New York ISBN-13 978-3-540-38232-4 Springer Berlin Heidelberg New York This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable for prosecution under the German Copyright Law. Springer is a part of Springer Science+Business Media springer.com c Springer-Verlag Berlin Heidelberg 2007 The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typesetting: by the authors and techbooks using a Springer LATEX macro package Cover design: Erich Kirchner, Heidelberg Printed on acid-free paper

SPIN: 11588696

89/techbooks

543210

Preface

Soft Computing in Image Processing: Recent Advances covers a wide range of both theoretical and practical applications of this exciting topic. The chapters have been grouped into five parts: Applications in Remote Sensing (Chapters 1 to 3), Applications in Image Retrieval (Chapters 4 and 5), Applications in Image Analysis (Chapters 6 to 8), Other Applications (Chapters 9 to 12), Theoretical Contributions (Chapters 13 to 16). The first part deals with applications in remote sensing. In the first chapter a wavelet feature based classification approach for land cover classification of multispectral remotely sensed images is presented. This method explores the possible advantages of using a wavelet transform as a preprocessor for both non-fuzzy and fuzzy classifiers. The improvement in performance of the classification scheme is verified from the results obtained from different remote sensing images. In Chapter 2, efficient algorithms for shift, rotation, scaling and registration, and their applications to geosciences are discussed, especially in low-accuracy situations. The proposed algorithms are extensions of existing algorithms with a special emphasis on algorithms that are used to process low accuracy images. The performance of the extended method is illustrated with some examples for which the existing algorithms do not work. Finally, Chapter 3 surveys key neural and non-neural methods in multiple classifier systems for remote sensing image classification and develops some new methods in this area. Several architectures of neural networks are employed, including traditional feed-forward neural networks, probabilistic neural networks and radial-basis nets. All methods are evaluated on two real remote sensing datasets. Image retrieval, which is the topic of the second part of the book, is also a well-known research area. In Chapter 4, a linguistic expression-based image description framework for image retrieval is discussed. The aim is to provide images with semantic expression based on linguistic variables in fuzzy set theory. The method is illustrated with some examples in texture image retrieval and these experiments show that the semantic feature can effectively capture image texture semantics. Furhtermore, the semantic-based image retrieval method is effective and convenient. Finally, the framework can deal with vagueness and subjectivity thanks to the advantage of fuzzy set theory. Chapter 5 addresses the problem of keyword focusing. In this chapter, the keyword focusing problem is defined and discussed. Furthermore, a statistical

VI

Preface

correlation method for keyword focusing on the conceptual level is presented. Several experiments show the promising performance of this method and provided some ideas for further improvement. The third part of the book deals with applications in image analysis. A fuzzy edge-dependent interpolation algorithm is presented in Chapter 6. The proposed algorithm is based on the edge-based line average algorithm and aims at an efficient tradeoff between performance and low computational cost. The superior performance of the novel interpolation algorithm is proven by extensive simulations on standard images and video sequences. In Chapter 7, new fuzzy cluster algorithms within the image segmentation procedure are proposed. These algorithms are based on the triangular function and its ndimensional extension. The usage of the algorithms is illustrated for both gray level and colour image segmentation. Chapter 8 addresses one of the most important application areas of soft computing in image processing, namely image restoration. Several neuro-fuzzy methods for noise filtering, noise detection and edge extraction in digital images corrupted by impulse noise are presented. The efficiency of the presented approach is demonstrated by appropriately designed simulation experiments, and the performance is compared with a number of selected operators from the literature. The chapter is completed with a brief summary of the advantages and other potential applications of the presented neuro-fuzzy operators in image processing. Some various other applications are discussed in the fourth part of the book. Chapter 9 is focused on creating photorealistic models by data fusion with genetic algorithms. A software system is presented for building photorealistic 3D models. It operates with accurate 3D models measured by a laser scanner and high quality images of the object acquired separately by a digital camera. Some test results with real data demonstrate the efficiency of the proposed method. In Chapter 10, the use of fuzzy information granular for content representation of video sequences is demonstrated. This method is based on a general-purpose content representation framework for video sequences that employs fuzzy information granulation to capture human subjectivity. Experimental results using different types of video sequences are presented to demonstrate the efficiency and accuracy. An important application in medical image processing is discussed in Chapter 11, namely the detection of different kinds of lesions in Digital Breast Tomosynthesis (DBT) datasets. The advantage of the proposed approach is that it works directly on the DBT projected views such that it allows to work independently of the reconstruction algorithm used to generate the 3D images. Furthermore, the processing time is significantly reduced compared to the application of similar operators on reconstructed slices. Finally, Chapter 12 presents the concept of intelligent biometric information fusion using support vector machine. First, the amplitude and phase feature based face recognition algorithm are described. After this, three fusion algoritms are presented which use support vector machine to intelligently fuse the information at three different levels.

Preface

VII

Finally, Part 5 of the book offers the reader some theoretical contributions. In Chapter 13 a generalized definition of fuzzy mathematical morphology operations is presented. Besides the theoretical base, different applications are discussed: some applications in the field of computation with uncertainties, but also applications to image segmentation. Chapter 14 establishes a flexible mathematical framework for image processing based on the concepts of intuitionistic fuzzy set theory. The efficiency of the proposed intuitionistic fuzzy framework is demonstrated in the context of contrast enhancement and edge detection and yields satisfactory and promising results. Chapter 15 presents some integer programming and combinatorial optimization methods that are specifically proposed to solve practical problems in image processing. The presented algorithms focus on the minimization of functions that can appear in different image processing applications. Finally, in Chapter 16 a colour quantisation by black-box optimization is introduced. The main advantage of this approach is that it is able to provide a near optimal solution without any domain specific knowledge. The effectiveness of this approach is evaluated by comparing its performance with several other colour quantisation algorithms. It should also be noted that colour figures are reproduced in black and white throughout the book, but that the original coloured images are included in an appendix at the end.

Gent, May 2006

1

Mike Nachtegael1 Dietrich Van der Weken Etienne E. Kerre Wilfried Philips

Editors’ contact address: Ghent University, Department of Applied Mathematics and Computer Science, Fuzziness and Uncertainty Modelling Research Unit, Krijgslaan 281-S9, B-9000 Gent, Belgium. Email: [email protected], [email protected], [email protected], [email protected]

Contents

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . V List of Contributors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . XV List of Reviewers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .XIX Part I Applications in Remote Sensing 1. Multispectral Remote Sensing Image Classification Using Wavelet Based Features Saroj K. Meher, B. Uma Shankar and Ashish Ghosh . . . . . . . . . . . . . . . . . 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Wavelet transformation based feature extraction . . . . . . . . . . . . . . . . . . 3 Classification techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Performance measurement index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Results and discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2. Images with Uncertainty: Efficient Algorithms for Shift, Rotation, Scaling, and Registration, and Their Applications to Geosciences Cara G. Schiek, Roberto Araiza, Jose Manuel Hurtado, Aaron A. Velasco, Vladik Kreinovich, Victor Sinyansky . . . . . . . . . . . . . . . . . . . . . . . . 1 General motivation and goals of the chapter . . . . . . . . . . . . . . . . . . . . . 2 Structure of the chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Registering images known with high accuracy . . . . . . . . . . . . . . . . . . . . 4 Images known with low accuracy: case study . . . . . . . . . . . . . . . . . . . . . 5 How to extend the existing image registration techniques to images known with low accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Results of applying the new image registration algorithm . . . . . . . . . . 7 Conclusions and future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3 3 7 14 20 21 29 30

35 35 36 36 48 53 56 56 59

X

Contents

3. Neural Networks in Multiple Classifier Systems for Remote-Sensing Image Classification Moumen T. El-Melegy, Safaa M. Ahmed . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 The problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Neural classifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Neural decision fusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Classifier selection using neural networks . . . . . . . . . . . . . . . . . . . . . . . . 6 Selection and fusion using neural networks . . . . . . . . . . . . . . . . . . . . . . . 7 Experimental results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

65 65 69 70 74 75 77 78 89 91

Part II Applications in Image Retrieval 4. Linguistic Expression Based Image Description Framework and its Application to Image Retrieval Qingyong Li, Zhiping Shi, Zhongzhi Shi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 3 Linguistic expression based image description framework . . . . . . . . . . 101 4 Application in texture image retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 5. Focusing Keywords to Automatically Extracted Image Segments Using Self-Organising Maps Ville Viitaniemi, Jorma Laaksonen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 3 Image content analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 4 Image segmentation for image content analysis . . . . . . . . . . . . . . . . . . . 126 5 Focusing keywords in image segments . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 6 System implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 7 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 8 Conclusions and future views . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 Part III Applications in Image Analysis 6. A Fuzzy Edge-Dependent Interpolation Algorithm Piedad Brox, Iluminada Baturone, Santiago S´ anchez-Solano . . . . . . . . . . . 157 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 2 Fuzzy-ELA algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158

Contents

XI

3 De-interlacing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 4 Image enlargement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 5 Hardware implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 7. Fuzzy Image Segmentation Based on Triangular Function and Its n-dimensional Extension Vasile P˘ atra¸scu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 2 The triangular function and its n-dimensional extension . . . . . . . . . . . 188 3 The fuzzy clustering algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 4 The color systems IJK and ISH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 5 The fuzzy image segmentation methods . . . . . . . . . . . . . . . . . . . . . . . . . 199 6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206 8. Application of Neuro-Fuzzy Methods for Noise Filtering, Noise Detection and Edge Extraction in Digital Images Corrupted by Impulse Noise Emin Y¨ uksel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 2 Literature review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 3 The neuro-fuzzy operator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214 4 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 5 Conclusions and remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231 6 Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232 Part IV Other Applications 9. Creating Photorealistic Models by Data Fusion with Genetic Algorithms Dmitry Chetverikov, Zsolt Jank´ o, Evgeny Lomonosov, Aniko Ek´ art . . . . . 239 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 2 Pre-registration of surfaces using a genetic algorithm . . . . . . . . . . . . . . 242 3 Fusion of surface and image data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246 4 Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257 5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262 10. The Use of Fuzzy Information Granular for Content Representation of Video Sequences Congyan Lang, De Xu, Jiehua Yu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265

XII

Contents

2 System overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268 3 Pre-processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 4 Salient region extraction based on fuzzy information granular . . . . . . 270 5 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282 6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291 11. Wavelets and Fuzzy Contours in 3D-CAD for Digital Breast Tomosynthesis Gero Peters, Serge Muller, Sylvain Bernard, Isabelle Bloch . . . . . . . . . . . . 293 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 2 Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295 3 Candidate particle detection using wavelets . . . . . . . . . . . . . . . . . . . . . . 299 4 Fuzzy contours and fuzzy measurements . . . . . . . . . . . . . . . . . . . . . . . . . 305 5 Partial defuzzification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311 6 Aggregation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313 7 3D processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315 8 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317 9 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321 12. Intelligent Biometric Information Fusion using Support Vector Machine Richa Singh, Mayank Vatsa, Afzel Noore . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325 2 Face recognition using phase and amplitude features . . . . . . . . . . . . . . 326 3 Overview of support vector machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329 4 Information fusion in face biometrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334 5 Experimental results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339 6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345 7 Acknowledgment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346 Part V Theoretical Contributions 13. General Definition of Fuzzy Mathematical Morphology Operations Antony T. Popov . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353 2 Binary mathematical morphology and the abstract lattice theory . . 354 3 Fuzzy morphological operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359 4 T-invariant fuzzy morphological operations . . . . . . . . . . . . . . . . . . . . . . 362 5 Fuzzy geodesic morphological operations . . . . . . . . . . . . . . . . . . . . . . . . 364 6 Computations with intervals and fuzzy numbers . . . . . . . . . . . . . . . . . . 366 7 Fuzzy mathematical morphological operations for colour images . . . . 370

Contents

XIII

8

Morphological operations and an efficient way to compute fractal dimension . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 376 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379 14. Intuitionistic Fuzzy Image Processing Ioannis K. Vlachos, George D. Sergiadis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 383 2 Elements of intuitionistic fuzzy sets theory . . . . . . . . . . . . . . . . . . . . . . 384 3 Towards intuitionistic fuzzy image processing . . . . . . . . . . . . . . . . . . . . 387 4 The intuitionistic fuzzy image processing framework . . . . . . . . . . . . . . 392 5 Intuitionistic fuzzy histogram hyperbolization . . . . . . . . . . . . . . . . . . . . 398 6 Intuitionistic fuzzy contrast intensification . . . . . . . . . . . . . . . . . . . . . . . 401 7 Edge detection driven by hesitancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407 8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412 15. Integer Programming Methods in Image Processing and Bayes Estimation Boris A. Zalesky . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415 2 Image processing as discrete optimization problem . . . . . . . . . . . . . . . . 417 3 Multiresolution network flow minimum cut algorithm . . . . . . . . . . . . . 419 4 Integer minimization of Umod,f (m) and Usec,f (m) . . . . . . . . . . . . . . . . . 425 5 Combinatorial minimization of submodular functions . . . . . . . . . . . . . 431 6 Representability of some submodular polynomials by graphs . . . . . . 438 7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444 16. Colour Quantisation by Black-Box Optimisation Gerald Schaefer, Lars Nolle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445 2 Simulated annealing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 446 3 Simulated annealing for colour quantisation . . . . . . . . . . . . . . . . . . . . . . 449 4 A hybrid approach to colour quantisation . . . . . . . . . . . . . . . . . . . . . . . . 450 5 Experimental results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 451 6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 462 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 462 Appendix: Colour Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 497

List of Contributors

Safaa M. Ahmed National Authority of Remote Sensing and Space Science Cairo Egypt

Dmitry Chetverikov Computer and Automation Research Institute Budapest Hungary

Roberto Araiza University of Texas at El Paso El Paso United States

Aniko Ek´ art Computer and Automation Research Institute Budapest Hungary

Iluminada Baturone Instituto de Microelectr´ onica de Sevilla Sevilla Spain Sylvain Bernard GE Healthcare Europe Buc France Isabelle Bloch Ecole Nationale Supérieure des Télécommunications Paris France Piedad Brox Instituto de Microelectr´ onica de Sevilla Sevilla Spain

Moumen T. El-Melegy Assiut University Assiut Egypt Ashish Ghosh Indian Statistical Institute Kolkata India Jose Manuel Hurtado University of Texas at El Paso El Paso United States Zsolt Jank´ o Computer and Automation Research Institute Budapest Hungary Vladik Kreinovich University of Texas at El Paso El Paso United States

XVI


Jorma Laaksonen Helsinki University of Technology Helsinki Finland Congyan Lang Beijing Jiaotong University Beijing China

Gero Peters GE Healthcare Europe Buc France Antony T. Popov St. Kliment Ohridski University of Sofia Sofia Bulgaria

Qingyong Li Chinese Academy of Sciences Beijing China

Santiago S´ anchez-Solano Instituto de Microelectrónica de Sevilla Sevilla Spain

Evgeny Lomonosov Computer and Automation Research Institute Budapest Hungary

Gerald Schaefer Nottingham Trent University Nottingham United Kingdom

Saroj K. Meher Indian Statistical Institute Kolkata India

Cara G. Schiek University of Texas at El Paso El Paso United States

Serge Muller GE Healthcare Europe Buc France

George D. Sergiadis Aristotle University of Thessaloniki Thessaloniki Greece

Lars Nolle Nottingham Trent University Nottingham United Kingdom

Bhavan Uma Shankar Indian Statistical Institute Kolkata India

Afzel Noore West Virginia University Morgantown United States

Zhiping Shi Chinese Academy of Sciences Beijing China

Vasile Patrascu Tarom Company Otopeni Romania

Zhongzhi Shi Chinese Academy of Sciences Beijing China


Richa Singh West Virginia University Morgantown United States Victor Sinyanski Raytheon Waltham United States Mayank Vatsa West Virginia University Morgantown United States Aaron A. Velasco University of Texas at El Paso El Paso United States Ville Viitaniemi Helsinki University of Technology Helsinki Finland

Ioannis K. Vlachos Aristotle University of Thessaloniki Thessaloniki Greece De Xu Beijing Jiaotong University Beijing China Jiehua Yu Beijing Jiaotong University Beijing China Emin Y¨ uksel Erciyes University Kayseri Turkey Boris A. Zalesky National Academy of Sciences Minsk Belarus

XVII

List of Reviewers

Ahmad Ashraf National Chiao Tung University Hsinchu Taiwan Krassimir Atanassov Centre for Biomedical Engineering Sofia Bulgaria Sudhirkumar Barai Indian Institute of Technology Kharagpur India Isabelle Bloch Ecole Nationale Supérieure des Télécommunications Paris France Ulrich Bodenhofer Software Competence Center Hagenberg Hagenberg Austria Ioan Buciu University of Oradea Oradea Romania Pascual Campoy Universidad Politecnica Madrid Madrid Spain

Olivier Colot Université des Sciences et Technologies de Lille Villeneuve d’Asq Cedex France Sergio Damas University of Granada Granada Spain Alexandre G. Evsukoff COPPE/UFRJ Rio de Janeiro Brazil Peter Goebel Karl Landsteiner Institute for Biotelematics Wien Austria Mustafa Gunes University of Dokuz Eyl¨ ul Buca-Izmir Turkey Alexandru Isar Politehnica University of Timisoara Timisoara Romania Daniel Keysers RWTH Aachen University of Technology Aachen Germany

XX

List of Reviewers

Vladik Kreinovich University of Texas at El Paso El Paso United States

Enrique Cabello Pardos Universidad Rey Juan Carlos (ESCET) Mostoles (Madrid) Spain

Senthil Kumar National Remote Sensing Agency Balanagar Hyderabad India

Vasile Patrascu TAROM Company Otopeni, Ilfov Romania

Thomas Lehmann Aachen University of Technology Aachen Germany

Witold Pedrycz University of Alberta Edmonton Canada

Jesus Chamorro Martinez University of Granada Granada Spain

Tuan Pham James Cook University Townsville Australia

Francesco Masulli Universita di Pisa Genova Italy

Jo˜ ao Miguel Fernandes Rodrigues University of Algarve Faro Portugal

Kalana Mendis Advanced Technical Institute Labuduwa, Akmeemana Sri Lanka

Stefano Rovetta University of Genova Genova Italy

Roberto Rodriguez Morales Institute of Cybernetics, Mathematics and Physics Habana Cuba

Daniel Sanchez University of Granada Granada Spain

Mirko Navara Czech Technical University Praha Czech Republic

George Sergiadis Aristotle University of Thessaloniki Thessaloniki Greece

Okun Oleg University of Oulu Oulu Finland

Aureli Soria-Frisch Fraunhofer IPK Berlin Germany

List of Reviewers

Muhammad Suzuri Hitam University College of Science and Technology Malaysia Mengabang Telipot Malaysia Constantin Vertan Politehnica University of Bucharest Bucharest Romania Ioannis Vlachos Aristotle University of Thessaloniki Thessaloniki Greece

Brendon Woodford University of Otago Dunedin New Zealand

Jiahua Wu Wellcome Trust Sanger Institute Hinxton United Kingdom

XXI

Part I

Applications in Remote Sensing

Chapter 1 Multispectral Remote Sensing Image Classification Using Wavelet Based Features Saroj K. Meher, Bhavan Uma Shankar and Ashish Ghosh Machine Intelligence Unit Indian Statistical Institute 203 B. T. Road Kolkata 700108, India {saroj t,uma,ash}@isical.ac.in

1 Introduction Multispectral remotely sensed images composed information over a large range of variation on frequencies (information) and these frequencies change over different regions (irregular or frequency variant behavior of the signal) which need to be estimated properly for an improved classification [1, 2, 3]. Multispectral remote sensing (RS) image data are basically complex in nature, which have both spectral features with correlated bands and spatial features correlated within the same band (also known as spatial correlation). An efficient method for utilization of these spectral and spatial (contextual) information can improve the classification performance significantly compared to the conventional non-contextual information based methods. In general, the spectral component of a remotely sensed image pixel may be a mixture of more than one spectral information that usually comes from the different regions of the study area. However, the conventional multispectral remotely sensed image classification systems detect object classes only according to the spectral information of the individual pixel/pattern in a particular frequency band, while a large amount of spatial and spectral information of neighboring pixels of different regions at other frequency bands are neglected. Hence the pixels are classified based on its spectral intensities of a specific band and does not give attention to its spatial and spectral dependencies and thus the spectral intensities of the neighbors at different frequency bands are assumed to be independent. Such approaches may be reasonable if spatial resolution is high or when the spectral intensities are well separated for different classes, which is rarely found in any real life data sets. For example, in the classification of urban areas, the densities of the spectral intensities are seldom well separated. Thus it is important to decide whether the arrangements of spatial data can be used as features directly in its original form or with a set of extracted features obtained through any feature-extraction method where, the information S.K. Meher et al.: Multispectral Remote Sensing Image Classification Using Wavelet Based Features, StudFuzz 210, 3–34 (2007) c Springer-Verlag Berlin Heidelberg 2007 www.springerlink.com

4

S.K Meher et al.

are uncorrelated both in spatial and spectral domain to separate the data into desired classes. Many research efforts have been made to take the advantages of neighboring pixel information (texture feature information [1, 4, 5]) and applied for the classification of remotely sensed data. These include texture features extracted from angular second moments, contrast, correlation, entropy, variance, etc., computed from the gray level co-occurrence matrices [4]. These extracted texture features play an important role and help to increase the performance of the classifier. However, these methods are computationally expensive due to the estimation of autocorrelation parameters and transition probabilities. Also, the texture elements are not easy to quantify and it deals with the spatial distribution of the gray levels over a portion of an image. Later Gaussian Markov random fields (GMRF) [6, 7, 8] and Gibbs random fields [9] were proposed to characterize textures. Further, local linear transformations are also used to compute texture features [10]. These conventional statistical approaches to texture analysis such as co-occurrence matrices, second order statistics, GMRF and local linear transforms are restricted to the analysis of spatial interactions over relatively small neighborhoods on a single scale. As a matter of fact, their performance is best for the analysis of micro-textures only [11]. Apart from the texture feature extraction, various other aspects like uncorrelation of spectral information called as spectral unmixing, subpixel analysis and super resolution [12, 13, 14], spectral angle mapping are the important methods of exploring the spectral and spatial behavior of pixels. A brief review of spectral unmixing methods have been described by Keshava and Mustard in [15]. Brown et al. [16] proposed a support vector Machines based spectral unmixing and made a comparison with the linear spectral mixture models (LSMMs) and artificial neural networks. Chang and Ren in [17] performed a comparative study of mixed pixel classification algorithms. In an another study, Faraklioti and Petrou [18] described an statistical approach of spectral unmixing named as illumination invariant unmixing of sets of mixed pixels for classification. Shah and Varshney [19] proposed an higher order statistical approach to spectral unmixing of remote sening imagery, that is derived from independent component analysis [20, 21, 22]. In [23], Burke et al. described the application of parametric feature extraction and classification strategy for braincomputer interfacing. Song in [24] described the spectral mixture analysis for subpixel vegetation fractions in the remote sensing environmental analysis. An automated extraction of endmember (uncorrelated synthesized information) in both spatial and spectral domain for multidimensional morphological operations has been described by Plaza et al. [25], where the authors proposed a new automated method that performs unsupervised pixel purity determination and endmember extraction from multidimensional datasets, which is achieved with the use of both spatial and spectral information. In a similar context, Subpixel analysis is another important method of information extraction from a pixel. In this regard Barducci et al. [26] described the performance and the characteristics of a new algorithm for subpixel image

Wavelets for Multispectral Image Classification

5

classification, performed by means of a fully constrained least-squares estimator operated with a multi-resolution approach in the framework of the linear mixing model. Broadwater et al. [27] described a hybrid algorithm for subpixel detection in hyperspectral imagery, where the hybrid detector based on both adaptive matched subspace detector and fully constrained least squares was developed to take the advantage of each detectors strengths. In a soft computing paradigm, Tatem et al. [28] described a super-resolution land cover pattern prediction using a Hopfield neural network. Weiguo et al. [29] proposed an adaptive resonance theory MAP (ARTMAP) neural network based mixture analysis model for subpixel classification of land covers in remote sening images. Pastina et al. [30] described a two-dimensional super-resolution spectral analysis applied to synthetic aperture radar images, where he discussed a technique for improving the geometrical resolution of the image with respect to the numerical values related to the compressed coded waveform and the synthetic aperture, so that subsequent classification procedures will have improved performance as well. A super resolution feature extraction techniques for military and civilian applications such as the detection and classification of moving targets in strong clutter background have been described in [31]. Foody et al. [32], described a super-resolution mapping of the shoreline through soft classification analysis, where they applied simulated annealing to the output of a soft classification. In a similar approach of information extraction, spectral angle mapping model has been used for rapid assessment of soil salinization in arid area [33]. However, in the conventional multispectral image classification method, the pixel information in either of spectral or spatial/textural domain are usually not sufficient to discriminate properly the different land cover classes. Hence, use of both spectral and textual information in classification problem is more appropriate. In this regard, many approaches like spectral unmixing, subpixel classification, etc. as discussed above have been adopted in the literature. Some attempts were also been made to use both spectral and textual features [5, 34, 35, 36, 37]. But, the texture feature extraction based on these methods normally assume the stationarity of the texture within the analysis range and hence may not be much useful. For this reason the characterization of the texture should be in both spatial and frequency domains simultaneously. One efficient way to deal with such problems is to recognize the image by a number of subsampled approximations of it at different resolutions and desired scales. This approach is conventionally called multiresolution and/or filter bank analysis [38, 39]. In this analysis particularly, Gabor filters and wavelet transform (WT) have received a lot of attention [11, 40, 41, 42]. However, a major disadvantage in using Gabor transform is that the output of Gabor filter banks are not mutually orthogonal, which may result in a significant correlation between texture features. Moreover, these transformations are usually not reversible, which limits their applicability for texture synthesis. Using the WT, which provides a precise and unifying framework for the analysis and characterization of a signal at different scales [11], most of these

6

S.K Meher et al.

disadvantages can be avoided. Another advantage of WT over Gabor filter is that, the low pass and high pass filters used in the WT remain the same between two consecutive scales while the Gabor approach requires filters of different parameters [40]. In other words, Gabor filters require proper tuning of filter parameters at different scales. As discussed above, to capture the local variations in the orientation and frequency of texture elements that lead to the large scale frequency variation behavior of remotely sensed image textures, we need a joint spatial and frequency representation. In this regard, WT is found to be a promising tool for texture analysis in both spatial and frequency domains [43] simultaneously, as it has the ability to examine the signal at different scales. This means, the WT represents the input information in both spatial and frequency domain in a combined manner. The WT scheme thus, analyzes the coarse image first and gradually increases the resolution to analyze the finer details. Basically the WT coefficients represent the characteristic in sub-frequency bands, indicating the characteristic of the original pixel information in that location from where the WT coefficients originated. Thus the WT coefficients obtain neighboring pixels information that are uncorrelated in spatial domain. Hence the use of classification methods with these coefficients instead of original pixel value or the pixel with extracted features either in the spatial or spectral domain separately is more justifiable. These characteristics of the WT motivates us to use it for extraction of hidden features from images taken from remote sening imagery with large scale frequency variation behavior of the pixels. Research works related to texture classification using WT [11, 41] has already been carried out. Also the use of statistical correlation based features in the WT domain for classification have also been reported in [42, 44]. In a similar approach, some research studies have been made to use WT for target classification of remote sensing images [45, 46]. A comparative study of multiwavelet [47, 48], wavelet, and shape features [4], for microcalcification classification in mammogram has been described in [49], where it is experimentally shown that the results of multiwavelet based approach were better compared to wavelet and statistical based feature. However the computational complexities of multiwavelet based classification is much higher than the wavelet based one. In the present work we have tried to explore the advantages of WT, instead of multiwavelet and statistical based approaches, by incorporating it as a preprocessor in both non-fuzzy [50] and fuzzy classifiers [51, 52, 56, 57]. In this method we first extract features of the input patterns/pixels using WT and use these features for the next stage of classification. We have evaluated the performance of the WF based classification methods with different wavelets. Six classifiers are used for the performance evaluations of the wavelet and original feature based methods. Two of them are from non-fuzzy and the rest four from fuzzy. In the non-fuzzy cases, the simple and most popular classifiers known as: conventional maximum likelihood (ML) [50] and k-nearest neighbor (k-NN) [53] are used. Similarly in the fuzzy domain, we have considered the fuzzy version of the non-fuzzy classifiers considered above i.e.,


7

fuzzy ML (FML) [54] and fuzzy k-NN (Fk-NN) [55]. The intention behind the consideration of the fuzzy version of non-fuzzy classifiers is that, a comparative performance can also be made in both WF based fuzzy and non-fuzzy classifiers. In addition to these, we have also used two newly developed fuzzy classifiers known as, fuzzy product aggregation reasoning rule (FPARR) proposed by Ghosh et al. [56] and and fuzzy explicit (FE) method proposed by Melgani et al. [57]. Comparison of classification results showed that the WF based classification scheme is yielding superior performance with all wavelets and classifiers. However the performance of the classifiers are better with biorthogonal wavelets. All the classifiers are tested on three multispectral remote sening images; two four-band Indian Remote Sensing 1A (IRS-1A) satellite images and one three-band SPOT image [58] for land covers classification. The organization of the chapter is as follows. Section 2, describes the wavelet transform based feature extraction method. Different non-fuzzy and fuzzy classifiers have been discussed in Section 3. A brief discussion of performance analysis index has been made in Section 4. Section 5 summarized the comparative results with discussion. Finally a concluding remark is made in Section 6.

2 Wavelet transformation based feature extraction To deal with the frequency variation behavior of signals in an appropriate way, many research efforts have been made. Efforts are made to overcome the disadvantages of the Fourier transform (FT) [39, 43], which assumes the signal to be stationary (frequency invariant) within its total range of analysis. WT [38, 39, 43, 59] is an example of such a transform. The most important difference of the WT from FT is that the time localization of the signal frequencies will not be lost. This problem can be sorted out to some extend using short time FT (STFT) [39]. However, the selection of window is a main problem in STFT. As, narrow windows give good time resolution, but poor frequency resolution and wide windows give good frequency resolution, but poor time resolution; furthermore, wide windows may violate the condition of stationarity. The problem, in fact, is the selection of window function once and use that window in the entire analysis. The WT on the other hand is an extension of these transforms. Prominent frequencies in the original signal will appear as high amplitude in the region of the wavelet transformed signal that includes those frequencies. If the main information of the signal lies in the high frequency region, as happens most often, the time localization of these frequencies will be more precise, since they are characterized by more number of samples. If the main information lies only at very low frequency regions, the time localization will not be very precise, since few samples are used to express signal at these frequencies. This procedure in effect offers a good time resolution at high frequencies, and good frequency resolution at low frequencies.

8

S.K Meher et al.

WT basically extends single scale analysis to multiscale analysis. The multiscale behavior of the WT analyze or decompose the signal in multiple number of scales, where each scale represents a particular coarseness of the analyzed signal. Thus the decomposition steps divide the signal into a set of signals of varying coarseness ranging from low frequency to high frequency components. Accordingly the WT tries to identify both the scale and space information of the events simultaneously. This makes the WT useful for signal feature analysis. This property is more useful for remote sensing data analysis, where, any characteristic of the scene is first analyzed using low resolution and then analyze an area of interest in detail using an appropriate higher resolution level. Since we are dealing with irregular textures of remotely sensed images, the decomposition of the signal into different scales, which can uncorrelate the data as much as possible without losing their main distinguishable characteristic is particularly more useful when the WT is done on an orthogonal basis [38]. The orthogonal basis is more compact in its representation, as it allows the decomposition of the underlaying space into orthogonal subspaces, which makes it possible to ignore some of the decomposed signals. WT is also suitable for operations like feature extraction [60, 61], parameter estimation [62, 63] and exact reconstruction of the signal series because of its invertible properties [38, 39]. WT is identical to a hierarchical subband system, where the subbands are logarithmically spaced in frequency. The wavelets are the functions used in the transformation which act as a basis for representing many functions of the same family. A series of functions can be generated by translation and dilation of these functions called mother wavelets ψ(x). The translation and dilation of the mother wavelet can be given by t−τ , γ = 0 and, γ ε R, τ ε R (1) ψγ,τ (t) = |γ|−1/2 ψ γ where τ and γ are the translation and dilation parameters, respectively. R is any real number. We have tested the classification of the remote sensing images using wavelets from different (Daubechies, Biorthogonal, Coiflets, Symlets) groups [59], however results are given for four wavelets as their performances are comparatively (empirically) better than others. These are Daubechies 3 (Db3), Daubechies 6 (Db6), Biorthogonal 3.3 (Bior3.3) and Biorthogonal 3.5 (Bior3.5) wavelets [59]. In Daubechies wavelets, N (3 or 6) is related to support width, filters length, regularity and number of vanishing moments for the wavelets. For example, DbN is having support width of 2N − 1, filter length (number of filter coefficients) of 2N , regularity about 0.2N for large N and number of vanishing moments is N. Similarly, 3.3 and 3.5 of biorthogonal wavelets indicate some properties like regularities, vanishing moments, filter length etc. The filter coefficients for the decomposition (g as lowpass ˜ as highpass) of and h as highpass) and reconstruction (˜ g as lowpass and h


9

these four wavelets [59] are given in Tables 1 and 2. These wavelets are implemented with the multiresolution scheme given by Mallat [38], which is briefly described below. Table 1. Decomposition and reconstruction filter coefficients for Db3 and Db6 wavelet bases [59] No. of filter coefficients 0 1 2 3 4 5 6 7 8 9 10 11 12

Db3 Decomposition Reconstruction ˜ g h g˜ h 0.0352 -0.3327 0.3327 0.0352 -0.0854 0.8069 0.8069 0.0854 -0.1350 -0.4599 0.4599 -0.1350 0.4599 -0.1350 -0.1350 -0.4599 0.8069 0.0854 -0.0854 0.8069 0.3327 0.0352 0.0352 -0.3327 -

Db6 Decomposition Reconstruction ˜ g h g˜ h -0.0011 -0.1115 0.1115 -0.0011 0.0048 0.4946 0.4946 -0.0048 0.0006 -0.7511 0.7511 0.0006 -0.0316 0.3153 0.3153 0.0316 0.0275 0.2263 -0.2263 0.0275 0.0975 -0.1298 -0.1298 -0.0975 -0.1298 -0.0975 0.0975 -0.1298 -0.2263 0.0275 0.0275 0.2263 0.3153 0.0316 -0.0316 0.3153 0.7511 0.0006 0.0006 -0.7511 0.4946 -0.0048 0.0048 0.4946 0.1115 -0.0011 -0.0011 -0.1115

Table 2. Decomposition and reconstruction filter coefficients for Bior3.3 and Bior3.5 wavelet bases [59] No. of filter Coefficients 0 1 2 3 4 5 6 7 8 9 10 11 12

Bior3.3 Decomposition Reconstruction ˜ g h g˜ h 0.0663 0 0 0.0663 -0.1989 0 0 0.1989 -0.1547 -0.1768 0.1768 -0.1547 0.9944 0.5303 0.5303 -0.9944 0.9944 -0.5303 0.5303 0.9944 -0.1547 0.1768 0.1768 0.1547 -0.1989 0 0 -0.1989 0.0663 0 0 -0.0663 -

Bior3.5 Decomposition Reconstruction ˜ g h g˜ h -0.0138 0 0 -0.0138 0.0414 0 0 -0.0414 0.0525 0 0 0.0525 -0.2679 0 0 0.2679 -0.0718 -0.1768 0.1768 -0.0718 0.9667 0.5303 0.5303 -0.9667 0.9667 -0.5303 0.5303 0.9667 -0.0718 0.1768 0.1768 0.0718 -0.2679 0 0 -0.2679 0.0525 0 0 -0.0525 0.0414 0 0 0.0414 -0.0138 0 0 0.0138

10

S.K Meher et al.

2.1 Discrete WT and multiresolution analysis 1-Dimensional The discrete WT analyzes the signal at different frequency bands with different resolutions by decomposing the signal into low frequency (approximation) and high frequency (details) band information. WT employs two sets of functions, called scaling functions and wavelet functions, which are associated with low pass and highpass filters, respectively. The decomposition of the signal into different frequency bands is obtained by successive highpass and lowpass filtering of the signal. The 1-dimensional (1-D) original signal, (e.g., sn = [snk ]) is decomposed using a highpass filter hk and a lowpass filter gk , where n = 1, 2, ...Q is the number of decomposition levels and k = 1, 2, ...S is the number of signal samples. The decomposed signals consist of two parts sn−1 and dn−1 , called the smooth or lowpass and fine or highpass information. The two decomposed signal can be expressed as sn−1,j =

S

snk .gk−2j ,

(2)

snk .hk−2j

(3)

k=1

dn−1,j =

S k=1

with j as the length of the convolution masks. The reconstruction of the original signal from the decomposed WT coefficients can be performed as ˜ k−2j ]. [sn−1,j .˜ gk−2j + dn−1,j .h (4) snk = j

where g˜k and h˜k are the reconstruction filter coefficients. The convolution operation between two discrete sequences, e.g., y and z, as in Eqs. (2) and (3), can be expressed as CNV = y ∗ z. Here y and z represents the signal and wavelet filter sequences and their convolution operation can be defined as CN V j =

S

yk .zj−k .

(5)

k=1

The notation (↓ 2)y in Fig. 1 denotes the downsampled version of the sequence y by 2, i.e., ((↓ 2)y)k = y2k . Here the odd-numbered WT coefficients are dropped and the even-numbered WT coefficients are renumbered. The notation (↑ 2)y in Fig. 2 denotes the upsampled version of the sequence y by 2, i.e., ((↑ 2)y)2k = yk , ((↑ 2)y)2k+1 = 0.


11

Here a zero is inserted between each pair of adjacent WT coefficients and re-numbered. Thus the decomposition step consisted of two discrete convolution operations as (h ∗ sn )j =

S

snk .h( j − k), and

(6)

k=1

(g ∗ sn )j =

S

snk .g( j − k).

(7)

k=1

where ∗ represents the convolution operation. This convolution operation is followed by downsampling as sn−1 = (↓ 2)(g ∗ sn ), dn−1 = (↓ 2)(h ∗ sn ).

(8) (9)

The reconstruction step consists of upsampling, followed by two convolution operations. The reconstruction of the original signal can be expressed as sn = Reconstructed low f requency part of the signal(RLF P S) + Reconstructed high f requency part of the signal(RHF P S) ˜ ∗ (↑ 2)dn−1 , are ob˜ ∗ (↑ 2)sn−1 and RHF P S= h where, RLF P S = g tained from their corresponding WT coefficients. These reconstructed signals thus summed to represent the original signal. If we use discrete WT, the reconstructed signal may not be exactly the same as original one; however, it will be a fair approximation. Due to these properties, WT is widely used for signal and image compression [64]. Similarly, in case of 2-D image, the reconstructed images are obtained from the corresponding subbands and summed up to approximate the original image. The above description is for one step of discrete WT. The multiple levels of decomposition can be performed as sn → sn−1 , dn−1 sn−1 → sn−2 , dn−2 ... → ...

(10)

The inverse WT for reconstruction is performed similarly, but in the reverse way. Decomposition halves the time resolution since only half of the number of samples now characterizes the entire signal. However, this operation doubles the frequency resolution, since the frequency band of the signal now spans only half the previous frequency band, effectively reducing the uncertainty in the frequency by half. The above procedure, which is also known as the subband coding, can be repeated for further decomposition. At every level, the filtering

12

S.K Meher et al.

and downsampling will result in half the number of samples (and hence half the time resolution) and half the frequency band spanned (and hence double the frequency resolution). This process of subband coding is also known as multiresolution analysis [39, 65]. 2-Dimensional The two-dimensional (2-D) WT is performed by consecutively applying 1-D WT on rows and columns of the 2-D data. The basic idea of the discrete WT for a 2-D image (signal) is described as follows. A 2-D WT, which is a separable filter bank in row and column directions, decomposes an image into four sub-images [38, 39]. Fig. 1 shows this dyadic decomposition of a 2-D image for a 2-level decomposition. H and L in Fig. 1 denote a highpass and lowpass filter, respectively. ↓ 2 denotes the downsampling (decrease the sample occurrence rate) by a factor of two. Thus in one level decomposition of the 2-D WT, i.e., after convolution of the input image with the low and high pass filters followed by downsampling in both row and column, four subsampled versions of the original image are obtained. Among them, one is the WT coefficients in low frequency range called the approximation part, i.e., LL (Fig. 1) and the other three in high frequency range in three directions: vertical (LH), horizontal (HL) and diagonal (HH) called the detail parts. By decomposing the approximation coefficients (LL image) of each level into four sub-images iteratively a pyramidal tree structure is acquired. In this process there is hardly any loss or redundancy of information between the levels, since the dyadic decomposition is an orthogonal representation of the image. From these WT coefficients the corresponding reconstructed images are obtained, which will be considered subsequently as the extracted features of the original image for classification purpose. The number of extracted features depend on the level of decomposition. The reconstruction process is normally called the inverse WT. The block representation of the inverse WT that generates the reconstructed image from different scales is given in Fig. 2. This procedure is extended to reconstruct all the sub-images in different scales. 2.2 Feature generation Since the decomposed WT coefficients at different levels in different subimages represent the information of the original pixel value, we have used these coefficients to construct features. At first different bands of images are decomposed into the desired level (in our experiment we have decomposed the image up to 2nd level since the performance did not improve much with more level of decomposition) using the 2-D WT, which provides four subband images from each band. As a whole 16 subband images can be obtained from a four-band image (original input) after one level of decomposition. It becomes 28 band sub-images with two levels of decomposition and so on. However the pixels of the sub-images are reconstructed to get the image information

Wavelets for Multispectral Image Classification rowwise

L L

rowwise

columnwise

2

columnwise

LL L

2

lowpass

13

L

2

LL

H

2

LH

L

2

HL

H

2

HH

2

lowpass

H

2

LH

L

2

HL

Image

highpass

H

highpass

H

2

H

2

2

HH 2nd level

1st level

Fig. 1. Two-level discrete wavelet transform columnwise

LL

2

rowwise

L 2

LH

2

H

HL

2

L

Reconstructed image

2

HH

2

L

H

H

Fig. 2. Reconstruction of a single band sub-image using inverse WT

from the corresponding subband. In this operation the representation of the original image for a particular band is obtained through the reconstruction process. The sub-images are then cascaded as shown in the Fig. 3 so that the extracted feature vectors of the original multispectral image can be obtained for the next step (classification). Cascading of different bands for generation of feature vector with Q-level of decomposition can be performed as 1 1 1 1 1 1 1 , ILH−Q , IHL−Q , IHH−Q , ..., ILH−1 , IHL−1 , IHH−1 , ..., (ILL−Q B B B B B B B , ILH−Q , IHL−Q , IHH−Q , ..., ILH−1 , IHL−1 , IHH−1 ). ILL−Q

14

S.K Meher et al.

B where, ILH−1 denotes the sub-image at first level for first band with B as the number of spectral bands of the original image. Hence the feature vector of each pattern of the above decomposition will be of length B(3Q + 1). Thus, a two-band multispectral image with three-levels of decomposition creates a feature vector of length 20. Figure 3 shows the cascading of sub-images of a single band image, which can be extended to the desired level of decomposition.

el

Q

v Le

Neural Classifier

Class Map

HH

l1

HL

ve

LH

Le

LL

Remote Sensing Image

HH

Wavelet Transform

HL LH LL

Fig. 3. Classification procedure

3 Classification techniques The above discussed WF based classification scheme has been implemented with six different classification models. These are: maximum likelihood (ML) [50] and k-nearest neighbor (k-NN) [53] from the non-fuzzy domain and fuzzy ML (FML) [54], fuzzy k-NN (Fk-NN) [55], fuzzy product aggregation reasoning rule (FPARR) [56] and fuzzy explicit (FE) [57] from fuzzy domain. A brief description of these methods are as follows. 3.1 Non-fuzzy classifiers Maximum likelihood (ML) The ML [50] is one of the most popular classification methods. It is based on the probabilistic classification procedure, which assumes that each class can be adequately described or modeled by a multivariate probability distribution in the feature space. If we assume that apriori probabilities for the occurrence


15

of all classes are equal then the probability that a measurement vector x (x = [x1 , x2 , ...xd , ...xD ]t ) actually comes from class c, is given by Pc (x) =

pc (x) C pj (x)

(11)

j=1

with j, c = 1, 2, ..., C, where C and D are the total number of classes and dimensions, respectively. The likelihood function pj (x) can be computed as 1 1 t −1 exp − (x − µj ) Σj (x − µj ) , (12) pj (x) = 2 (2π)D/2 |Σj |1/2 with mean (µj ) and covariance matrix (Σj ) for class j are given as 1 xr , R r=1

(13)

1 (xr − µj )(xr − µj )t . R r=1

(14)

R

µj = and R

Σj =

respectively with R as the total number of patterns/pixels in a particular class of the training data sets. The performance of this type of classifier thus depends on how well the data match the pre-defined model. The pattern x is assigned to the class c for which the normalized likelihood functional value (probability value = Pc (x)) is maximum. k-nearest neighbor (k-NN) In the k-NN classification method, no classifier model is built in advance. Basically, k-NN refers back to the raw training data in the classification of each new pixel. Therefore, it can be said that the entire training set is the classifier. The k-NN is based on some pre-selected distance metric (like Euclidian, Manhattan, Mahalanobis, Minkowski distance of some order) and it finds the k most similar or nearest training pixels (of the test pixel) and assign the plurality class of those k pixels to the new sample [53]. The value for k is preselected. Using relatively larger k may include some dissimilar pixels and on the other hand, using very smaller k may exclude some potential candidate pixels. In both the cases classification accuracy will decrease. The optimal value of k depends on the size and nature of the data set. The steps of the k-NN classification process are: • Determine a suitable distance metric. • Find the k nearest neighbors using the selected distance metric. • Find the plurality class of the k-nearest neighbors (voting on the class labels of the nearest neighbors). • Assign that class to the pixel to be classified.

16

S.K Meher et al.

3.2 Fuzzy classifiers Fuzzy maximum likelihood (FML) The FML [54] is a fuzzy evaluation of the conventional maximum likelihood parameters. Accordingly, the mean and variance-covariance matrix estimated using the fuzzy membership values for each pattern are called fuzzy!mean and fuzzy!variance-covariance matrix. The fuzzy mean can be defined as R

µc =

fc (xr )xr

r=1 R

(15) fc (xr )

r=1

where R is the total number of patterns in a particular class, fc is the membership function of class c, and xr is the rth pattern. The fuzzy variancecovariance matrix can be defined as R

Σc =

fc (xr )(xr − µc )(xr − µc )t

r=1 R

.

(16)

fc (xr )

r=1

The fuzzy mean and fuzzy variance-covariance matrix can be considered as extensions of the conventional mean and covariance matrix with fc (x) = 1 for class c to which the pattern belongs, and 0 for other classes, Eqs. (15) and (16) then correspond to the conventional mean and covariance matrix. The fuzzy partition matrix is then evaluated using the fuzzy mean and fuzzy variance-covariance matrix. Thus the membership function for class c of a pattern x can be expressed as fc (x) =

pc (x) , C pj (x)

(17)

j=1

where the fuzzy likelihood function (pj (x)) with fuzzy mean (µc ) and fuzzy covariance matrix (Σc ) can be computed as in Eq. (12). For the estimation of the membership function in FML, the parameters mean and covariance matrix require the fuzzy representation of the pattern to different classes. However, Chen [66] described a suitable method, which estimates the fuzzy representation in an iterative manner and does not require the prior information regarding the land covers in a remote sensing image. First it initializes the fuzzy representation as random or crisp. For crisp case


17

the representation value is 1 or 0 for either belongs or does not belongs to a class. Using this representation the fuzzy mean and covariance matrix is estimated. These parameters then estimate the new fuzzy matrix. This process repeats until a stable mean and covariance matrix are calculated. The fuzzy matrix thus estimated can be used for testing the unlabeled patterns. In the present chapter we have used this iterative procedure for the generation of fuzzy matrix that provides a fuzzy representation of belonging of the pixels to different classes. Fuzzy k-nearest neighbor (Fk-NN) The k-NN as discussed above is a non parametric pattern classification method. The classification principle is based on the determination of k-nearest neighbors of a test pattern and allocate the class map that has the majority of neighbors belonging to a particular class. Keller et al. [55] incorporated the concepts of fuzzy set theory into the k-NN voting procedure and proposed a fuzzy version of k-NN rule. The translation to fuzzy logic requires two modifications: the introduction of distance from the test pattern to the centroid of classes of all neighbors to measure the membership values, and the introduction of a membership vector for each pattern. The membership degree of ˆ to class c is calculated by a test pattern x k

1 µij ˆ x − xj 2/(mf −1) j=1 x) = µi (ˆ k 1 ˆ x − xj 2/(mf −1) j=1

(18)

where i = 1, 2, ...C, and j = 1, 2, ..., k, with C number of classes and k number of nearest neighbors. µij is the membership degree of the pattern xˆj from ˆ . For this the training set to the class i, among the k-nearest neighbors of x particular study, the algorithm was implemented with mf equal to 2 (selected on the basis of performance). Fuzzy product aggregation reasoning rule (FPARR) The FPARR based classification method uses three steps [56] as shown in Fig. 4. In the first step, it takes the input feature vector and fuzzifies the feature values. This step uses a π-type membership function (MF) [67] to get the degree of belonging of a pattern into different classes based on different features. The membership matrix fd,c (xd ) thus generated expresses the degree of belonging of different features (D) to different classes (C), where xd is the dth feature of pattern x with d = 1, 2, ...D and c = 1, 2, ...C. A pattern is thus defined as

18

S.K Meher et al.

FUZZYFICATION

Pixel vector

PROD FUZZY RR

Fuzzy inputs

DEFUZZIFICATION

Fuzzy outputs

Hard output

Fig. 4. Block diagram of the proposed fuzzy classification model

x = [x1 , x2 , ...xd , ...xD ]T

(19)

with D as the dimension of the pattern. Thus membership value estimated for feature d in class c for pattern x is fd,c (xd ) = πd,c (xd ), where the π-type MF is given by π(x; a, r, b) = 0, = 2m−1 [(x − a)/(r − a)]m , = 1 − 2m−1 [(r − x)/(r − a)]m , = 2m−1 [(x − r)/(b − r)]m , = 1 − 2m−1 [(b − x)/(b − r)]m , = 0,

x≤a a<x≤p p<x≤r r<x≤q q<x βF k−N N > βk−N N , βtraining > βF M L > βM L and βtraining > βF P ARR > βF E > βF M L ≡ βF k−N N . From these relations we can get the performance quality of the different classification methods. Another important relation with all the classifiers in the WF domain i.e., βBior3.3 >βDb3 >βBior3.5 ≡βDb6 is also evident from the Tables 3 and 4. Table 3. Comparison of β value for different WF based non-fuzzy classification methods Classification methods Training patterns ML k-NN WT with ML WT with ML WT with ML WT with ML WT with k-NN (k=8) WT with k-NN (k=8) WT with k-NN (k=8) WT with k-NN (k=8)

wavelets Db3 Db6 Bior3.3 Bior3.5 Db3 Db6 Bior3.3 Bior3.5

IRS Cal IRS Bom SPOT Cal 9.4212 21.4783 9.3343 6.8931 16.8967 6.8712 6.8732 16.8421 6.8789 7.3835 17.3834 7.3135 6.9132 16.9893 6.9353 7.5131 17.5432 7.2012 7.0978 17.0012 7.0832 7.4387 17.4571 7.3720 7.0313 17.1101 6.8789 7.5891 17.5001 7.4301 7.1342 17.0813 7.0911

5.3 Classification of IRS-1A Bombay image As in the classification of IRS-1A Calcutta image, the classified images of IRS1A Bombay (Fig. 6b) using Fk-NN and FPARR based classification methods and their WF based methods with Bior3.3 wavelet are shown in Figs. 9a and 10a, and Figs. 9b and 10b, respectively. It can be seen from Figs 9b and 10b that the Dockyard, Butcher Island, Elephanta Cave, Elephanta Island and Santa Cruz Airport are very well detected. This detection can also be made

26

S.K Meher et al.

Table 4. Comparison of β value for different WF based fuzzy classification methods Classification methods Training patterns FPARR FE FML Fk-NN WT with FPARR WT with FPARR WT with FPARR WT with FPARR WT with FE WT with FE WT with FE WT with FE WT with FML WT with FML WT with FML WT with FML WT with Fk-NN (k=8) WT with Fk-NN (k=8) WT with Fk-NN (k=8) WT with Fk-NN (k=8)

wavelets Db3 Db6 Bior3.3 Bior3.5 Db3 Db6 Bior3.3 Bior3.5 Db3 Db6 Bior3.3 Bior3.5 Db3 Db6 Bior3.3 Bior3.5

IRS Cal 9.4212 8.1717 7.1312 7.0011 7.0121 8.6348 8.1913 8.7413 8.2012 7.7017 7.1934 7.7918 7.1997 7.5431 7.0675 7.6312 7.1803 7.5412 7.1675 7.6589 7.2113

IRS Bom SPOT Cal 21.4783 9.3343 19.4531 8.1078 17.6283 7.0137 17.0287 6.9456 17.0013 6.9212 20.0134 8.7315 19.5012 8.1101 20.1017 8.7411 19.5102 8.1210 18.3312 7.6513 17.7131 7.1345 18.4013 7.7022 17.7343 7.2123 17.7174 7.3647 17.1232 7.0236 17.7532 7.3987 17.1131 7.2652 17.6070 7.4012 17.2010 7.0021 17.6431 7.4902 17.1341 7.1011

Santa Cruz airport

Dockyard

(a)

Butcher island

Elephanta cave

Elephanta island

(b)

Fig. 9. Classified IRS-1A Bombay image by (a) Fk-NN, and (b) WT (Bior3.3) + Fk-NN classification method


(a)

27

(b)

Fig. 10. Classified IRS-1A Bombay image by (a) FPARR, and (b) WT (Bior3.3) + FPARR classification method

with the Fk-NN and FPARR classification methods. However a better crisp and homogeneous behavior is observed in the WF based classification method compared to the original spectral feature only. This improvement is still better in case of FPARR and further enhanced in case of WF based FPARR based classification methods which can be seen from Figs. 10a and 10b, respectively. The classified images obtained from the WF based ML, k-NN, FML and FE and individual classifiers (original spectral features only) for all wavelets are not shown in figures. However their β values are shown in Tables 3 and 4. Thus it can be summarized that the two classified images in Figs. 9a and 9b showed some visual differences and this difference are still more clear in case of FPARR and WF based FPARR classification methods which are shown in Figs. 10a and 10b, respectively. However a proper comparison can be made from β value, which supports the classification results. Tables 3 and 4 depict the β values for different classification methods. β for the Bombay image with training data is found to be 21.4783, which is 16.8967, 16.8421, 19.4531, 17.6283, 17.0287 and 17.0013 for the ML, k-NN, FPARR, FE, FML and FkNN classifiers, respectively. The β value increment for WF based classifiers in all cases are observed from Tables 3 and 4. It is seen that the same relation on β values are maintained for IRS-1A Bombay image as in the case of Calcutta image. 5.4 Classification of SPOT Calcutta image In case of SPOT Calcutta image, the classified images are shown in Fig. 11a for Fk-NN (original spectral feature only) and Fig. 11b for the WF based FkNN classifiers with Bior3.3 wavelet only. Similarly the classified images with

28

S.K Meher et al. Howrah bridge

Garden reach lake

(a)

Khidirpore dock

Talis nala

Beleghata canal

Race course

(b)

Fig. 11. Classified SPOT Calcutta image by (a) Fk-NN, and (b) WT (Bior3.3) + Fk-NN classification method Howrah bridge

Garden Reach Lake

(a)

Khidirpore Dock

Talis Nala

Beleghata canal

Race Course

(b)

Fig. 12. Classified SPOT Calcutta image by (a) FPARR, and (b) WT (Bior3.3) + FPARR classification method

FPARR and its WF based classification with Bior3.3 wavelet are shown in Figs. 12a and 12b, respectively. From the figures it is observed that there is a clear separation of different classes and some known regions like race course, Howrah Setu(Bridge), Talis nala(Canal), Beleghata canal, Khiderpore dock and Garden Reach Lake. WF based classifier is clearly identifying (Fig. 11b) the above regions compared to those generated by Fk-NN only (Fig. 11a). As in case of IRS-1A Calcutta and Bombay images, here also the identification of


29

different known regions in the classified images are more distinct with FPARR and its WF based classification methods. However, a better performance comparison with the help of β value can be seen from Tables 3 and 4. The β value for the training data set is 9.3343. Its values are 6.8712, 6.8789, 8.1078, 7.0137, 6.9456 and 6.9212 for the ML, k-NN, FPARR, FE, FML and Fk-NN classifiers, respectively. In this case also the improved performance of the WF based classifiers over their original versions are observed and is still better with Bior3.3 wavelet based classification schemes. The β relation in the classification of SPOT Calcutta image is also seen to corroborated the case of IRS-1A Calcutta and IRS-1A Bombay images. From the land cover classification of three remote sensing images, we found that the WF based classification method is superior compared to the corresponding individual classification methods (original spectral feature only). Performance comparison among the six classifiers with its WF based version have been made using a quantitative index (QI). The value of QI supports the superiority of the proposed WF based classification methods over the individual original feature based methods. It is also seen that the Bior3.3 wavelet based classification method is outperforming the other wavelets. Further the improvement of performance with Bior3.3 wavelet and FPARR and FE based classification methods are more prominent, where the β value is close to the training data set compared to the other combination methods. Thus, it is justified that the classified regions using the WF based classification scheme are more crisp and homogeneous compared to their respective original versions as tested in the present case.

6 Conclusion In the present chapter we showed an application of soft computing to remotely sensed image classification. We discussed a wavelet feature based classification approach for land cover classification of multispectral remotely sensed images. The scheme basically tried to explore the possible advantages of using WT as a preprocessor for both non-fuzzy and fuzzy classifiers. In this experiment, we have used two non-fuzzy and four fuzzy classifiers for the performance evaluation. The WT in this case is used to extract features from the original patterns. The extracted features acquire contextual information both in spatial and spectral domains and thus increases classification accuracy. We have used a two-level decomposition of WT, as complexity increases proportionally with the level of decomposition and with an insignificant increase in performance. The improvement in performance of the WF based classification scheme is verified from the results obtained from three remote sensing images. The β measure provide the classification accuracy as shown in Tables 3 and 4. Further, different wavelets are being used in the preprocessing stage of the proposed scheme. Classification results are provided for four wavelets from

30

S.K Meher et al.

two wavelet groups (namely, Daubechies and Biorthogonal). It is observed that WF based approach provided better classification results than any of the corresponding individual classification methods for these remote sensing images. Among the different wavelets, the Bior3.3 wavelet performed better than others. It is observed that the β value for the WF based FPARR and FE classification with Bior3.3 wavelet is close to that of the training data set compared to the others for all the three remote sensing images. Thus, the performance of the FPARR and FE methods convey the information of classification efficiency of fuzzy methods over other methods for these remote sensing images. Hence, it can be concluded that the WF based classification scheme with Bior3.3 wavelet is superior than others in classifying the aforementioned remote sensing images. The present scheme of classification of land covers in remotely sensed imagery can be compared with M-band wavelets [70]; and a comparative study of the wavelet feature based methods with statistical feature (obtained from the gray level co-occurrence matrices) based methods may also be carried out. Other quantitative measures for evaluating the goodness of the classifiers may also be tried. Effectiveness of wavelet features will be tested using other classifiers like neural networks and support vector machines.

Acknowledgements The authors would like to thank the reviewers for their valuable and constructive suggestions. Thanks are also due to the Department of Science and Technology, Government of India and University of Trento, Italy, the sponsors of the India - Trento Program on Advanced Research (Ref: ITALY/ITPAR (IITB, ISI, JU)/2003 dated 22/01/2004), under which a project titled “Advanced Techniques for Remote Sensing Image Processing” is being carried out at the Machine Intelligence Unit, Indian Statistical Institute, Calcutta.

References 1. Tso B., Mather P. M (2001) Classification Methods for Remotely Sensed Data. Taylor and Francis, London 2. Chang C.-I. (2001) Hyperspectral Imaging: Techniques for Spectral Detection and Classification. Kluwer Academic Publishers 3. Varshney P. K., Arora M. K. (2004) Advanced Image Processing Techniques for Remote Sensed Hyperspectral Data. Springer-Verlag, Germany 4. Haralick R. M., Shanmugam K., Dinstein I. (1973) Texture features for image classification. IEEE Transactions on Systems, Man and Cybernetics, 8(6), pp. 610–621 5. Tuominen S., Pekkarinen. A (2005) Performance of different spectral and textural aerial photograph features in multi-source forest inventory. Remote Sensing of Environment, 94(2), pp. 256–268


31

6. Cross G. R., Jain A. K. (1983) Markov random field texture models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 5(1), pp. 25–39 7. Chellappa R., Chatterjee S. (1986) Classification of textures using gaussian markov random fields. IEEE Transaction on Acoustics Speech Signal Processing, 33(4), pp. 959–963 8. Cohen F. S., Fan Z., Patel M. A. (1991) Classification of rotation and scaled textured images using gaussian markov random field models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(2), pp. 192–202 9. Derin H., Elliot H. (1987) Modeling and segmentation of noisy and textured images using gibbs random fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 9(1), pp. 39–55 10. Unser M. (1986) Local linear transforms for texture measurements. Signal Processing, 11, pp. 61–79 11. Unser M. (1995) Texture classification and segmentation using wavelet frames. IEEE Transactions on Image Processing, 4(11), pp. 1549–1560 12. Park S. C., Park M. K., Kang M. G. (2003) Super-resolution image reconstruction: a technical overview. IEEE Signal Processing Magazine, 20(3), pp. 21–36 13. Ng M. K., Bose N. K. (2003) Mathematical analysis of super-resolution methodology. IEEE Signal Processing Magazine, 20(3), pp. 62–74 14. Rajan D., Chaudhuri S., Joshi M. V. (2003) Multi-objective super resolution: concepts and examples. IEEE Signal Processing Magazine, 20(3), pp. 49–61 15. Keshava N., Mustard J. F. (2002) Spectral unmixing. IEEE, Signal Processing Magazine, 19(1), pp. 44–57. 16. Brown M., Gunn S. R., Lewis H. G. (1999) Support vector machines for optimal classification and spectral unmixing. Ecological Modelling, 120, pp. 167–179 17. Chang C. I., Ren H. (2000) An experiment-based quantitative and comparative analysis of target detection and image classification algorithms for hyperspectral imagery. IEEE Transactions on Geoscience and Remote Sensing, 38(2), pp. 1044–1063 18. Faraklioti M., Petrou M. (2001) Illumination invariant unmixing of sets of mixed pixels. IEEE Transactions on Geoscience and Remote Sensing, 39(10), pp. 2227–2234 19. Shah C. A., Varshney P. K. (2004) A higher order statistical approach to spectral unmixing of remote sensing imagery. Proceedings of the IEEE International Symposium on Geoscience and Remote Sensing, volume 39, pp. 1065–1068 20. Parra L., Mueller K. R., Spence C., Ziehe A., Sajda P. (2000) Unmixing hyperspectral data. Advances in Neural Information Processing Systems, volume 12, MIT Press 21. Robila S. A. (2002) Independent component analysis feature extraction for hyperspectral images. PhD thesis, EECS Department. Syracuse University, Syracuse, NY 22. Chang C. I., Chiang S.-S., Smith J. A., Ginsberg I. W. (2002) Linear spectral random mixture analysis for hyperspectral imagery. IEEE Transactions on Geoscience and Remote Sensing, 40(2), pp. 375–392 23. Burke D. P., Kelly S. P., Chazal P. de, Reilly R. B., Finucane C. (2005) A parametric feature extraction and classification strategy for brain-computer interfacing. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 13(1), pp. 12–17

32

S.K Meher et al.

24. Song C. (2005) Spectral mixture analysis for subpixel vegetation fractions in the urban environment How to incorporate endmember variability? Remote Sensing of Environment, 95, pp. 248–263 25. Plaza A., Martinez P., Perez R., Plaza J. (2002) Spatial/spectral endmember extraction by multidimensional morphological operations. IEEE Transactions on Geoscience and Remote Sensing, 40(9), pp. 2025–2041 26. Barducci A., Mecocci A., Alamanno C., Pippi I., Marcoionni P. (2003) Multiresolution least-squares spectral unmixing algorithm for subpixel classification of hyperspectral images. Proceedings of the IEEE International Symposium on Geoscience and Remote Sensing, volume 3, pp. 1799–1801 27. Broadwater J., Meth R., Chellappa R. (2004) A hybrid algorithm for subpixel detection in hyperspectral imagery. Proceedings of the IEEE International Symposium on Geoscience and Remote Sensing, volume 3, pp. 1601–1604 28. Tatem A. J., Lewisa H. G., Atkinsonb P. M., Nixona M. S. (2002) Superresolution land cover pattern prediction using a hopfield neural network. Remote Sensing of Environment, 79, pp. 1–14 29. Weiguo L., Seto K. C., Wu E. Y., Gopal S., Woodcock C. E. (2004) ART-MMAP: A neural network approach to subpixel classification. IEEE Transactions on Geoscience and Remote Sensing, 42(9), pp. 1976–1983 30. Pastina D., Farina A., Gunning J., Lombardo P. (1998) Classification of rotation and scaled textured images using gaussian markov random field models. IEE Proceedings Radar, Sonar Navigation, 145(5), pp. 281–290 31. Jiang N., Wu R., Li J. (2001) Super resolution feature extraction of moving targets. IEEE transactions on Aerospace and Electronic Systems, 37(3), pp. 781–793 32. Foody G. M., Muslim A. M., Atkinson P. M. (2003) Super-resolution mapping of the shoreline through soft classification analyses. Proceedings of the IEEE International Symposium on Geoscience and Remote Sensing, volume 6, pp. 3429–3431 33. Qing K., Yu R., Li X., Deng X. (2005) Application of spectral angle mapping model to rapid assessment of soil salinization in arid area. Proceedings of the IEEE International Symposium on Geoscience and Remote Sensing, volume 4, pp. 2355–2357 34. Shih E. H. H., Schowengerdt R. A. (1983) Classification of arid geomorphic surfaces using spectral and textural features. Photogrammetric Engineeing and Remote Sensing, 49(3), pp. 337–347, 1983 35. Yuan X., King D., Vlcek J. (1991) Sugar maple decline assessment based on spectral and textural analysis of multispectral aerial videography. Remote Sensing of Environment, 37(1), pp. 47–54 36. Chica-Olmo M., Abarca-Hern´ andez F. (2000) Computing geostatistical image texture for remotely sensed data classification. Computers and Geosciences, 26(4), pp. 373–383 37. Jouan A., Allard Y. (2004) Land use mapping with evidential fusion of features extracted from polarimetric synthetic aperture radar and hyperspectral imagery. Information Fusion, 5(4), pp. 251–267 38. Mallat S. (1999) A Wavelet Tour of Signal Processing. Academic Press, 2nd edition 39. Strang G., Nguyen T. (1996) Wavelets and Filter Banks. Wellesley College


33

40. Chang T., Kuo C. C. J. (1993) Texture analysis and classification with treestructured wavelet transform. IEEE Transactions on Image Processing, 2(4), pp. 429–440 41. Unser M., Eden M. (1989) Multiresolution feature extraction and selection for texture segmentation. IEEE Transactions on Pattern Analysis Machine Intelligence, 2, pp. 717–728 42. De Wouwer G. V., Schenders P., Dyek D. V. (1999) Statistical texture characterization from discrete wavelet representation. IEEE Transactions on Image Processing, 8(4), pp. 592–598 43. Abbate A., Das P., Decusatis C. M. (2001) Wavelets and Subbands: Fundamentals and Applications. Springer 44. Arivazhagan S., Ganesan L. (2003) Texture classification using wavelet transform. Pattern Recognition Letters, 24, pp. 1513–1521 45. Szu H. H., Moigne J. L., Netanyahu N. S., Hsu C. C. (1997) Integration of local texture information in the automatic classification of landsat images. SPIE Proceedings, 3078, pp. 116–127 46. Yu J., Ekstrom M. (2003) Multispectral image classification using wavelets: A simulation study. Pattern Recognition, 36, pp. 889–898 47. Alpert B. K. (1992) Wavelets and other bases for fast numerical linear algebra. In Wavelets: A Tutorial in Theory and Applications, C. K. Chui, Ed. New York: Academic Press, pp. 181–216 48. Strela V., Heller P. N., Strang G., Topiwala P., Heil C. (1999) The application of multiwavelet filter banks to image processing. IEEE Transactions on Image Processing, 8(4), pp. 548–563 49. Soltanian-Zadeh H., Rafiee-Rad F., Pourabdollah-Nejad D. S. (2004) Comparison of multiwavelet, wavelet, haralick, and shape features for microcalcification classification in mammograms. Pattern Recognition, 37, pp. 1973–1986 50. Duda R. O., Hart P. E., Stork D. G. (2000) Pattern Classification. Wiley Interscience Publications, 2nd edition 51. Pedrycz W. (1990) Fuzzy sets in pattern recognition: methodology and methods. Pattern Recognition, 23(1/2), pp. 121–146 52. Kuncheva L. I. (2000) Fuzzy Classifier Design. Springer-Verlag 53. Cover T. M., Hart P. E. (1967) Nearest neighbor pattern classification. IEEE Transactions on Information Theory, 13(1), pp. 21–27 54. Wang F. (1990) Fuzzy supervised classification of remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 28(2), pp. 194–201 55. Keller J. M., Gray M. R., Givens J. A. (1985) A fuzzy k-nearest neighbor algorithm. IEEE Transactions on Systems, Man and Cybernetics, 15(4), pp. 580–585 56. Ghosh A., Meher S. K., Shankar B. U. (2005) Fuzzy supervised classification using aggregation of features. Technical Report, Indian Statistical Institute. No. MIU/TR-02/2005 57. Melgani F., AI Hashemy B. A. R., Taha S. M. R. (2000) An explicit fuzzy supervised classification method for multispectral remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 38(1), pp. 287–295 58. Richards J. A., Jia X. (1999) Remote Sensing Digital Image Analysis: An Introduction. New York: Springer Verlag, 3rd edition 59. Daubechies I. (1995) Ten Lectures on Wavelets. Society for Industrial and Applied Mathematics 60. Liu H., Motoda H. (1998) Feature Extraction, Construction and Selection: A Data Mining Perspective. Kluwer Academic Publishers, Norwell, MA, USA

34

S.K Meher et al.

61. Mazzaferri J., Ledesma S., Iemmi C. (2003) Multiple feature extraction by using simultaneous wavelet transforms. Journal of Optics, A Pure and Applied Physics, 5, pp. 425–431 62. Soltani S., Simard P., Boichu D. (2004) Estimation of the self-similarity parameter using the wavelet transform. Signal Processing, 84(1), pp. 117–123 63. Mujica F. A., Leduc J. P., Murenzi R., Smith M. J. T. (2000) A new motion parameter estimation algorithm based on the continuous wavelet transform. IEEE Transactions on Image Processing, 9(5), pp. 873–888 64. Topiwala P. K. (2000) Wavelet Image and Video Compression. Springer 65. Vetterli M., Kovacevic J. (1995) Wavelets and Subband Coding. Prentice Hall, 1st edition 66. Chen C. F. (1999) Fuzzy training data for fuzzy supervised classification of remotely sensed images. Asian Conference on Remote Sening (ACRS 1999) 67. Zadeh L. A. (1965) Fuzzy sets. Information Control, 8, pp. 338–353 68. Cohen J. (1960) A coefficient of aggrement for nominal scale. Education and psychological measurement, 20(1), pp. 37–46 69. Pal S. K., Ghosh A., Shankar B. U. (2000) Segmentation of remotely sensed images with fuzzy thresholding, and quatitative evaluation. International Jounal of Remote Sensing, 21(11), pp. 2269–2300 70. Acharyya M., De R. K., Kundu M. K. (2003) Segmentation of remotely sensed images using wavelet features and their evaluation in soft computing framework. IEEE Transactions on Geoscience and Remote Sensing, 41(12), pp. 2900–2905 71. Mitra P., Shankar B. U., Pal S. K. (2004) Segmentation of multispectral remote sensing images using active support vector machines. Pattern Recognition Letters, 25(12), pp. 1067–1074 72. IRS data users hand book (1989) Technical Report. Document No. IRS/NRSA/NDC/HB-02/89

Chapter 2 Images with Uncertainty: Efficient Algorithms for Shift, Rotation, Scaling, and Registration, and Their Applications to Geosciences Cara G. Schiek1 , Roberto Araiza1 , Jose Manuel Hurtado1 , Aaron A. Velasco1 , Vladik Kreinovich1 , and Victor Sinyansky2 1

2

University of Texas at El Paso, USA [email protected], [email protected], [email protected], [email protected], [email protected] (contact author) Raytheon Victor [email protected]

1 General motivation and goals of the chapter In geosciences, we often need to combine two (or more) images of the same area: • different images bring different information; so, to get a better understanding, we must fuse the corresponding data; e.g., we must combine a satellite image with a radar image; • comparison of two images – e.g., images made at different moments of time – can also give us information about the changes: e.g., by comparing preand post-earthquake images, we can determine the effect of the earthquake. Compared images are often obtained from slightly different angles, from slightly different positions. Therefore, in order to compare these images, we must first register them, i.e., find the shift, rotation, and scaling after which these images match as much as possible, and then apply these transformations to the original images. There exist efficient algorithms for registration and for the corresponding transformations. However, these algorithms are only effective when we know these images with high accuracy. In many real-life situations – e.g., when comparing pre- and post-earthquake images – the accuracy with which we know these images is of the same order of magnitude as the desired difference between the compared images. In this chapter, we describe how the existing image registration algorithms can be extended to such low-accuracy (high uncertainty) situations. Specifically, we describe a new algorithm, and we show how this new algorithm can be efficiently applied in geosciences.

C.G. Schiek et al.: Images with Uncertainty: Efficient Algorithms for Shift, Rotation, Scaling, and Registration, and Their Applications to Geosciences, StudFuzz 210, 35–64 (2007) c Springer-Verlag Berlin Heidelberg 2007 www.springerlink.com

36

C.G. Schiek et al.

2 Structure of the chapter Since our new algorithm is an extension of the existing image registration algorithms, we first briefly overview these existing algorithms, with a special emphasis on image registration algorithms that we will be using to process low accuracy images. These existing algorithms are described in Section 3. In Section 4, we provide an example of low accuracy geoscience-related images, an example for which the existing image registration algorithms (originally designed for high accuracy images) do not work. In Section 5, we explain how to modify the existing image registration algorithm so that it can be applicable to low accuracy images as well. The results of applying the new algorithm to the selected images are described in Section 6. Finally, Section 7 contains conclusions and future work.

3 Registering images known with high accuracy Motivation In order to adequately process satellite and radar information, it is necessary to find the exact correspondence between different types of images and between these images and the existing maps. In other words, we need to register these images. There exist several efficient algorithms for image registration; see, e.g., [5, 33] and references therein. Most of these algorithms work well for images known with high accuracy. The main objective of this chapter is to extend these algorithms to images which are only known with low accuracy. To describe the needed extensions, let us first provide a brief overview of the existing algorithms for registering high-accuracy images, with a special emphasis on algorithms based on the ideas that can be extended to low accuracy images as well. Image registration algorithms: brief overview There exist many methods for image registration. Among the most widely used methods are methods of point matching, where we find the matching points in the two images, and then the most appropriate transformation (rotation and/or shift) which maps the points from one image into the corresponding points from the other image. Point matching methods work well when the images have clearly identifiable matching points, and when we know the images with a high accuracy – so that we can identify and match these matching points with a reasonable accuracy. For example, in satellite images, we often have clear matching points representing special landmarks such as landmark city areas, landmark bridges, or tips of peninsulas. Such landmarks can usually be easily found

Images with Uncertainty: Registration, Applications to Geosciences

37

in highly populated areas or in special terrains in which the area is highly non-homogeneous: e.g., there may be a clear shore line with a clear landmark point, or there may be a large clearly distinguishable river with a clear landmark turn. However, there are many homogenous areas where it is not easy to find landmarks. For example, in the desert areas of the U.S. Southwest, the only visible landmarks are road intersections, and there are usually several similarlooking road intersections in the same image, so it is difficult to find the matching points between the two images. Similarly, in the mountain areas, there many landmarks like summits and ridges, but usually, there are several similar-looking summits and ridges in each image, so it is difficult to match points in the two images. For images known with very low accuracy, we may still find landmarks. However, since we only know the images with a very low accuracy, we may only be able to locate these landmarks with a very low accuracy, too low to enable us to adequately register the two images. Sometimes, instead of landmark points, we have landmark features. For example, we may not have a landmark bridge, but we may have a clearly distinguishable river. In such situations, instead of matching points, we can match features. Such feature-matching algorithms are also efficiently used in image registration. However, in homogenous terrains and/or in situations when we only know the images with low accuracy, we may only be able to locate these features with a very low accuracy, too low to enable us to adequately register the two images. In some cases, e.g., in many astronomical images, we have an image surrounded by an empty space. In this case, even when we cannot find the landmark points in the two images, we can match these images by comparing, e.g., the centers of gravity of these images. Alas, this is not the case in images like satellite images or radar images. As an example of low accuracy images for which registration is practically important, we will actually consider low accuracy satellite images. So, in order to come up with an algorithm for registering low accuracy images, an algorithm which should be applicable for satellite images, we must place our emphasis on image registration techniques which go beyond point matching, feature matching, or simple geometric transformations in the image domain. We will see that many such algorithms are based on the use of the Fast Fourier Transform (FFT). Before we start describing these methods, let us provide a motivation for using FFT in image registration. Why Fourier-based methods in image registration Let us consider the simplest case when the two images have the same orientation, and they differ only by shift. Each image is naturally described as a function I(x) which assigns to every pixel x = (x1 , x2 ) the intensity I(x) at

38

C.G. Schiek et al.

this pixel. The fact that the two images I(x) and I (x) differ only by shift means that I (x) ≈ I(x + a) for some (unknown) shift a = (a1 , a2 ). In this case, the problem is as follows: we have two images I(x) and I (x), and we must find the shift a for which the image I (x) will be, in some reasonable sense, the closest to the shifted image I(x+a). A reasonable way to describe the closeness between the two images is to require that for every pixel x, the corresponding intensities are close to each other. We can use, e.g., the squared difference (I(x + a)) − I (x))2 between these values as the measure of the similarity at x, and we can use the sum (integral) (I(x+a)−I (x))2 dx of these square differences over all locations x as the measure of overall similarity between the two images. For two similarly oriented 2-dimensional images I(x) and I (x), the problem of finding the shift a which minimizes the above integral takes the following form: find a for which the integral (I (x) − I(x + a))2 dx attains the smallest possible value. By representing the square of the difference (I − I)2 as the sum of three terms (I )2 + I 2 − 2 · I · I, we can represent the above scoring function as 2 2 (I (x)) dx + I(x + a) dx − 2 I (x) · I(x + a) dx. The first integral in the sum does not depend on the shift at all. By using the y = x + a, we can show that the second integral is equal to new coordinates I(x)2 dx and thus, also does not depend on the shift. So, finding the shift for which the sum is the smallest possible is equivalent to finding the shift for which the cross-correlation term I (x) · I(x + a) dx attains the largest possible value. For images defined on an n × n grid, a straightforward approach would require that we compute the value of the scoring function for all n2 shifts a. Computing each integral requires time O(n2 ), so overall, we need time O(n2 ) · O(n2 ) = O(n4 ). This computation can be performed much faster if we take into account that the cross-correlation term is a convolution between the images I(x) and I (x). Convolution is one of the main techniques in signal processing, and it is well known that we can compute convolution faster (in time O(n2 · log(n))) by using Fast Fourier Transform (FFT); see, e.g., [7]. Specifically, to compute the convolution, we need the following steps: • first, we apply FFT to the original images, resulting in functions F (ω) and F (ω); • then, for each frequency ω, we compute the product R(ω) = F (ω) · (F )∗ (ω) def

(where F ∗ means complex conjugation); • finally, we apply the inverse Fourier transform to the resulting function R(ω), and get the desired auto-correlation function.


39

We can now find the shift as the vector a for which the cross-correlation attains the largest possible value. FFT of an image of size N requires N ·log(N ) steps, so for a 2-dimensional image of size N = n2 , we need O(n2 · log(n)) steps. Multiplication of the two Fourier transforms and the final search for the largest value both require processing each pixel once, so both require time O(n2 ). As a result, we can find the desired shift a in time O(n2 · log(n)) + O(n2 ) = O(n2 · log(n)). The existing FFT-based registration algorithms The main purpose of the above simple algorithm is to explain why FFT is used in image registration. This simple algorithm only detects the shift, and even for shift estimation, it is not the best possible algorithm; see, e.g., [5, 33]. The best of known FFT-based registration algorithms are presented in [3, 4, 5, 11, 12, 13, 21, 24, 25, 30, 31, 33]. The main ideas behind these FFT-based image registration algorithms are as follows. The simplest case: shift detection in the absence of noise Let us first consider the above case when two images differ only by shift. It is known that if two images I(x) and I (x) differ only by shift, i.e., if I (x) = I(x + a) for some (unknown) shift a = (a1 , a2 ), then their Fourier transforms 1 · I(x) · e−2π·i·(x·ω) dx1 dx2 , F (ω) = 2π 1 · I (x) · e−2π·i·(x·ω) dx1 dx2 , F (ω) = 2π def √ def where i = −1 and (x · ω) = x1 · ω1 + x2 · ω2 , are related by the following formula: (1) F (ω) = e2π·i·(ω·a) · F (ω). Indeed, since I (x) = I(x + a), the Fourier transform F (ω) of the image I (x) takes the form 1 · I(x + a) · e−2π·i·(x·ω) dx, F (ω) = 2π def

where we denoted dx = dx1 dx2 . def We can simplify this expression if we introduce a new vector variable y = x + a, so that x = y − a. Here, dx = dy, so 1 · I(y) · e−2π·i·((y−a)·ω) dy. F (ω) = 2π Here, ((y − a) · ω) = (y · ω) − (a · ω), hence e−2π·i·((y−a)·ω) = e−2π·i·(y·ω) · e2π·i·(a·ω) .

40

C.G. Schiek et al.

The second factor does not depend on y, so we can move it outside the integral and conclude that 1 2π·i·(a·ω) −2π·i·(y·ω) F (ω) = e · I(y) · e · dy . 2π The expression in parentheses is exactly F (ω), so indeed the formula (1) is true. It is known that the magnitude |e2π·i·(ω·a) | (also known as the modulus or the absolute value) of the complex value e2π·i·(ω·a) = cos(2π · (ω · a)) + i · sin(2π · (ω · a)) is equal to 1. Therefore, if the images are indeed obtained from each other by shift, then their Fourier transforms have the same magnitude: M (ω) = M (ω),

(2)

where we denoted M (ω) = |F (ω)|,

M (ω) = |F (ω)|.

(3)

The actual value of the shift a can be obtained if we use the formula (1) to compute the value of the following ratio: R0 (ω) =

F (ω) . F (ω)

(4)

Substituting (1) into (4), we get R0 (ω) = e2π·i·(ω·a) .

(5)

Therefore, the inverse Fourier transform P0 (x) of this ratio is equal to the delta-function δ(x − a). In other words, in the ideal no-noise situation, this inverse Fourier transform P0 (x) is equal to 0 everywhere except for the point x = a; so, from P0 (x), we can easily determine the desired shift by using the following algorithm: • first, we apply FFT to the original images I(x) and I (x) and compute their Fourier transforms F (ω) and F (ω); • on the second step, we compute the ratio (4); • on the third step, we apply the inverse FFT to the ratio R0 (ω) and compute its inverse Fourier transform P0 (x); • finally, on the fourth step, we determine the desired shift a as the only value a for which P0 (a) = 0.


41

Shift detection in the presence of noise The above algorithm assumes that the images I(x) and I (x) are exactly the same image, differing only by a shift: I (x) = I(x+a). In real life, the measured intensity values have some noise in them. For example, I(x) and I (x) may be two satellite images obtained during two different satellite overflights over the same area. The lighting conditions may slightly change from one overflight to another, which can be represented as the fact that a “noise” was added to the actual image. Due to the noise, even after the shift, the intensity values will be only approximately equal: I (x) ≈ I(x + a). In the ideal non-noise case, the inverse Fourier transform P0 (x) of the ratio (4) is equal to the delta-function δ(x−a), i.e., equal to 0 everywhere except for the point x = a. In the presence of noise, the values of P0 (x) will be slightly different from the delta-function. It seems reasonable to expect that still, the value |P0 (a)| should be much larger than all the other values of this function. Thus, in principle, it may seem that the value of the shift can be determined as the value at which |P0 (a)| is the largest. In practice, however, due to noise, for some spatial frequencies ω, the value of the Fourier transform F (ω) corresponding to the image I(x) may be close to 0, while the value of the Fourier transform F (ω) corresponding to the image I (x) may be non-zero. For such frequencies, the ratio (4) can be very high. These high values dominate the ratio R0 (ω) and thus, distort the inverse Fourier transform P0 (x). To avoid this distortion, it is desirable to replace the formula (4) with a more noise-resistance one. In general, one of the general techniques for making a data processing algorithm more noise-resistant is to take into account constraints on the input data. In the ideal case, the magnitude |R0 (ω)| of the complex ratio R0 (ω) (as described by the expression (4)) is equal to 1. In the presence of noise, the observed values of the intensities may differ from the actual values; as a result, their Fourier transforms also differ from the values and hence, the magnitude of the ratio (4) may be different from 1. Let us therefore describe how we can improve the accuracy of this method if, instead of simply processing the measurement results, we take into consideration the additional knowledge that the magnitude of the actual ratio (4) is exactly equal to 1. Let us denote the actual (unknown) value of the value e2π·i·(ω·a) by r. Then, in the absence of noise, the equation (1) takes the form F (ω) = r · F (ω).

(5)

In the presence of noise, the computed values F (ω) and F (ω) of the Fourier transforms can be slightly different from the actual values, and therefore, the equality (5) is only approximately true: F (ω) ≈ r · F (ω).

(6)

42

C.G. Schiek et al.

In addition to the equation (6), we know that the magnitude of r is equal to 1, i.e., that (7) |r|2 = r · r∗ = 1, where r∗ denotes a complex conjugate to r. As a result, we know two things about the unknown value r: • that r satisfies the approximate equation (6), and • that r satisfies the additional constraint (7). We would like to get the best estimate for r among all estimates which satisfy the condition (7). To get the optimal estimate, we can use the Least Squares Method (LSM). According to this method, for each estimate r, we define the error (8) E = F (ω) − r · F (ω) with which the condition (6) is satisfied. Then, we find among all estimates which satisfy the additional condition (7), a value r for which the square |E|2 = E · E ∗ of this error is the smallest possible. The square |E|2 of the error E can be reformulated as follows: ∗ E · E ∗ = (F (ω) − r · F (ω)) · F (ω) − r∗ · F ∗ (ω) = ∗

∗

F (ω)·F (ω)−r∗ ·F ∗ (ω)·F (ω)−r ·F (ω)·F (ω)+r ·r∗ ·F (ω)·F ∗ (ω). (9) We need to minimize this expression under the condition (7). For conditional minimization, there is a known technique of Lagrange multipliers, according to which the minimum of a function f (x) under the condition g(x) = 0 is attained when for some real number λ, the auxiliary function f (x) + λ · g(x) attains its unconditional minimum; this value λ is called a Lagrange multiplier. For our problem, the Lagrange multiplier technique leads to the following unconditional minimization problem: ∗

F (ω) · F (ω) − r∗ · F ∗ (ω) · F (ω)− ∗

r · F (ω) · F (ω) + r · r∗ · F (ω) · F ∗ (ω) + λ · (r · r∗ − 1) → min .

(10)

We want to find the value of the complex variable r for which this expression takes the smallest possible value. A complex variable is, in effect, a pair of two real variables, so the minimum can be found as a point at which the partial derivatives with respect to each of these variables are both equal to 0. Alternatively, we can represent this equality by computing the partial derivative of the expression (10) relative to r and r∗ . If we differentiate (10) relative to r∗ , we get the following linear equation: −F ∗ (ω) · F (ω) + r · F (ω) · F ∗ (ω) + λ · r = 0. From this equation, we conclude that

(11)


r=

F ∗ (ω) · F (ω) . F (ω) · F ∗ (ω) + λ

43

(12)

The coefficient λ can be now determined from the condition that the resulting value r should satisfy the equation (7). In other words, we must have |F ∗ (ω) · F (ω)| = 1, |F (ω) · F ∗ (ω) + λ| i.e., equivalently, that |F ∗ (ω) · F (ω)| = |F (ω) · F ∗ (ω) + λ|.

(13)

The expression F (ω) · F ∗ (ω) + λ is a real number, so – depending on the sign – its magnitude (absolute value) is equal either to this same number or to its opposite, i.e., F (ω) · F ∗ (ω) + λ = ±|F (ω) · F ∗ (ω) + λ|.

(14)

Due to (13) and (14), we thus have F (ω) · F ∗ (ω) + λ = ±|F ∗ (ω)| · |F (ω)|.

(15)

Substituting the expression (15) into the formula (11), we conclude that r=±

F ∗ (ω) · F (ω) . |F ∗ (ω)| · |F (ω)|

In principle, the sign can depend on the spatial frequency ω. However, since the observed images I(x) and I (x) are functions which are different from 0 only in a bounded area, their Fourier transforms are continuous. It is therefore reasonable to consider expressions which are continuously depending on the frequency ω. To make the above expression continuous, we must use the same sign for all frequencies. If we use the positive sign for all the frequencies, then we arrive at the following ratio: r=

F ∗ (ω) · F (ω) . |F ∗ (ω)| · |F (ω)|

(16)

(One can check that if we select a negative sign for all the spatial frequencies, we will end up with the exact same algorithm.) So, in the presence of noise, instead of using the ratio (4), we should compute, for every ω, the optimal approximation R(ω) =

F ∗ (ω) · F (ω) . |F ∗ (ω)| · |F (ω)|

(17)

This expression is known in signal and image processing; it is called a “crosscorrelation power spectrum” (see, e.g., [5, 33]). What we have just shown

44

C.G. Schiek et al.

is that this expression provides an optimal estimate for the desired value e2π·i·(ω·a) . How can we use this ratio in registration? In the ideal non-noise case, when I (x) = I(x + a), the ratio (17) also equals to e2π·i·(ω·a) , Hence, in the ideal no-noise case, the inverse Fourier transform P (x) of this ratio is equal to the delta-function δ(x − a), i.e., it is equal to 0 everywhere except for the point x = a. In the presence of noise, we expect the values of P (x) to be slightly different from the delta-function, but still, the value |P (a)| should be much larger than all the other values of this function. Thus, the value of the shift can be determined as the value at which |P (a)| is the largest. Thus, we arrive at the following algorithm: • first, we apply FFT to the original images I(x) and I (x) and compute their Fourier transforms F (ω) and F (ω); • on the second step, we compute the ratio (17); • on the third step, we apply the inverse FFT to the ratio R(ω) and compute its inverse Fourier transform P (x); • finally, on the fourth step, we determine the desired shift a as the vector a for which the magnitude |P (a)| attains the largest possible value. The role of soft computing: a comment In the above explanation of why FFT-based techniques are useful in image registration, we started with the need to describe the degree with which the are similar, and ended up with an integral expression images I (x) and I(x+a) (I (x) − I(x + a))2 dx. To come up with this expression, we simply made a heuristic transition. A more justified way would be to use a formalism which is specifically designed to translate natural-language expressions like “close” and “similar” to explicit expressions – the formalism of fuzzy logic. For example, to describe the function “similar”, we can use a Gaussian membership function µ(I − I) = exp(−c · (I − I)2 ), for some real number c > 0. In this case, for each pixel x, the degree with which the corresponding values I (x) and I(x + a) are similar is equal to exp(−c · (I (x) − I(x + a))2 ). We want to find the degree to which them corresponding values are similar for the first pixel x(1) , and the corresponding values are similar for the second pixel x(2) , etc. If we use the algebraic product a · b to describe “and” – one of the choices proposed in Zadeh’s original papers – then the resulting degree that the images I (x) and I(x + a) are similar is equal to the product exp(−c · (I (x) − I(x + a))2 ). x

Since the product of the exponents is equal to the exponent of the sum, def this degree is equal to exp(−c · S), where S = (I (x) − I(x + a))2 . Since x

the degree of similarity exp(−c · S) is a monotonically decreasing function of


45

S, to find the shift a for which the degree of similarity between the image I (x) and the shifted image I(x + a) is the largest, we must find the shift for which the sum S is2 the smallest. This sum is, in effect, the desired integral (I (x) − I(x + a)) dx. We have selected this quadratic expression simply to explain that even in this simple setting, we need to use FFT to make image registration more efficient. If this quadratic integral was the most adequate expression of the degree of similarity between the two images, then we would be able to argue that, since this expression is an example of least square expressions used in statistical data processing, we could probably use the traditional statistical techniques to derive this expression. However, it is known that in many practical problems, the above more sophisticated FFT-based algorithm works much better than the minimization of the quadratic integral (see, e.g., [5]) – which shows that the quadratic integral is, in general, not necessarily the most adequate description of image matching. Intuitively, it is reasonably clear why the more sophisticated algorithm is often better: in the ideal case of the exact shift, this algorithm returns an impulse function, which has 0 values everywhere except for the desired shift point a, and which has a high value at a. In contrast, the value of the crosscorrelation function is slowly decreasing around x = a. Not surprisingly, in the presence of a reasonable noise, the new algorithm leads to a more more accurate reconstruction of the shift than the method based on the minimization of the quadratic integral. The fact that a more sophisticated expression for the degree of similarity is needed is a good indication that soft computing techniques are needed here: for soft computing, if we select more complex membership functions and more complex “and” operations (t-norms), we would end up with more complex expression for the degree of similarity between the two images. Finding the shift with subpixel accuracy By finding the vector a for which the value |P (a)| attains the largest possible value, we end up with a shift which is proportional to the grid step. In other words, by using this method, we can only determine this shift with an accuracy of 1 pixel. For images on a computer screen, 1 pixel accuracy may sound like a very high accuracy. However, for satellite images, 1 pixel is about 15 m. By using GPS, we can locate objects with higher accuracy; it is therefore desirable to register images with higher (subpixel) accuracy as well. In the ideal case – when the actual shift is exactly proportional to the grid step, and the noise level is very low – the function |P (x)| attains the exact maximum when x is equal to the desired shift a, and |P (x)| is equal to practically 0 when x = a. When the actual shift is not proportional to the grid step, then the corresponding “ideal” continuous function |P (x)| still has the same property, with a sharp maximum at a value a which does not coincide

46

C.G. Schiek et al.

with one of the grid points; however, the actually computed function |P (x)| is based on the extrapolation of that function to the grid. As a result, instead of a single maximum at a and 0 in all other points, we will have non-zero values at the grid points close to a. The closer the actual maximum a to the grid point, the higher the value of |P (x)| at this grid point. Thus, to estimate the actual shift, we can find the grid points where the function |P (x)| attains its largest values, and then locate a as a weighted combination of these points, with a weight monotonically depending on the value of |P (x)|. In the 1-dimensional case, if the largest values of |P (x)| are attained at two neighboring points x and x , then this idea leads to the following estimate for the shift: w(|P (x)|) · x + w(|P (x )|) · x , a= w(|P (x)|) + w(|P (x )|) for some monotonically increasing weight function w(z). It can be theoretically proven that the optimal image registration is attained when w(z) = z α for some α > 0, and the empirically optimal value of α is 0.65; see [13]. Similarly, in the 2-dimensional case, to get a subpixel accuracy, we arrive at the following algorithm [13]: • we find the point x = (x1 , x2 ) for which |P (x)| takes the largest possible value; • then, among 4 points (x1 ± 1, x2 ± 1), we select a point (x1 , x2 ) for which the value |P (x1 , x2 )| is the largest; • after that, we apply the formulas a1 =

w1 · x1 + w1 · x1 w2 · x2 + w2 · x2 ; a = , 2 w1 + w1 w2 + w2

(18)

where w1 = |P (x1 , x2 )|α + |P (x1 , x2 )|α ; w1 = |P (x1 , x2 )|α + |P (x1 , x2 )|α ; w2 = |P (x1 , x2 )|α + |P (x1 , x2 )|α ; w2 = |P (x1 , x2 )|α + |P (x1 , x2 )|α ; (19) with α = 0.65, to find the coordinates (a1 , a2 ) of the shift. Resulting algorithm So, we arrive at the following algorithm for determining the shift a: • first, we apply FFT to the original images I(x) and I (x) and compute their Fourier transforms F (ω) and F (ω); • on the second step, we compute the ratio (17); • on the third step, we apply the inverse FFT to the ratio R(ω) and compute its inverse Fourier transform P (x); • finally, on the fourth step, we do the following: • we find the point x = (x1 , x2 ) for which |P (x)| takes the largest possible value;


• •

47

then, among 4 points (x1 ± 1, x2 ± 1), we select a point (x1 , x2 ) for which the value |P (x1 , x2 )| is the largest; after that, we apply the formulas (18) and (19) to find the coordinates (a1 , a2 ) of the shift.

Reducing rotation and scaling to shift If, in addition to shift, we also have rotation and scaling, then the magnitudes Mi (ω) of the corresponding Fourier transforms are not equal, but differ from each by the corresponding rotation and scaling. If we go from Cartesian to polar coordinates (r, θ) in the ω-plane, then rotation by an angle θ0 is described by a simple shift-like formula θ → θ + θ0 . In these same coordinates, scaling is also simple, but not shift-like: r → λ·r. If we go to log-polar coordinates (ρ, θ), where ρ = log(r), then scaling also becomes shift-like: ρ → ρ + b, where b = log(λ). So, in log-polar coordinates, both rotation and scaling are described by a shift. How to determine rotation and scaling In view of the above reduction, in order to determine the rotation and scaling between M and M , we can do the following: • transform both images from the original Cartesian coordinates to log-polar coordinates; • use the above FFT-based algorithm to determine the corresponding shift (θ0 , log(λ)); • from the corresponding “shift” values, reconstruct the rotation angle θ0 and the scaling coefficient λ. Comment. The main computational problem with the transformation to logpolar coordinates is that we need values M (ξ, η) on a rectangular grid in logpolar space (log(ρ), θ), but computing (log(ρ), θ) for the original grid points leads to points outside that grid. So, we need interpolation to find the values M (ξ, η) on the desired grid. One possibility is to use bilinear interpolation. Let (x, y) be a rectangular point corresponding to the desired grid point (log(ρ), θ), i.e., x = elog(ρ) · cos(θ), y = elog(ρ) · sin(θ). To find the value M (x, y), we look at the intensities Mjk , Mj+1,k , Mj,k+1 , and Mj+1,k+1 of the four grid points (j, k), (j + 1, k), (j, k + 1), and (j + 1, k + 1) surrounding (x, y). Then, we can interpolate M (x, y) as follows: M (x, y) = (1 − t) · (1 − u) · Mjk + t · (1 − u) · Mj+1,k + (1 − t) · u · Mj,k+1 + t · u · Mj+1,k+1 , where t is a fractional part of x and u is a fractional part of y.

48

C.G. Schiek et al.

Final algorithm: determining shift, rotation, and scaling • First, we apply FFT to the original images I(x) and I (x) and compute their Fourier transforms F (ω) and F (ω). • Then, we compute the magnitudes M (ω) = |F (ω)| and M (ω) = |F (ω)| of these Fourier transforms. • By applying the above rotation and scaling detection algorithm to the functions M (ω) and M (ω), we can determine the rotation angle θ0 and the scaling coefficient λ. • Now, we can apply the corresponding rotation and scaling to one of the original images, e.g., to the first image I(x). As a result, we get a new image I(x). • Since we rotated and re-scaled one of the images, the images I(x) and I (x) are already aligned in terms of rotation and scaling, and the only difference between them is in an (unknown) shift. So, we can again apply the above described FFT-based algorithm for determining shift: this time, actually to determine shift. As a result, we get the desired values of shift, rotation, and scaling; hence, we get the desired registration. Comment. Similar techniques can be applied to images in other applications areas; see, e.g., [19]; in particular, applications to pavement engineering are described in [2, 26].

4 Images known with low accuracy: case study Introduction In the previous section, we described algorithms for registering images which are known with high accuracy. In this section, we will give examples of images that need to be registered but which are only known with low accuracy. It turns out that for these images, the existing high-accuracy image registration techniques do not work, so new methods are needed. These images are related to terrain changes. In this section, we will describe types of terrain change, explain why detecting terrain change is important, explain why image registration is needed for detecting this change, and select a specific image registration problem related to terrain change. Terrain changes, and why it is important to measure these changes Different natural phenomena can cause the change in the terrain, such as: • earthquakes, via interseismic and coseismic slip along a fault, • glacier advance and retreat, • soil creep, and


49

• landslide processes. For all these phenomena, it is important to be able to estimate the actual terrain change. This estimation is important because it enable us to gauge the hazards associating with these phenomena and to predict the future landscape evolution caused by these phenomena. Vertical terrain changes can be detected by radar techniques, without the need for image registration Some terrain changes can be detected and measured without image registration: by analyzing the radar data. In the past decade, interferometric synthetic aperture radar (InSAR) has become a powerful tool for monitoring such deformation and surface changes [6]. Because this tool detects displacements along the line of sight of the radar system, it is most sensitive to terrain changes due to vertical deformation, such as those associated with thrust faulting, and less sensitive to lateral deformation [29]. To detect lateral terrain changes, we need to register satellite images While InSAR has been used for studying lateral displacements, such as those due to strike-slip earthquakes [20], decorrelation problems in the near-field commonly arise. Moreover, appropriate radar data is not widely available due to the lack of synthetic aperture radar (SAR) satellites in orbit. Currently, the two best SAR satellites in operation are Radarsat and ERS-2. The cost per scene for data from these satellites can range from $950 to $3000, with Radarsat data being the most expensive. Considering the high cost and scarcity of SAR data, the scientific community has looked to other data sets with wider availability, such as the satellite images. Terrain changes can be monitored with optical remote sensing data using image processing algorithms that measure apparent offsets in the geographic locations of the corresponding pixels in two (or more) images of the same portion of the Earth’s surface taken at different times. These inter-image pixel offsets define vectors whose orientations indicate the direction of terrain displacement and whose lengths denote the magnitude of that displacement. Different types of satellite images At present, most lateral terrain changes have been detected by using the Satellite Pour l’Observation de la Terre (SPOT) optical imaging instrument [8, 10, 17, 22, 29]. Previous work with these SPOT images has shown the feasibility of using optical imagery for lateral displacement change detection using Fourier-based cross-correlation (17) [8, 29]. For example, Dominguez et al. [9] were able to resolve coseismic displacement along a major thrust fault associated with the 1999 Chi Chi earthquake in the Central Range of Taiwan from SPOT images using the Fourier approach. These results have

50

C.G. Schiek et al.

shown optical imagery to optimally work in the proximal area of lateral terrain changes, which is the regime where InSAR techniques are weakest [29]. In [23], we have shown that a similar change detection can be obtained with Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) imagery; see, e.g., [1]. The benefits of using ASTER are: • its dramatically lower cost compared to SPOT, • the capability of generating Digital Elevation Maps (DEMs) from ASTER imagery [15], and • the redundancy afforded by ASTERs stereo capability. The latter may aid in increasing the precision of terrain change measurements made using optical image processing techniques. Test case: the November 14, 2001 Kokoxili earthquake

The left-lateral Kunlun fault, located in northern Tibet, was chosen for this study because it experienced a large earthquake during a time period for which ASTER imagery is available. On November 14, 2001, an Ms = 8.1 earthquake occurred causing a 400 km-long surface rupture and as much as 16.3 m of left-lateral strike-slip [16]. This slip is most evident as offset of, and fault scarps developed on, alluvial terraces. Three time separation window cases were considered. These three test cases focus on three different segments of fault. • Test Case 1 spans a two-year time window from March 2000 to July 2002. The image pair for this case exhibits extensive erosion due to the long time span of two years. The image pair for this case also has vastly different pointing angles. • Test Case 2 has a time separation of two-months, from November 4, 2001 to December 22, 2001. The imagery used in this test case has 13% cloud cover, and one of the images contains snow, both of which caused decorrelation. • Test Case 3 has a time separation of thirteen months, between October 2000 and December 2001. The image pair for this case has the least amount of preprocessing problems and the smallest pointing angle difference. Although there is some snow cover and changes in water level along the Kusai Hu lake next to the fault, decorrelation problems were minor. This test case is the only one with a well-defined fault scarp. For each case, the accuracy of the change detection methods was assessed by comparing the geodetic image processing results to field measurements ([16], Table 2; [14, 28]).


51

Site 1

~3 m

Site 2

~10 m

Site 3 ~5 m

= Inferred surface Rupture

Before we start registering the corresponding satellite images, we must apply topographic corrections Our objective is to detect the shift between the images with a subpixel accuracy. To achieve this accuracy, we must first apply the corrections that will lead to an exact alignment of the images. One distortion that needs to be corrected is parallax, which is defined as the apparent shift in the location of an object due to changes in the observers position and, as a result, the relative geometry between the observer and the target. This concept is the same principle on which human stereoscopic vision is based: our right and left eyes view the same scene from slightly different angles, resulting in parallax that we perceive as depth. Thus, topographic

52

C.G. Schiek et al.

parallax is created by the change in the position of a satellite as it scans the uneven surface of the Earth. It results in an apparent shift in the position of terrain features between two overlapping images taken from different angles; see, e.g., [18]. Topography can also impart apparent pixel offsets due to scale changes within the image. In an area of high relief, those parts of the image within valleys will be a greater distance from the observer than those parts on ridge tops. The variable distance between target and observer results in mountaintops having a larger map scale than the valley bottoms, an effect that nonlinearly distorts images. If left uncorrected, this distortion can create apparent pixel shifts when comparing two images. Another apparent pixel offset due to topography stems from the calculation of geographic coordinates. The geographic coordinates of a pixel in an image are calculated from orbital and attitude data and lie on the Earths ellipsoid. Due to the fact the Earth has topography, however, the true latitude and longitude coordinates of a point on the Earths surface will be displaced from its ellipsoidal coordinates. Of the three apparent pixel offsets produced by topography, this last type can cause the greatest apparent shift [29]. All three topographic apparent pixel offsets can be minimized by orthorectifying the image with a DEM. The ortho-rectification process uses the DEM to remove topographic distortions and will re-project the ellipsoidal position of a given pixel to one that better approximates its true coordinates on the Earths surface. In our work, we have applied ortho-rectification techniques to pre-process the images before registering them. All images used in the Kunlun fault test cases are VNIR band 3n images from ASTER level 1B scenes, which has already been through pre-process for sensor artifacts by the ASTER corporation. This band was chosen for two reasons: • because it has the highest resolution, and • because a sensor model is available [32], which describes the interior and exterior orientations of the image ([18]). The sensor model is required for the orthorectification process done using the Leica Photogrammetry Suite in 19 ERDAS IMAGINE 8.7 [32]. ASTER VNIR 3b bands were not considered in the test cases due to registration problems. Orthorectification of the 3b images in ERDAS IMAGINE was poor, and there were residual geometric pixel shifts of at least 20 pixels (210 m). The poor orthorectification can be due to either excessive pointing angle differences, or excessive variation in the viewing geometry. The DEM used for orthorectification was the 90-m Shuttle Radar Topography Mission (SRTM; www.jpl.nasa.gov/srtm) DEM. ENVI 4.0 software was used to register the before and after images, and check the orthorectification process done in ERDAS IMAGINE.


53

5 How to extend the existing image registration techniques to images known with low accuracy Problem When we applied the above algorithm to detect the earthquake-caused shift between the pre- and post-earthquake images, we did not get any meaningful shift value. Specifically, the inverse Fourier transform P (x) of the crosscorrelation power spectrum looks random, and its maximum a was attained at values which are very different from the actual shift. Analysis of the problem In the above algorithm, for every frequency ω, we compute the complex-valued product F (ω) · F (ω) and then consider only the phase of this complex value – i.e., equivalently, the value R(ω) – in the further computations. Due to the uncertainty in the satellite geometry and the wobble of the satellite, we only know the images with low accuracy. As a result, the corresponding Fourier transforms F (ω) and F (ω) can also be only determined with low accuracy. So, for every spatial frequency ω, the product F (ω) · F (ω) can also be only determined with low accuracy. How does this low accuracy translate into the accuracy with which we know R(ω)? Let ε be the accuracy with which we know the value of the product. In general, if we multiply a value x known with accuracy ∆x ≈ ε by a number λ, the resulting new value y = λ · x is known with accuracy ∆y = λ · ∆x = λ · ε. Similarly, if we divide the value x by a number λ, then the resulting new value z = x/λ is known with accuracy ∆z = ∆x/λ ≈ ε/λ. In our algorithm, the cross-correlation power spectrum R(ω) is obtained by dividing the product F (ω) · F (ω) by its magnitude |F (ω)| · |F (ω)|. Since we know the product with accuracy ε, we thus know the value R(ω) with accuracy ε/(|F (ω)| · |F (ω)|). As a result: • for the frequencies ω for which the magnitude is high, we know the corresponding value R(ω) with a reasonable accuracy; • however, for the frequencies ω for which the magnitude is low, the the corresponding value R(ω) is really inaccurate – all noise. In the above algorithm, when we compute P (x) as the Fourier transform of the function R(ω), we take all the values R(ω) with the same weight. In effect, we are taking the average of several values, some known with reasonable accuracy and some very inaccurate. Not surprisingly, the resulting average is very inaccurate. For example, if we have two measurements of the same quantity whose actual value is 1.0, • the first measurement is very accurate and results in 1.05, and • the second measurement is way off and results in 5.61,

54

C.G. Schiek et al.

then when we take the average, we get (1.05 + 5.61)/2 = 3.33 in which the noisy values dominated the accurate ones. How to make resulting estimate more accurate: an idea In view of the above analysis, to make the measurements more accurate, we should: • assign less weight to less accurate values R(ω), i.e., values for which the magnitude |F (ω)| · |F (ω)| is small, and • assign more weight to more accurate values R(ω), i.e., values for which the magnitude |F (ω)| · |F (ω)| is large. The simplest way is to assign weight 0 to less accurate measurements and weight 1 to more accurate measurements. In other words, the simplest way to implement this idea is: • to fix some threshold, and • for all the frequencies for which the magnitude |F (ω)| · |F (ω)| is below this threshold, set R(ω) to 0. A natural idea is to select, as a threshold, a certain portion of the largest (or mean) value of |F (ω)| · |F (ω)|. As a result, we arrive at the following algorithm. Registration algorithm for images that are known with low accuracy: first attempt To find the shift a between the two images: • first, we apply FFT to the original images I(x) and I (x) and compute their Fourier transforms F (ω) and F (ω); • on the second step, we do the following: – we find the mean value m of the product |F (ω)| · |F (ω)|; – we set the threshold to be a certain portion of the mean, i.e., to α · m for some α > 0; – for those frequencies for which |F (ω)| · |F (ω)| ≥ α · m, we compute the value R(ω) by using the formula (17); – for other frequencies, we set R(ω) = 0; • on the third step, we apply the inverse FFT to the function R(ω) and compute its inverse Fourier transform P (x); • finally, on the fourth step, we determine the first approximation to the desired shift a as the point for which |P (x)| takes the largest possible value, and then do the interpolation by using formulas (18)–(19). Empirically, the best value for the parameter α turned out to be 10−3 .


55

How to improve the above algorithm We have mentioned that due to the inaccuracy, it is very difficult to detect the lateral shift. In general, when measurements are not very accurate, a natural way to increase the accuracy is to perform repeated measurements and then average the measurement results. With respect to images, this means that we have to consider more pixels, i.e., large parts of the compared image, corresponding to larger sliding window size. However, the problem is that the lateral shift differs from location to location: its value decreases as we get farther away from the fault. So, when we increase the window size, instead of processing several pixels with the same shift (which would have helped), we instead bring together pixels corresponding to different values of lateral shift. Good news is that while the magnitude of the lateral shift is different at different pixels, the direction of this shift remains largely the same. So, at the first stage of our analysis, we take a large sliding window (larger that 75 × 75 pixels, where 1 pixel is ≈ 15 m), and use the above algorithm to determine the direction of the lateral shift. Once the direction at different locations is determined, we can now take smaller sliding windows (40 × 40 pixels), and determine the magnitudes of the lateral shift. The directions can also be obtained from these smaller windows, but these direction are determined from he analysis of fewer pixels and are, thus, much less accurate than the directions obtained form the analysis of a larger window. Thus, to get the best results, we combine the direction obtained form the analysis of a larger window with the magnitude obtained from the smaller window. In other words, we need to apply the above algorithm twice: • first, with a larger sliding window, to find the direction of the lateral shift; • then, with a smaller sliding window, to find the shift’s magnitude. Finally, we combine the direction obtained from a larger window with the magnitude obtained from a smaller window. Thus, we arrive at the following algorithm: Registration algorithm for images that are known with low accuracy: final description To find the shift a between the two images at a given location x = (x1 , x2 ), we select the two window sizes δ < ∆. Then: • First, we apply the above “first attempt” algorithm to subimages corresponding to a larger box (window) [x1 − ∆, x1 + ∆] × [x2 − ∆, x2 + ∆] def a around x. Based on the resulting shift a, we find the direction e = a def of the actual (unknown) lateral shift (here a = a21 + a22 ).

56

C.G. Schiek et al.

• Then, we apply the above “first attempt” algorithm to subimages corresponding to a smaller box (window) [x1 − δ, x1 + δ] × [x2 − δ, x2 + δ] around def

x. Based on the resulting shift a, we find the shift’s magnitude M = a. • Finally, we combine the direction e obtained from a larger window with the magnitude m obtained from a smaller window into the estimate M · e for the desired shift. This is the new FFT-based algorithm that we propose to use for registering images know with low accuracy.

6 Results of applying the new image registration algorithm To test our new algorithm, we applied it to all three test zones. All test cases display good results in the near field of the faulted area; the accuracy with which we can determine the shift decreases as we move to distal areas. Test Case 2 gives the best results, with a measured lateral displacement of 4.5 ± 0.4 m with left-lateral slip and an average slip direction of 270◦ . This magnitude is similar to the 4.6-4.8 m displacement of a gulley measured by Lin et al. [16] (site 2 in Table 2 from [16]), and the sense and slip direction are consistent with the known trace and kinematics of the Kunlun fault. Test Case 3 is fairly consistent in direction, with left lateral movement and an average slip direction of 265◦ . However, the magnitude obtained from this analysis, ≈ 8.4 m, is a much cruder approximation to the 5.7 m of displacement measured by Lin et al. [16] (site 3 in Table 2). This could be attributed to the long 13-month time separation window during which non-earthquake terrain change occurred, such as changes in the water level in Kusai Hu lake. Test Case 1 results in left-lateral slip with an azimuth of 268◦ in the nearfield of the fault and a magnitude of ≈ 8.3 m. The sense of slip and azimuth are consistent with field observations, but assessing the accuracy of the resulting magnitude is less straightforward. The closest of the Lin et al. [16] field sites is site 7. Several offset features were measured here, with displacements ranging from 3.3 m on a road to 6.8 m on a stream channel. The latter is similar, as is the field measurement at another nearby locality, site 6, where Lin et al. [16] report 7.2 m of displacement on a stream channel.

7 Conclusions and future work Main conclusion Our results have shown that the new algorithm provides a good reconstruction of shift between the two images.


57

Using better DEM Future work needed includes improving the pre-processing protocol. This improvement is needed in order to fully remove any residual apparent pixel offsets and to optimize the true pixel offsets. This can be accomplished by using a 30-m DEM instead of a 90-m DEM in the ortho-rectification process. A higher-resolution DEM can be obtained from aerial photographs or LIDAR, among other sources, but will require an investment of time and resources. By using a DEM with a higher resolution, the elevation uncertainty can be improved, thus lowering the apparent pixel offsets caused by parallax. Comparing images of different type Future work should also include applying the change detection procedures developed in this thesis to heterogenous input imagery, for instance, a combination of an ASTER “after” image with a Landsat TM scene or aerial photographs as “before” images. By using a heterogonous pair of input imagery, a greater number of possible image candidates can be used to do change detection. In addition, since Landsat images and aerial photographs are available for times prior to the beginning of ASTER image acquisition, using heterogeneous datasets can also lengthen the time separation windows that can be considered. This can be especially useful for monitoring terrain change due to slow processes such as glacier movement. It can also make possible the study of events that occurred before ASTER was launched. Comparing images with gridded data sets The algorithms desscribed in this chapter should be able to detect lateral movements in any kind of registered imagery. Thus, the possibility exists to apply these methods to gridded gravity and other geophysical data. Use redundancy of ASTER images As an effort to improve our knowledge of ASTER attitude parameters and to minimize residual apparent pixel offsets during ortho-rectification, as well as improve the performance of the change detection techniques with ASTER data, it may be possible to exploit the redundancy in the ASTER VNIR imagery [29]. The redundancy provided by the ASTER is possible due to its stereo capability, a feature which, given two ASTER scenes, essentially provides four independent sets of images to process for terrain displacement. Given a single “before” scene and a single “after” scene, there are a total of four unique permutations of image pairs that can be used as input to a terrain change detection algorithm. All else being equal, each permutation should result in identical terrain change measurements. Differences in the estimates, however, can be reconciled by optimization of poorly-constrained parameters such as the satellite attitude (e.g., roll, pitch, and yaw). We can also use the fact that the images are multi-spectra [3, 4] (see also Appendix).

58

C.G. Schiek et al.

Towards more accurate shift, rotation, and scaling It is important to find the lateral shift between pre- and post-earthquake images. Once this relative shift is determined, it is desirable to shift one of the images to see what changes occurred. The difficulty is that the shift is subpixel, so when we shift, we move away from the original rectangular grid: the brightness values of the first image were given on a grid, but the values of the shifted image are on the shifted points, which are in between the points of the original grid. Thus, to adequately compare the two images pixel-by-pixel, we must interpolate the brightness values from the shifted grid points (at which we know the brightnesses of the shifted first image) to the original grid point (at which we know the brightnesses of the second image). In the above text, we used bilinear interpolation to actually perform the corresponding geometric transformation (shift, rotation, or scaling). This methods is efficient – it requires only a few computations per pixel – but because of its localized character, it is not always accurate. It is well known that the more points we use for interpolation, the better results we can achieve. Ideally, interpolation should use all the available points. Such methods have indeed been developed based on efficient FFT-based implementations of socalled chirp-z transform – a generalization of Fourier transform [27]. It is desirable to apply these methods to geosciences-related images. Methods from [27] can perform shifts and scalings in an arbitrary rectangular grid, but efficient rotation techniques are only available for the case when we have a rectangular grid with exactly the same step sizes in two dimensions, i.e., when the grid is actually a square grid. For satellite images, it is often not the case. To handle such situations, we must thus: • first, interpolate from the original rectangular grid to the square grid; • then, perform the rotation in the square grid, and • finally, interpolate the rotated image back into the original rectangular grid. Towards explicit representation of interval and fuzzy uncertainty in images In the current image processing, an image is represented as follows: for each pixel x, we describe the approximate value I(x) of the brightness at this pixel. It is desirable to describe not only this approximate value, but also the accuracy with which we know this value. For example, if for each pixel, we know the guaranteed upper bound ∆(x) for the inaccuracy of the corresponding brightness value, this means that at each pixel x, the actual (unknown) value of the brightness I(x) belongs to the interval def + ∆(x)]. − ∆(x), I(x) I(x) = [I(x), I(x)] = [I(x) In a more realistic situation, instead of the guaranteed bound, we may have def with different different values which bound the difference ∆I(x) = I(x)− I(x)


59

degrees of certainty. In other words, for every pixel x, we have nested intervals corresponding to different degrees of certainty – i.e., in effect, a fuzzy value I(x). A fuzzy-valued image is thus simply a nested (layered) family of intervalvalued images. How can we process such interval and fuzzy images? To transform (shift, rotate, scale) an interval image [I(x), I(x)], it is sufficient to rotate the corresponding endpoint images I(x) and I(x). To transform a fuzzy image, it is sufficient to rotate the corresponding interval images layer-by-layer.

Acknowledgments This work was supported in part by NASA under cooperative agreement NCC5-209, NSF grants EAR-0225670 and DMS-0532645, by Army Research Lab grant DATM-05-02-C-0046, by the Star Award from the University of Texas System, and by the Texas Department of Transportation grant No. 0-5453. The authors are very thankful to the anonymous referees for their thorough analysis and valuable suggestions.

References 1. Abrams M., Hook S., Ramachandran B. (2002) Aster User Handbook, NASA Jet Propulsion Laboratory, Pasadena, California 2. Adidhela J.E. (2004) Using FFT-based Data Processing Techniques to Characterize Asphaltic Concrete Mixtures, Master Thesis, Department of Computer Science, University of Texas at El Paso 3. Araiza R. (2003) Optimal FFT-Based Algorithms for Referencing MultiSpectral Images, Master Thesis, Department of Computer Science, University of Texas at El Paso 4. Araiza R., Xie H., Starks S.A., Kreinovich V. (2002) Automatic Referencing of Multi-Spectral Images. In: Proceedings of the IEEE Southwest Symposium on Image Analysis and Interpretation, Santa Fe, New Mexico, USA, April 7–9, 2002, pp. 21–25 5. Brown L.G. (1992) A Survey of Image Registration Techniques. ACM Computing Surveys 24(4):325–376 6. Burgmann R., Rosen P.A., Fielding E.J. (2000) Synthetic Aperture Radar Interferometry to Measure Earth’s Surface Topography and Its Deformation. Annu. Rev. Earth Planet. Sci. 28:169–209 7. Cormen T.H., Leiserson C.E., Rivest R.V., Stein C. (2001) Introduction to Algorithms. MIT Press, Cambridge, Massachusetts 8. Crippen R.E., Blom R.G. (1991) Measurement of Subresolution Terrain Displacements using SPOT Panchromatic Imagery. In: Proceedings of the 1991 International Geoscience and Remote Sensing Symposium IGARSS’91 “Remote Sensing: Global Monitoring for Earth Management”, June 3–6, 1991, Vol. 3, pp. 1667–1670

60

C.G. Schiek et al.

9. Dominguez S., Avouac J.P., Michel R. (2003) Horizontal coseismic deformation of the 1999 Chi-Chi earthquake measured from SPOT satellite images: Implications for the seismic cycle along the western foothills of central Taiwan. Journal of Geophysical Research 108(B2):2083 10. Feigl K.L., Sarti F., Vadon H., McClusky S., Ergintav S., Durand P., Burgmann R., Rigo A., Massonnet D., Reilinger R. (2002) Estimating Slip Distribution for the Izmit Mainshock from Coeseismic GPS, ERS-1, RADARSAT, and SPOT Measurements. Bulletin of the Seismological Society of America 92(1) 138–160 11. Garcia F., Araiza R., Rzycki B. (2000) Towards optimal mosaicking of multispectral images. In: Proc. 2nd Nat’l NASA Student Conference, Nashville, Tennessee, April 7–10, 2000. 12. Gibson S. (1999) An optimal FFT-based algorithm for mosaicking images, Master Thesis, Department of Computer Science, University of Texas at El Paso 13. Gibson S., Kreinovich V., Longpré L, Penn B, Starks SA (2001) Intelligent Mining in Image Databases, With Applications to Satellite Imaging and to Web Search. In: Kandel A, Bunke H, Last M (eds.), Data Mining and Computational Intelligence, Springer-Verlag, Berlin, 309–336 14. King G., Klinger Y., Bowman D., Tapponnier P. (2005) Slip-Partitioned Surface Breaks for the Mw = 7.8 2001 Kokoxili Earthquake, China. Bulletin of the Seismological Society of America 95:731–738 15. Lang H.R. (1999) Algorithm Theoretical Basis Document for ASTER Digital Elevation Models (Standard Product AST14), Version 3.0: NASA Jet Propulsion Laboratory 16. Lin A., Fu B., Guo J., Qingli Z., Guangming D., He W., Zhao Y. (2002) Coseismic strike-slip and rupture length produced by the 2001 Ms = 8.1 central Kunlun earthquake. Science 296(5575):2015–2017 17. Michel R., Avouac J.P. (2002) Deformation due to the 17 August 1999 Izmit, Turkey, earthquake measured from SPOT images. Journal of Geophysical Research 107(B4) 18. Mikhail E.M., Bethel J.S., McGlone J.M. (2001) Introduction to Modern Photogrammetry. New York, John Wiley & Sons, Inc. 19. Osegueda R., Keller G.R., Starks S.A., Araiza R., Bizyaev D., Kreinovich V. (2004) Towards a General Methodology for Designing Sub-Noise Measurement Procedures. In: Proceedings of the 10th IMEKO TC7 International Symposium on Advances of Measurement Science, St. Petersburg, Russia, June 30–July 2, 2004, Vol. 1, pp. 59–64 20. Peltzer G., Crampe F., Rosen P. (200) The Mw = 7.1, Hector Mine, California earthquake: surface rupture, surface displacement, and fault slip solution from ERS SAR data. Earth and Planetary Sciences 333:545–555 21. Reddy B.S., Chatterji B.N. (1996) An FFT-based technique for translation, rotation, and scale-invariant image registration. IEEE Transactions on Image Processing 5(8):1266–1271 22. Riazanoff S. (2002) SPOT Satellite Geometry Handbook. SPOT Image 23. Schiek C.G. (2004) Terrain change detection using aster optical satellite imagery along the Kunlun fault, Tibet, Master Thesis, Department of Geological Sciences, University of Texas at El Paso 24. Srikrishnan S. (2002) Referencing noisy images: complexity of the problem, analysis and efficient implementation of the Fast Fourier approach, Master Thesis, Department of Computer Science, University of Texas at El Paso


61

25. Srikrishnan S., Araiza R., Xie H., Starks S.A., Kreinovich V. (2001) Automatic referencing of satellite and radar images. In: Proceedings of the 2001 IEEE Systems, Man, and Cybernetics Conference, Tucson, Arizona, October 7–10, 2001, pp. 2176–2181 26. Starks S.A., Nazarian S., Kreinovich V., Adidhela J., Araiza R. (2004) Using FFT-Based Data Processing Techniques to Characterize Asphaltic Concrete Mixtures. In: Proceedings of the 11th IEEE Digital Signal Processing Workshop DSP’04, Taos Ski Valley, New Mexico, August 1–4, 2004, pp. 241–245 27. Tong R., Cox R.W. (1999) Rotation of NMR images using the 2D chirp-z transform. Magnetic Resonance in Medicine 41:253–256 28. Van der Woerd J., Meriaux A.S., Klinger Y., Ryerson F.J., Gaudemer Y., Tapponnier P. (2002) The 14 November 2001, Mw = 7.8 Kokoxili Earthquake in Northern Tibet (Qinghai Province, China). Seismological Research Letters 73(2):125–135 29. Van Puymbroeck N., Michel R., Binet R., Avouac J.P., Taboury J. (2000) Measuring earthquakes from optical satellite images. Applied Optics 39(20):3486– 3494 30. Xie H., Hicks N., Keller G.R., Huang H., Kreinovich V (2000) Automatic image registration based on a FFT algorithm and IDL/ENVI. In: Proceedings of the ICORG-2000 International Conference on Remote Sensing and GIS/GPS, Hyderabad, India, December 1–4, 2000, Vol. 1, pp. 397–402 31. Xie H., Hicks N., Keller G.R., Huang H., Kreinovich V. (2003) Implementation, test, and analysis of an automatic image registration algorithm based on FFT and IDL/ENVI. Computers and Geosciences 29(8):1045–1055 32. Yu J., Cowgill E., Healy S. (2003) Digital Stereomapping in Imagine Orthobase, California Institute of Technology, pp. 1–10 33. Zitov´ a B., Flusser J. (2003) Image registration methods: a survey. Image and Vision Computing 21:977–1000

Appendix: Registration of Multi-Spectral Satellite Images Formulation of the problem With the new generation of multi-spectral satellites, for each area, we have several hundred images which correspond to different wavelengths. At present, when we register two images, we only use one of the wavelengths and ignore the information from the other wavelengths. It is reasonable to decrease the registration error by using images corresponding to all possible wavelengths in registration. Similarly, in detecting the known text in colored web images, we would like to take into consideration all color components. In this appendix, we present an algorithm for such optimal registration.

62

C.G. Schiek et al.

Derivation of the new algorithm For multi-spectral imaging, instead of a single image I(ω), we get several images Ii (ω), 1 ≤ i ≤ n, which correspond to different wavelengths. So, we have two groups of images: • the images Ii (ω) which correspond to one area, and • the images Ii (ω) which correspond to an overlapping area. Let us first consider the case when two images differ only by some (unknown) shift a. For every wavelength i, the corresponding two images Ii (x) and Ii (x) differ only by shift, i.e., Ii (x) = Ii (x + a). Therefore, for every wavelength i, their Fourier transforms 1 · Ii (x) · e−2π·i·(x·ω) dx1 dx2 , Fi (ω) = 2π 1 · Ii (x) · e−2π·i·(x·ω) dx1 dx2 , Fi (ω) = 2π are related by the formula: Fi (ω) = e2π·i·(ω·a) · Fi (ω).

(20)

In the ideal no-noise situation, all these equations are true, and we can determine the value r = e2π·i·(ω·a) from any of these equations. In the real-life situations, where noise is present, these equations (20) are only approximately true, so we have the following problem instead: find r for which, for all i, Fi (ω) ≈ r · Fi (ω).

(21)

and which satisfies the condition (7). We would like to get the best estimate for r among all estimates which satisfy the condition (7). To get the optimal estimate, we can use the Least Squares Method, according to which, for each estimate r and for each i, we define the error (22) Ei = Fi (ω) − r · Fi (ω) with which the condition (21) is satisfied. Then, we find among all estimates which satisfy the additional condition (7), a value r for which the sum of the squares |E1 |2 + . . . + |En |2 = E1 · E1∗ + . . . + En · En∗ of these errors is the smallest possible. The square |Ei |2 of each error Ei can be reformulated as follows: ∗ Ei · Ei∗ = (Fi (ω) − r · Fi (ω)) · F i (ω) − r∗ · Fi∗ (ω) = ∗

∗

Fi (ω)·F i (ω)−r∗ ·Fi∗ (ω)·Fi (ω)−r·Fi (ω)·F i (ω)+r·r∗ ·Fi (ω)·Fi∗ (ω). (23) We need to minimize the sum of these expressions under the condition (7).


63

For this conditional minimization, we will use the Lagrange multipliers technique, which leads to the following unconditional minimization problem: n

∗

Fi (ω) · F i (ω) − r∗ · Fi∗ (ω) · Fi (ω)−

i=1

∗ r · Fi (ω) · F i (ω) + r · r∗ · Fi (ω) · Fi∗ (ω) + λ · (r · r∗ − 1) → min .

(24)

∗

Differentiating (24) relative to r , we get the following linear equation: −

n

Fi∗ (ω) · Fi (ω) + r ·

i=1

n

Fi (ω) · Fi∗ (ω) + λ · r = 0.

(25)

i=1

From this equation, we conclude that n

r=

Fi∗ (ω) · Fi (ω)

i=1

n

.

(26)

Fi (ω) · Fi∗ (ω) + λ

i=1

The coefficient λ can be now determined from the condition that the resulting n value r should satisfy the equation (7). The denominator Fi (ω)·Fi∗ (ω)+λ i=1

of the equation (26) is a real number, so instead of finding λ, it is sufficient to find a value of this denominator for which |r|2 = 1. One can easily see that to achieve this goal, we should take, as this denominator, the magnitude of the numerator, i.e., the value n (27) Fi∗ (ω) · Fi (ω) . i=1

For this choice of a denominator, the formula (25) takes the following final form: n Fi∗ (ω) · Fi (ω) def . r = R(ω) = i=1 n ∗ Fi (ω) · Fi (ω)

(28)

i=1

So, for multi-spectral images, in the presence of noise, instead of using the exact ratio (4), we should compute, for every ω, the optimal approximation (28). Hence, we arrive at the following algorithm:

64

C.G. Schiek et al.

A new algorithm for determining the shift between two multi-spectral images If we have images Ii (ω) and Ii (ω) which correspond to different wavelengths, then, to determine the shift a between these two multi-spectral images, we do the following: • first, we apply FFT to the original images Ii (x) and Ii (x) and compute their Fourier transforms Fi (ω) and Fi (ω); • on the second step, we compute the ratio (28) – setting the value to 0 if the denominator is below the threshold; • on the third step, we apply the inverse FFT to the ratio R(ω) and compute its inverse Fourier transform P (x); • finally, on the fourth step, we determine the first approximation to the desired shift a as the point for which |P (x)| takes the largest possible value, and perform the interpolation (18)–(19) to find the actual shift with subpixel accuracy. For rotation and scaling, we can use the same reduction to shift as for mono-spectral images. As a result, we get the desired values of shift, rotation, and scaling; hence, we get the desired registration.

Chapter 3 Neural Networks in Multiple Classifier Systems for Remote-Sensing Image Classification Moumen T. El-Melegy1 and Safaa M. Ahmed2 1

2

Electrical Engineering Department Assiut University Assiut 71516, Egypt [email protected] National Authority of Remote Sensing and Space Science, Cairo, Egypt safaam [email protected]

Summary In recent years there has been a growing interest in the development of supervised classification techniques with higher classification reliability of satellite images. The superiority of one technique over the others cannot be claimed. Many experimental results showed that the classification accuracy depends more on the particular application than on the technique chosen to perform the task. Moreover in many applications it is very difficult to design a classification system that exhibits the required accuracy for the final classification product. Therefore a new technique is emerging that considers multiple classifier systems (MCSs) instead of a single classification technique. Neural networks can participate effectively in a MCS in several ways. This chapter focuses on the role of neural networks in MCSs, which can be either: (1) an individual classifier among the classifiers ensemble, (2) the fusion center that integrates the decisions of individual classifiers, (3) the selector that picks some classifiers’ decisions and ignores the others, (4) or the selector and fuser at the same time. The chapter surveys key neural and non-neural methods in this area and develops some new ones. It also considers several architectures of neural networks including traditional feed-forward neural networks, probabilistic neural networks and radial-basis nets. All methods are evaluated on two real remotesensing datasets, one of which is the standard benchmark Satimage dataset.

1 Introduction Many efforts have been performed to extract information from remotely sensed images and various digital techniques have been developed in this field. Among those digital techniques, classification is a common and powerful information M.T. El-Melegy and S.M Ahmed: Neural Networks in Multiple Classifier Systems for RemoteSensing Image Classification, StudFuzz 210, 65–94 (2007) c Springer-Verlag Berlin Heidelberg 2007 www.springerlink.com

66

M.T. El-Melegy and S.M Ahmed

extraction tool in remote sensing. The first and most common product of image classification is a thematic map, which is the process of assigning the areas and features in the scene to one or more of a number of classes or labels. Examples of the types of classes of interest include soil and rock, crops and forests, and identification of land use in urban and suburban areas. Even where mapping is not an end in itself, classification is usually a prerequisite to further analysis. For example, remote sensing is also used to detect the presence, or confirm the absence, of a specific target class: mineral potential, insect infestations, militarily significant activity, environmental releases from industrial facilities, indications of prehistoric land use, habitat for a given species. Several supervised approaches have been considered for multi-spectral image classification including, statistical methods [21, 10], neural networks [53][64], fuzzy logic [50, 51], genetic algorithms [49], decision trees [48], and support vector machine [22, 36]. Each classification technique has its own advantages and drawbacks. The superiority of one technique over the others cannot be claimed [2, 5, 53, 13, 14], as this strongly depends on the selected dataset and on the effort devoted to the designing phase of each classifier (i.e., designing of classifier architecture, choice of learning parameters, ..., etc.). A recent study [46] of 15 years of peer-reviewed research concerned with classification in remote sensing has demonstrated that there is no discernible difference in classification accuracy between the various approaches. On the other hand, an algorithm may reach a certain level of classification accuracy through a reasonable “designing” effort. Further improvements often require an increasingly expensive designing phase [39]. Additionally, many experimental results showed that the classification accuracy depends more on the particular application than on the technique chosen to perform the task. Thus in many applications it is very difficult to design a classification system that exhibit the required accuracy for the final classification product. Therefore a new technique is emerging that considers multiple classifier systems (MCSs) instead of a single classification technique. MCSs are based upon the principle of consulting different classifiers applied to the given problem. In that regard, two strategies can be identified in literature: classifier fusion and classifier selection. The former combines (fuses) the results of the individual classifiers via some fusion scheme. As such, one abandons the attempt to find the best individual classifier, and instead, tries to use all of the available ones in a smart way. A decision is performed at each classifier. Then the set of decisions is combined in a decision fusion center. This center has a set of schemes to integrate the individual and local decision of each classifier. A variety of schemes have been proposed for combining multiple classifiers, such as the majority vote [14, 31], the weighted average [8, 26], average [35, 41], the Borda count [9], the Bayes approach [15, 44], the Dempster-Shafer theory [17], the fuzzy integral [4, 24, 47], fuzzy templates [29], and combination by neural networks [3]. If the classifiers make

Neural Networks in Multiple Classifier Systems

67

complementary decisions, the use of an MCS will improve the classification accuracy with respect to that of individual classifiers. Fusing the whole set of classifiers in some cases might not provide the expected classification accuracy improvement, however fusing a subset of the classifiers can provide more improvement. Choosing this subset is the objective of classifier selection [6, 13, 19]. The classifier selection approach assumes that each classifier has a region of competence in some area of the feature space, i.e. it exhibits the lower error rate for patterns belonging to that region. Classifier selection is thus aimed at assigning each classifier to its region of competence in the feature space, thus exploiting the complementarities among classifiers. Two types of classifier selection techniques have been proposed in literature. One is called static classifier selection, as the regions of competence are defined prior to classifying any test pattern [18]. The other one is called dynamic classifier selection as the regions of competence are determined during the operation phase, depending on the test pattern to be classified [6, 7]. This choice is typically based on the certainty of the current decision. Preference is given to more certain classifiers. For example, if the 5-nearest neighbor (NN) rule is being used in a 3-class problem, and 2 of the neighbors vote for class 1 and 2 for class 2, we can switch to, say, 3-NN or 1-NN, thereby changing the classification rule dynamically [12]. Another important aspect to the design of MCSs is how to develop a set of mutually complementary (diverse) classifiers to form the members of the MCS in order to achieve high accuracy via decision fusion. A common approach to the generation of such classifier ensembles is to use some form of a sampling technique [65], such that each classifier is trained on a different subset of the training data. Training set re-sampling is used by two very popular techniques called bagging and boosting [65, 66]. Bagging trains each classifier with a training set that consists of a set of patterns drawn randomly with replacement from the original training set. While bagging samples each training pattern with equal probability, boosting focuses on training patterns that are most often misclassified. Essentially, a set of weights is maintained over the training set and adaptive re-sampling is performed, such that the weights are increased for those patterns that are misclassified. A quite different sampling technique, called the Random Subspace Method, has been proposed by Ho [68]. The feature space is randomly sampled instead of the training data, so that complementary classifiers are obtained that have been trained with different feature sets. Neural networks can participate effectively in a MCS in several ways. One popular role is being an individual classifier among the classifiers ensemble [54]-[64]. In that regard, an advantage of neural network classifiers over classical statistical approaches is that neural networks need no priori statistical information about the input image. This is especially important for multi-spectral image, since it is very difficult to model the whole image by statistical methods. In terms of accuracy, neural networks have been shown in some literature [21, 36] to perform favorably in comparison to most statis-

68


tical methods. However, when a sufficiently accurate multivariate statistical model can be determined, statistical method may outperform neural networks in term of classification accuracies [53, 37]. Based on this, it is of interest to use statistical and neural network approaches together in the classification of remote sensing images. A neural network can serve for that purpose as well, in which it acts as fusion center that integrates the decisions of individual classifiers, as the selector that picks some classifiers’ decisions and ignores the others, or even as the selector and fuser at the same time. For the sake of classifier selection, we focus in this work on dynamic selection of classifiers for fusion due to its flexibility and adaptability. More particularly, in this work we attempt to push the flexibility and adaptability of the dynamic selection technique to an extreme; and assess how much one can gain out of this. Towards that end, we allow the subset of classifiers to be picked for final decision on a test pattern to vary depending on the location of the pattern in its feature space. In addition, the combination method that fuses the decisions of the classifiers in the subset is to change dynamically according to the certainty of each classifier’s decision. In order to realize this, we resort to the learning and generalization capabilities of feed-forward neural networks [21, 25], which results in an adaptive, data-dependent methodology for classifier selection and fusion. As the remote-sensing community has became very interested in applying neural networks to image classification [21, 10, 53]-[64] and in comparing the performances of neural network based systems with those of classical statistical and other systems, we outline the objectives of this work in the following: • The main goal is to emphasize the role of neural networks in MCSs, which ranges from a member classifier, to the fusion center that integrates the decisions of individual classifiers, or the selector that picks some classifiers’ decisions, and ending with being both the selector and fuser. We consider different architectures of neural networks including traditional feed-forward neural networks, probabilistic neural networks and radialbasis nets. In addition, several variants of the popular back-propagation algorithm are investigated and evaluated for network training. To the very best of our knowledge, this work is the first effort to highlight such a big picture of the neural world in MCSs. • The chapter surveys many key approaches in the area of MCSs, either for direct classification, decision fusion, or classifier selection. All approaches are compared with the neural approaches on two real remote-sensing datasets: a standard benchmark dataset and a multispectral dataset from Egypt. This work is thus also the first to evaluate and quantify the effectiveness of neural methods, as compared to other non-neural approaches, in all aspects of a MCS. • Another concern is to find answers to the questions “How much can classifier fusion improve on individual classifier accuracy? And how does it compare to classifier selection?”. Although the answer to the first question


69

has already been drafted in several papers (e.g., [35, 52, 2, 39, 42, 14, 27]) the second one remains without a clear answer and open to further investigation. • A new, adaptive neural approach for classifier selection and fusion is proposed. Its performance is assessed on the two test datasets and compared to recently-developed neural classifier selection approaches [11, 45]. • One last interesting question, which is rather of philosophical nature, is to investigate whether building more flexibility and freedom into the MCS brings about more efficiency to the system. The rest of this chapter is organized as follows: We formulate the problem at hand and give important notations in Section 2. Section 3 presents a brief review of neural networks as classifiers. Neural approaches for decision fusion and classifier selection are described in Section 4 and Section 5, respectively. We propose our new adaptive approach for unified classifier selection and fusion in Section 6. Several experiments to evaluate all the previous approaches are described in Section 7. Finally, Section 8 gives our conclusions.

2 The problem The problem considered in this work can be formulated as follows. The input is a set of data samples, each is represented by a d-dimensional feature vector x ∈ d . The goal is to assign it to one class label ωi out of C class labels Ω = {ω1 , · · · , ωC }. To accomplish this task, a set of L classifiers, that have already been trained separately to solve the classification task at hand, are to be consulted in parallel. For combining classifiers, we will focus on parallel architectures, see Fig. 1. The classifier outputs are treated as the input to a second-level decision making in some intermediate feature space. A classifier output can be: • Class label (hard output): Zj ∈ Ω. • Soft output: A C-element vector Zj = [zj1 , · · · , zjC ]T which represents the supports to the C classes. A special case of this vector is a probability distribution over Ω estimating the posterior probabilities P (ωi /x), i = 1, · · · , C. Combining classifiers means thus to find a class label in Ω for x ∈ d based on the L classifiers outputs.

70


Fig. 1. Basic architecture for combining multiple classifiers

3 Neural classifiers A neural network can be regarded as a learning process controlled by a learning algorithm, the function of which is to modify the weights of the network to attain a desired design objective. There are various benefits of neural networks [54], such as the input-output mapping, non-linearity, adaptivity, mathematical modeling, learning ability, knowledge representation, optimization, fault tolerance and real-time operation capabilities. Several neural classification methods are used here: the feed forward multilayered neural network (FFNN) with one and two hidden layers, radial basis network and probabilistic neural network. 3.1 Feed-forward multilayered neural networks Feed-forward, multilayered neural networks are the conventional neural network model widely used in the supervised classification of multi-spectral satellite image data [23],[54]-[60]. The network consists of an input layer, one or more hidden layers and output layer. The input layer has an input node for each value in the feature vector x and typically an output node for each class. Neurons perform a biased weighted sum of their inputs and pass this activation level through a transfer function, often a sigmoid function, to produce their output as shown in Fig. 2. The network thus has a simple interpretation as a form of input-output model, with the weights and thresholds (biases) being the free parameters of the model. The network is typically fully connected, with no intra-layer connections. The back-propagation algorithm or a variation of this algorithm is used to reduce the error and improve model function [21]. These networks can approximate any continuous function to an arbitrary precision and have been shown to be adept as classifiers [25]. By using training pixels, the network can be trained adaptively by updating its


Fig. 2. Non-linear model of a neuron, f (v) =

1 1+e−v

71

, where v = ( dk=1 wk xd + θ)

connection weights. The network output is a certain degree of confidence or a pattern may belong to more than one category. 3.2 Radial basis function (RBF) neural network Radial Basis Functions [34] have attracted a great deal of interest due to their rapid training, generality and simplicity. When compared with traditional feed forward multilayered, RBF networks present a much faster training, without having to cope with traditional back-propagation problems, such as network paralysis and the local minima. These improvements have been achieved without compromising the generality of applications. It has been proved that RBF networks, with enough hidden neurons, are also universal approximators [38]. The RBF network is based on the simple idea that an arbitrary function y(x) can be approximated as the linear superposition of a set of localized basis functions, leading to a structure very similar to the feed forward multi-layer. The RBF is basically composed of three different layers as shown in Fig. 3, each layer is fully connected to the following one: the input layer, which basically distributes the input data; one hidden layer, with a radially symmetric activation function, hence the network’s name; and one output layer, with linear activation function. For most applications, the Gaussian format is chosen as the activation function of the hidden neurons. Each radial function has a local response (opposite to the global response of sigmoid function) since their output only depends on the distance of the input from a center point. Centers should be assigned to reflect the natural clustering of the data [33]. Once centers are assigned, deviations are set. The size of the deviation (also known as a smoothing factor) determines how spiky the Gaussian functions are. If the Gaussians are too spiky, the network will not interpolate between

72


Fig. 3. Schematic diagram of a RBF neural network classifier

known points, and the network loses the ability to generalize. If the Gaussians are very broad, the network loses fine detail. This is actually another manifestation of the over/under-fitting dilemma. Deviations should typically be explicitly chosen so that Gaussians overlap with a few nearby centers. The Gaussian activation function for RBF networks is given by: φi (x) = exp(−(x − µi ) Σi−1 (x − µi )),

(1)

for i = 1 · · · Hn , where Hn is the number of hidden units. µi and Σi are the mean and covariance matrix of the ith Gaussian function used. The output of the RBF network is: F (x) = w0 +

Hn i=1

exp(−

wi − µi 2 ), σ2

(2)

where σ is the spread of the Gaussian function, wi , i = 1, 2, · · · , Hn are the synaptic weights, and . denotes the Euclidean norm. 3.3 Probabilistic neural network (PNN) The Probabilistic Neural Network [40] provides a general solution to pattern classification problems by following an approach similar to Parzen Estimators [21]. It is called a “neural network” because of its natural mapping onto a three-layer feed forward network consisting of input layer, a pattern layer, and a summation layer as shown in Fig. 4. The network contains an input layer that contains d nodes to accept a d-dimensional feature vector. It has a pattern layer consists of C pools of pattern nodes, which organizes the training set such that each input vector is represented by an individual processing element. A radial basis function and a Gaussian activation function are used for


73

Fig. 4. Schematic diagram of a PNN classifier

these pattern nodes. And finally, the network contains an output layer, called the summation layer, which has as many processing elements as there are classes to be recognized. Each element in this layer combines via processing elements within the pattern layer which relate to the same class and prepares that category for output. A linear basis function and a linear activation function are used in this layer. Sometimes a fourth layer is added to normalize the input vector, if the inputs are not already normalized before they enter the network. The probabilistic neural network uses a supervised training set to develop distribution functions within a pattern layer. These functions are used to estimate the likelihood of an input feature vector being part of a learned category, or class. The learned patterns can also be combined, or weighted, with the a priori probability, also called the relative frequency, of each category to determine the most likely class for a given input vector. If the relative frequency of the categories is unknown, then all categories can be assumed to be equally likely and the determination of category is solely based on the closeness of the input feature vector to the distribution function of a class. The likelihood function for feature vector x of class ωi is: Li (x) =

Ni 1 exp(− x − xj 2 /σi2 ), Ni (2πσ)d/2 j=1

(3)

where Ni is the number of input points N per class. The conditional probability for class ωi is P (ωi /x) = Li (x)/ j=1 Lj (x). The class likelihood functions are sums of identical isotropic Gaussian centered at the point x. The single free parameter of this algorithm is σ, the variance of the Gaussian (the rest of the terms in the likelihood functions are determined directly from the training data). Hence, training a PNN consists of optimizing σ relative to some evaluation criterion, typically the validation error.

74


4 Neural decision fusion Once the set of classifiers has been created, an effective way of combining their outputs must be found. As an example, the “majority” voting rule [31, 1, 20, 15, 44], by far the most popular fusion approach, interprets each classification result as a “vote” for one of the data classes and assigns the input pattern to the class receiving the majority of votes. The significance of classifier combination is that for a specific application problem, each of these classifiers could attain a different degree of success, but none of them may be perfect, or even as system which consists of these classifiers can be integrated so as to improve the recognition rates of the overall classification system which consists of these classifiers. A variety of schemes have been proposed for combining multiple classifiers. One can distinguish between two methods of combining decisions: fixed and trained methods. The maximum, majority and average rules [27, 35, 41] are among the approaches that can be categorized as fixed combining methods. These combining methods do not acquire any information from the input data. The other approach is trained combining methods, where the combining method possesses parameters acquired via training. Such approaches are Demspter-Shafer [17, 44], weighed average [8, 26, 42], Behavioral Knowledge Space and fuzzy integral approaches [4, 24, 47], fuzzy templates [29]. Another way of categorizing these different classifier combining techniques is based on the method of mapping between the input and output of the fusion module. This mapping may be linear or non-linear. Linear combination is the simplest approaches, in which a weighting factor is assigned to the output of each expert being combined. This weighting factor can be specific to the input pattern, or can be specific to the expert. Weighted average [8, 26, 42], fuzzy integrals are among the linear combination methods. Non-linear methods include approaches such as the majority, or maximum votes. The feature-based approach [11], stacked generalization [43], or rank based methods [28], which involve a more complex mapping of the input, also use a nonlinear mapping in the combining method. Decision fusion using a neural network is a trained, non-linear scheme. Feed-forward multilayered neural networks [3] are used for fusing the classifiers’ outputs having an input node for each value in the classifier’s outputs and typically an output node for each class, see Fig. 5. The input nodes can represent soft or hard classifiers’ decisions. We will use neural networks with both hard and soft outputs. We will also investigate using PNNs as a fusion scheme with soft inputs.


75

Fig. 5. Fusion using a feed-forward neural network with C × L input nodes and C output nodes

5 Classifier selection using neural networks In decision fusion, described earlier, all the available classifiers contribute to the classification outcome. If the classifiers make complementary decisions, fusion will improve the classification accuracy with respect to that of individual classifiers. Fusing the whole set of classifiers in some cases might not provide the expected classification accuracy improvement, however combining a subset of the classifiers may provide more improvement. Thus the classifier selection methodology aims to select one or more classifiers for inclusion in the final decision. One flexible way to do this is to choose adaptive, data dependent committees from the members of a MCS. In this called dynamic selection scheme, depending on an input data point, different members and a different number of them may be chosen to make a committee decision about this point. This seems reasonable, since the classifiers may have different accuracy in different regions of the input space. Several dynamic classifier selection approaches have been proposed in literature. Rastrigin and Erenstein [16] proposed the following dynamic selection scheme. The “competence” of each classifier is estimated in the vicinity x as the classifier’s accuracy. Two methods were suggested for this: the potential functions method and the k-nearest neighbors. The classifier with the highest competence is authorized to label x. Woods et al. [19] also used local analysis of competence to nominate a classifier to label x, in which they use local accuracy to select the classifier of choice for a given pattern, each test feature vector x is labelled by the classifier with the highest accuracy computed in a neighborhood of x. Exactly one classifier or more than can one be nominated to make the decision.

76


Of particular interest to our work here are neural-network based techniques. A well-known such technique is the one proposed by Kamel and Wanas [11], in which a feed-forward neural network (FFNN) takes the output of the ensemble of classifiers along with the input feature vector x to try and learn how to weight the different classifiers. These weights reflect the degree of confidence in each classifier. In other words, it tries to understand and collect information that might be helpful in determining how to combine the different classification outputs to achieve a better performance. Figure 6 illustrates the approach in [11]. The input to the FFNN is a vector consisting of the individual classifiers’ outputs and the input features. For each training data vector we form a L dimensional target vector ti = [ti1 , · · · , tiL ]T , with tij = 1, if the xi data vector was correctly classified by the j th classifier and tij = 0, otherwise. Using the training dataset and those target vectors the neural network is trained to predict whether or not the classification result obtained from the L classifiers for an input data vector of length (CL+d) will be correct. The output vector, P, of the selector network is then used in combining the outputs of the classifiers using any proper combination rule such as the weighted average fusion rule [8], with the elements of P being the required rule weights.

Fig. 6. Dynamic classifier selection based on FFNN having (d+ CL) input nodes and L output nodes

A similar scheme was adopted in [45], where the selector’s output P are used to select a subset of L to be fused by calculating a threshold value β. The j th classifier is included into a committee if pj > β, where pj is the j th output of the selector network. A proper value of β is determined in the training phase and then fixed for the use in the classification phase. Afterwards, a


77

combination scheme such as majority voting [14] is applied to the decisions of those “lucky” classifiers.

6 Selection and fusion using neural networks In this section, we propose an adaptive, data-dependent neural approach for both classifier selection and fusion. In this scheme the neural network is used as both the selector and the fuser, as illustrated in Fig. 7. The neural network takes the output of the ensemble of classifiers along with the input features x to learn how to weight the different classes. For each training data vector, we form a C dimensional target vector ti = [ti1 , · · · , tiC ]T , with tij = 1, if the xi data input was correctly belonging to class j, and tij = 0, otherwise. Using the training dataset and those target vectors the neural network is trained to predict the confidence values of each class. The final decision is given to the class with the highest confidence value.

Fig. 7. Proposed dynamic classifier selection based on a FFNN having (d + CL) input nodes and C output nodes

Obviously, this approach provides more flexibility and adaptability than the selection methodology of the previous section. The network learns which classifiers should be included in the final outcome and the relative reliabilities of the classifiers to be taken into consideration. The network also learns how to combine the decisions of those selected classifiers. It is important to note that this approach is similar in spirit to the versionset framework proposed in [67] to improve neural network generalization. However there is some difference. The approach in [67] was developed assuming that the input to its selector net comes from a set of neural networks and may

78


optionally include the original input vector. In the method described here, the feature vector x is a crucial input to the selector/fuser network so that the learning of the network takes into account the input feature vector, and the network output may thus change dynamically if the input x is changed. This results in an adaptive, data-dependent selection approach. In addition, the input to the network also includes the outputs from all individual classifiers, neural and non-neural. it is also interesting to note that as each member classifier provides C outputs, even the different outputs of the very same classifier may be treated differently by the network. This is not the case in the other selection approaches [11, 45].

7 Experimental results In this section, we evaluate the neural networks in the different positions within a MCS and compare their performance to other well-known methods. We report the experimental results on two remote-sensing datasets that will be described first. 7.1 Datasets The first dataset is the standard benchmark Satimage dataset that is available to the public at ftp://ftp.dice.ucl.ac.be/pub/neural-ets/ELENA/databases. It contains intensities of pixels derived from Landsat satellite images that have been segmented into 6 classes. 4 spectral bands were used and the feature vector contains intensities of the central pixel along with the 8 surrounding pixels, altogether 36 features quantized from 0 to 255. For our purpose in this work, the data were randomly subdivided into a training set with 3324 samples (50% of the whole dataset), a validation set with 1111 samples (20% of the dataset), and a test set with 2000 samples (30% of the dataset) keeping the class distributions similar to that of the full dataset. The second dataset used is a 252 × 251 sub-area of a scene of El-Fayum area in Egypt acquired from Landsat Thematic Mapper (TM). All 7 multispectral bands are used including the infrared thermal band, so each pixel is characterized by seven-element feature vector containing the brightness values in the six optical bands and the one of the infrared thermal band, see the first 3 bands represented as an RGB image in Fig. 11(a). Seven distinct classes have been selected by human experts from the National Authority of Remote Sensing and Space Science, Egypt. The selected samples are randomly divided into 420 points per band for training dataset, and 252 points per band in the test set and 168 points per band in the validation set.


79

Table 1. Settings for training FFNN classifiers Item

Setting

Learning rate 0.05 Maximum number of epochs 1000 Maximum number of epochs 1000 Early training stoppage if the network performance on the validation set fails to improve or remains the same for maximum consecutive epochs 5

7.2 Individual classifiers Neural classifiers were designed for the Satimage dataset. A careful designing phase has been carried out to tune each classifier for best performance using the already-prepared training/validation sets. Two feed-forward multilayered neural networks with one and two hidden layers are trained with more than 80 different topologies in order to choose the best training function and the appropriate numbers of neuron in the hidden layer(s). For each topology, ten trials with different initial random weights were performed. Eight different variants of the popular back-propagation training algorithm, available from the Matlab neural network toolboxes, have been examined. All trials are carried out on the same training and validation data under the settings listed in Table 1. Note that some training algorithms, such as traingdx and traingda, change the learning rate adaptively starting with the learning rate 0.05 as mentioned in the tabulated settings. Note also that for all neural network classifiers feature values have been normalized [21]. A typical method is to center each feature separately and then reduce it to a unit variance; this process is recommended when the original features have different scales. The average validation accuracies of the different training algorithms with each topology are compared in Fig. 8. The figure shows that we obtain the highest validation accuracy with the least complexity at topology 36-40-6 using the training function trainrp 3 . Using this training function, Fig. 9 shows the average validation accuracy against different network topologies for a FFNN network with two hidden layers. The number of neurons in the first hidden layer was varied from 10 to 100. At each number of hidden neurons at this layer, a family of curves has been obtained by trying different number of neurons in the second layer in the range 10 to 100 neurons. The best topology from the figure is 36-70-20-6 which is of a comparable complexity with the single hidden layer topology. A design phase is also carried out to obtain the value of the smoothing parameter σ for the RBF and PNN classifiers. Figure 10(a) shows that σ 3

Resilient back-propagation: Simple batch mode training algorithm with fast convergence and minimal storage requirements.

80


Fig. 8. The averaged validation accuracy of FFNN classifier for eight different training functions for the Satimage dataset, the best performance is achieved at topology 36-40-6 using trainrp

Fig. 9. The averaged validation accuracy of FFNN with two hidden layer classifier for the Satimage dataset, the best performance is achieved at topology 36-70-20-6


(a)

81

(b)

Fig. 10. Trials for obtaining the best value for σ in RBF network classifiers (a) and in PNN classifiers (b) for the Satimage dataset

reach its peak at 220 in RBN, whereas Fig. 10(b) shows that for PNN the peak occurs at σ = 0.5. For the sake of comparison and to have more members for our multiple classifier system, three classifiers from the popular statistical family are also implemented, namely, parametric Bayes classifier assuming normal densities, non-parametric Bayesian classifier using the Parzen density estimate, and knearest neighbor (k-NN) classifier. Following our previous strategy, we carried out different trials with different values of k for the k-NN classifier. In addition, several window widths h have been examined for the Parzen window approach. The classification accuracy of all the individual classifiers on the Satimage dataset, as assessed using the prepared test set, is reported in Table 2, along with the classifiers’ best working parameters. It is important to note that results obtained here are comparable to (and often outperform) those reported in literature using similar approaches on the Satimage dataset [27, 19]. This reflects our careful classifier designing phase. The very same scenario has also been applied to the El-Fayum Landsat dataset. Similar 7 classifiers have been carefully designed. Table 3 shows their accuracies on the test set along with the classifiers’ best working parameters. Sample results of pixel-wise classification are shown in Fig. 11. The figure shows 3 bands of the original data represented as an RGB image and the classified images by some classifiers. It is clear from Tables 2,3, the best performance on the Satimage dataset was obtained by the k-NN then PNN classifiers, while the Bayes classifier showed the least performance. On the other hand, all classifiers demonstrated rather high, comparable accuracy on the El-Fayum Landsat dataset, with the RBN and PNN classifiers being the best and the Parzen window approach being the worst.

82

M.T. El-Melegy and S.M Ahmed Table 2. Individual classifiers’ accuracies on the Satimage dataset Classifier

Accuracy(%)

Bayes k-NN, k = 3 Parzen, h = 5 FFNN, 36-40-6 FFNN, 36-70-20-6 RBN, σ = 220 PNN, σ = 0.5

84.85 90.10 89.10 88.68 87.85 84.85 89.95

Table 3. Individual classifiers’ accuracies on the Landsat dataset Classifier Bayes k-NN, k = 13 Parzen, h = 13 FFNN, 7-10-7 FFNN,7-10-20-7 RBN, σ = 3 PNN, σ = 0.1

Accuracy(%) 95.63 95.20 94.84 94.99 95.32 96.43 96.43

The results obtained in have revealed that the performance of a classifier may saturate at a certain level, behind which improvement may become difficult to obtain irrespective of any further classifier tuning. To get around this, decision fusion is to be pursued to combine the outputs of different classifiers applied on the same data, in the hope of combining the best of what each approach provides. 7.3 Decision fusion experiments The set of 7 classifiers designed and tuned for best performance in the previous subsection on the same prepared training/validation datasets are taken as the members of the MCS, whose soft outputs are to fused. For that purpose, a feed-forward neural network (FFNN) with one hidden layer was designed for the Satimage dataset. Trained with the trainrp training algorithm, several attempts were made to determine the appropriate number of hidden neurons. The role of training and validation sets was reversed in this phase; the network was trained with the previously-prepared validation set while the training set is used for early-stopping and validation. The number of hidden neurons is varied from 5 to 100 neurons in the hidden layer leading to 20 different network topologies. For each topology, ten trials with different initial random weights were performed. A topology of 42-35-6 was found to work best, see Fig. 12. Similarly, a PNN classifier is designed with σ = 2.5.


83

Fig. 11. Sample pixel-wise classification results of the El-Fayum dataset: (a) 3 bands of original data. (b) Bayes classifier. (c) FFNN with topology 7-10-7. (d) RBF network

Moreover, another fusion FFNN, with topology 6-30-6, was designed for hard outputs from the individual classifiers. Other non-neural fusion schemes were also implemented. Those include: fixed fusion methods (voting [31, 1], and the mean, minimum, maximum, median, simple Bayes average, and product rules [27, 35]) and trained fusion schemes (weighted average [8], decision templates [14], and Dempster-Shafer [17]). The performance of all fusion approaches on the test set is reported in the second column of Table 4. The table shows that the highest classification accuracy is obtained using the simple, yet efficient majority voting scheme and the least is obtained applying the maximum fusion rule. Majority voting scheme gives an improvement of about 1.8% over the best classifier’s accuracy and an improvement of nearly 4% over the classifiers’ average accuracy. The average fusion accuracy is about 3.2% higher than the classifiers’ average accuracy. The PNN fusion comes next with improvement of about 1.52% over best individual classifier and about 3.7% over the average classifier’s accuracy. The result also shows that we get better accuracy using the classifiers’ soft outputs than hard outputs as they contain more information. Analogously, all these fusion schemes are applied to the El-Fayum Landsat dataset. Having design a fusing FFNN of topology 49-30-7, the accuracy performance of all fusion schemes on the test set is reported in the third

84


Fig. 12. The averaged accuracy of the FFNN fusion scheme versus number of hidden units using the classifiers’ soft outputs on the Satimage dataset, the best performance is achieved at topology 42-35-6 using the trainrp algorithm Table 4. Accuracy (%) of various fusion schemes on the Satimage (Set 1) and El-Fayum Landsat (Set 2) dataset Scheme

Set 1 Set 2

FFNN with soft inputs 91.21 95.65 FFNN with hard inputs 88.74 94.48 PNN 91.60 96.03 Majority voting Mean Max Median Minimum Product Weighted average Decision Template Dempster-Shafer

91.88 91.50 90.20 91.30 90.95 91.20 91.35 91.10 90.70

96.78 96.03 96.03 96.03 96.43 96.03 96.03 96.03 96.03

column of Table 4. The highest classification accuracy is obtained on this dataset again using majority voting scheme, with an improvement of about 1.2% over the classifiers’ average accuracy. It is also quite obvious that almost all fusion schemes exhibited similar performance on the test set and that the improvement gained by fusion is considerably small. This can be attributed to two things. First, the classification accuracy of the individual classifiers on this set is sufficiently high, refer to Table 3. Secondly, it is noticed that there is some correlation among the individual classifiers’ outputs as they tend to


85

commit same errors on same test samples. It is well-known [37, 30, 9] that such dependencies among classifiers negatively influence MCS performance. Clearly, there are no advantages that can be foreseen in combining classifiers that generalize identically, no matter how ingenious a combination method is employed. This last observation implied that attaining highly-accurate individual classifiers may not fully exploit the benefit gainable from fusion and, thus the effort and time spent on tuning the classifiers may be not needed or even desired. This also has suggested another experiment to check whether a set of classifiers, designed without any performance tuning, can achieve through decision fusion such high accuracy. To see this, the same experiment is repeated on the El-Fayum dataset using the same classifiers without going through the tuning phase. Besides, the Bayes classifier is replaced with a linear classifier. The accuracies of this set of intentionally-made weak classifiers on the test set are listed in Table 5, along with parameters settings. Applying the same fusion schemes on the test set for this set of classifies, the results reported in Table 6 are found. The improvement due to fusion now becomes more evident.The majority voting scheme tops the other schemes with an improvement of about 2.9% over the best classifier’s accuracy and an improvement of about 10.8% over the classifiers’ average accuracy. Average fusion accuracy is about 6.9% higher than the classifiers’ average accuracy. This clearly shows that the performance of a MCS with weak individual members can be comparable to or even better than that of a set of strong classifiers that are costly designed. Table 5. Accuracies of the weak classifiers on the El-Fayum Landsat dataset Classifier

Accuracy(%)

Linear Classifier k-NN, k = 3 Parzen, h = 5 FFNN, 7-10-7 FFNN,7-10-20-7 RBN, σ = 5 PNN, σ = 0.71

86.90 92.26 94.23 69.05 77.38 92.86 91.67

7.4 Classifier selection experiments The neural dynamic classifier selection approach [11] is applied to the MCS to classify the Satimage dataset. In order to determine the appropriate number of hidden neurons in the FFNN, the number of hidden neurons is varied from 5 to 100 neurons in the hidden layer leading to 20 different network topologies. For each topology, ten trials with different initial random weights were performed on the validation set and early-stopping is based on the training

86


Table 6. Accuracy of various fusion schemes on the El-Fayum Landsat using weak classifiers Scheme

Accuracy (%)

FFNN with soft inputs, 49-35-7 FFNN with hard inputs,7-25-7 PNN, σ = 2.6

95.89 91.55 96.03

Majority voting Mean Max Median Minimum Product Weighted average Decision Template Dempster-Shafer

97.13 96.03 95.24 96.43 95.63 95.63 96.43 82.14 96.43

set. Determination of the number of hidden neurons is illustrated in Fig. 13. The best working topology was found to be 78-15-7. The network is trained by the trainrp algorithm. The weighted average rule was the fusion scheme employed to combine the decisions of the individual classifiers. On testing this selection scheme, the performance on the test set is found to be 91.45%. That is, there is an improvement of about 1.35% over the best individual classifier (which is the k-NN, see Table 2), and an improvement of about 3.5% over the classifiers’ average accuracy.

Fig. 13. The averaged accuracy of FFNN selector network versus number of hidden units on the Satimage dataset


87

Moreover, the approach in [45] was implemented. During the training phase, a value for the threshold β is determined and tuned for best performance. Figure 14 shows the accuracy of the selector FFNN as a function of β during the training phase. The best performance has been obtained at β = 0.4. That is, classifiers corresponding to a selector’s response above 0.4 are fused via the majority voting rule, and the remainders are discarded. On the test set, that scheme resulted in an accuracy of 91.20%, which is slightly (by 0.25%) inferior to that of [11].

Fig. 14. Selector accuracy versus β for the Satimage dataset during the training phase, the best performance is obtained at β = 0.4

The same neural approaches were also applied to the El-Fayum Landsat dataset for the set of weak classifiers. After a series of experiments, the best topology for the neural network was found to be 56-30-7. The network is also trained using the training algorithm trainrp. The weighted average rule was the fusion scheme employed to combine the decisions of the individual classifiers. The performance on the test set is found to be 95.83%. It thus improves by about 1.6% on the best weak classifier’s accuracy and by about 9.5% on the classifiers’ average accuracy. Alternatively, a subset of classifiers is selected by choosing a threshold β. For the selected classifiers, the majority voting rule is applied. The best performance has been obtained at β = 0.7. Using this fixed value of β on the pre-prepared test set, an accuracy of 96.03% is obtained. That is, the approach this time has slightly enhanced the performance above that of [11].

88


7.5 Experiments on selection and fusion using neural networks The proposed neural approach for classifier selection and fusion is then applied to the Satimage dataset. Using the same training algorithm trainrp, a topology of 78-35-7 has provided the best results on the training/validation sets. This approach has resulted in an accuracy of 91.24% on the test set, with a 1.14% advantage over the best member classifier. When applied to the El-Fayum dataset for the MCS with weak members, a network of topology 56-45-7 has provided the best results on the training/validation sets. This approach has yielded 95.48% accuracy on the test set, which is about 1.25% higher than the best individual in the MCS. 7.6 General comments Table 7 summaries the main results of the MCS on the standard Satimage benchmark dataset, while Table 8 gives a summary of the main results on the El-Fayum Landsat dataset. Care must be taken to read Table 8 as it lists the best individuals from among both the set of well-trained and weak classifiers, and the best working fusion and selection schemes for only the MCS with weak members. Table 7. Summary of main results of the MCS on the Satimage dataset Scheme

Test Accuracy(%)

Best individual classifier (k-NN) Best individual neural classifier (PNN) Individual classifiers’ average accuracy

91.10 89.95 87.91

Best fusion scheme (majority voting) Best neural fusion scheme (PNN)

91.88 91.60

Neural classifier selection with fixed combiner rule [11] Neural classifier selection with fixed combiner rule [45] , β = 0.4 Proposed adaptive neural selection and fusion

91.45 91.20 91.24

From all these results, one can see that decision fusion can indeed improve the performance of the individual classifiers. For example, a test accuracy of about 10.8% is gained using the majority voting fusion rule over the weak classifiers’ average accuracy on the El-Fayum dataset. Moreover, decision fusion can boost up the performance of a MCS with weak individual members to be comparable to or even better than that of a set of strong classifiers, each of which has been optimized in an effort and time expensive process. In that regard, it is also important to stress that the majority voting rule, although


89

Table 8. Summary of main results of the MCS on the El-Fayum Landsat dataset Scheme

Test Accuracy(%)

Best individual well-trained classifier (PNN, RBN) Best individual well-trained neural classifier (PNN, RBN) Best individual weak classifier (Parzen) Weak classifiers’ average accuracy

96.43 96.43 94.23 86.34

Best fusion scheme (majority voting) Best neural fusion scheme (PNN)

97.13 96.03

Neural classifier selection with fixed combiner rule [11] Neural classifier selection with fixed combiner rule [45] , β = 0.7 Proposed adaptive neural selection and fusion

95.84 96.03 95.48

being simple, has consistently provided the best performance among all fusion schemes. Neural individual classifiers did a very good job; In particular, PNN and RBN yielded the best performance on the El-Fayum dataset. Besides, the PNN came as the second best on the Satimage dataset. On the other hand, the neural fusion scheme based on a PNN has shown very satisfactory performance (e.g., it comes second within 0.3% from the best fusion result on the Satimage dataset). Our experiments have demonstrated that neural classifier selection methodology can also provide improvement over the accuracy of the individual classifiers. However this improvement is superseded by the improvement gained form fusing all classifiers without any selection. This agrees to a large extent with some earlier results reported in literature [19, 11], where neural-based and non-neural-based dynamic classifier selection provided marginal improvement over the best fusion accuracy on the Satimage benchmark dataset. One last thing to read from the previous experiments that more flexibility and adaptability do not necessarily lead to more efficiency. The fixed, nontrained majority voting fusion rule has exhibited the best overall performance, in spite of the efforts exerted to optimize the performance of trained, adaptive methods. Furthermore, the proposed adaptive FFNN that acts as selector and fuser has yielded lower performance on both datasets than that of an onlytrainable selector network with fixed fusion rule.

8 Conclusions Many efforts in the field of pattern recognition are being shifted towards multiple classifier systems as a way of further improvement of the classification rate. These systems are now perceived as a universal and obvious advancement of the single-best strategy [42].

90


In this chapter we have been interested in applying MCSs to remotesensing image classification. Since neural networks have enjoyed great success as universal classifiers, we are particularly concerned with their roles in a MCS. As such, the main goal of this chapter has been to underline and call attention to these roles that range from a member classifier, to the fusion center that integrates the decisions of individual classifiers, or the selector that picks some classifiers’ decisions, and ending with being both the selector and fuser. Towards that goal, no efforts have been reported before in literature to explore these aspects altogether in a MCS. We have evaluated the significance of these roles on two real remote-sensing datasets, one of which is the standard benchmark Satimage dataset. We considered different architectures of neural networks including traditional feedforward neural networks, probabilistic neural networks and radial-basis nets. Several variants of the popular back-propagation training algorithm are also investigated to obtain the best performance. In addition, experimental comparisons are made with many key non-neural approaches used in MCSs. We have also proposed a new approach that uses a multilayered feedforward neural network as a trainable selector and combiner. Our approach provides more flexibility and adaptability as it permits the fusion rule, as well as the classifiers to be selected for fusion, to dynamically change based on the input features to be classified. The new adaptive approach is evaluated in comparison with some recently-developed neural dynamic classifier selection approaches [11, 45]. Our experiments have shown that neural classifiers, especially probabilistic neural networks, can achieve high accuracy as individual classifiers. Furthermore they, in their role as a fuser, are capable of considerably enhancing the accuracy of the individual members. However, that enhancement was consistently outperformed by the simple, yet efficient non-neural majority voting rule. The fusion results have also stressed that a MCS consisting of a set of weak classifiers, without any efforts for performance optimization or tuning, can even exceed in accuracy a MCS with very-well-trained members. This last possibility has been exploited in some earlier efforts [27, 20]. Additionally, our experiments have demonstrated that the classifier selection schemes can improve the classification accuracy over the best/average individual classifier’s accuracy. However the improvement over the outcome of fusing all the classifiers’ decisions without any selection is questionable. Similar observation was also reported previously in literature [11, 19]. As a matter of fact, gain from selection methods was smaller than that from fusion in all our experiments. An interesting outcome from our experiments was that more flexibility and adaptability do not necessarily mean more efficiency. That is, in one sense, total freedom is not always a blessing. In spite of the efforts exerted to optimize the performance of the trained, adaptive methods during the training phase, the rigid majority voting fusion rule has exhibited the best overall performance. Likewise, the proposed trainable (and adaptable) network that


91

acts as both selector and fuser has yielded lower performance than that of an only-trainable selector network with non-trainable fuser. More experiments ought to be conducted to verify these findings.

References 1. Battiti R. and Colla A.M. (1994), Democracy in neural nets: Voting schemes for classification, Neural Networks, vol. 7, pp.691-707. 2. Benediktsson J. and Kanellopoulos I. (1999), Classification of Multisource and Hyperspectral Data Based on Decision Fusion, IEEE Tran. Geosci. Remote Sensing, vol. 37, pp. 1367-1377. 3. Ceccarelli M. and Petrosino A. (1997), Multi-feature adaptive classifiers for SAR image segmentation, Neurocomputing, vol.14, pp. 345-363. 4. Cho S.B. and Kim J.H. (1995), Combining multiple neural networks by fuzzy integral and robust classification, IEEE Transactions on Systems, Man, and Cybernetics, vol. 25, pp. 380-384. 5. Farag A. A., Mohamed R. M. and Mahdi H. (2002), Experiments in Image Classification and Data Fusion, Proceedings of 5th International Conference on Information Fusion, Annapolis, MD, vol. 1, pp. 299-308. 6. Giacinto G. and Roli F. (1997), Adaptive Selection of Image Classifiers, Proc. of the 9th ICIAP, Lecture Notes in Computer Science 1310, Springer Verlag, pp.38-45. 7. Giacinto G. and Roli F. (2001), Dynamic classifier selection based on multiple classifier behavior, Pattern Recognition, vol. 34, pp. 1879-1881. 8. Hashem S., Schmeiser B. and Yih Y. (1994), Optimal linear combinations of neural networks: an overview, IEEE International Conference on Neural Networks, pp.1507-1512. 9. Ho T.H., Hull J. J. and Srihari S.N. (1994), Decision Combination in Multiple Classifier System, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 16, pt. 1, pp. 66-75. 10. Jain A.K., Duin R.P.W. and Mao J. (2000), Statistical pattern recognition: a review, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, No. 1, pp.4-37. 11. Kamel M. S., Wanas N. M. (2001), Decision Fusion in Neural Network Ensembles, International Joint Conference on neural Networks (IJCNN’01), Washington D.C., USA, vo.l 4, pp. 2952-2957, Jul 15-19. 12. Kuncheva L.I. (1993), Change-glasses approach in pattern recognition, Pattern Recognition Letters, vol. 14, pp. 619-623. 13. Kuncheva L.I. (2002), Switching between selection and fusion in combining classifiers: An experiment, IEEE transactions on Systems Man and Cybernetics, Part B-cybernetics, vol. 32, pp. 146-156. 14. Kuncheva L.I., Bezedek J.C. and Dubin R.P.W. (2001), Decision Templates for Multiple Classifier Fusion: an Experimental Comparison, Pattern Recognition, vol. 34, pp. 299-314. 15. Lam L. and Suen C.Y. (1995), Optimal combination of pattern classifiers, Pattern Recognition Letters, vol. 16, pp. 945-954. 16. Rastrigin L.A. and Erenstein R.H. (1981), Method of Collective Recognition, Energoizdat, Moscow, In Russian.

92


17. Rogova G. (1994), Combining the results of several neural network classifiers, Neural Networks, vol. 7, pp. 777-781. 18. Verikasa A., Lipnickas A. and Malmqvista K. (1999), Soft combination of neural classifiers: A comparative study, Pattern Recognition Letters, vol. 20, pp. 429444. 19. Woods K., Kegelmeyer W. Ph. and Bowyer K. (1997), Combination of Multiple Classifiers Using Local Accuracy Estimates, IEEE Tran. Trans. Pattern Analysis and machine Intelligence, vol. 19, pp. 405-41. 20. Chuanyi J. and Sheng M. (1997), Combinations of Weak Classifiers, IEEE Trans. Neural Networks, vol. 8, No. 1. 21. Duda R. O., Hart P. E. and Stork D. (2001), Pattern Classification”, 2nd edition, Wiley. 22. ERDAS (1999), ERDAS field guide, ERDAS, Inc., 5th edition. 23. Fischer M.M., Gopal S., Staufer P. and Steinocher K. (1997), Evaluation of neural pattern classifiers for a remote sensing application, Geographical Systems, vol. 35, no.2, pp. 308-325. 24. Grabisch M. (1995), Fuzzy integral in multi-criteria decision making, Fuzzy Sets and Systems, vol.69, pp. 279-298. 25. Hornik, K. (1989), Multilayer Feedforward Networks are Universal Approximators, Neural Networks, Vol. 2,pp. 359-366. 26. Jacobs R.A. (1995), Methods for combining experts’ probability assessments, Neural Computation, vol 7, pp. 867-888. 27. Kittler J., Hatef M., Duin R.P.W. and J. Matas (1997), On Combining Classifier, IEEE Transactions on Pattern Analysis and machine Intelligence, vol. 20, pp. 226-239. 28. Kohavi R. and Wolpert DH. (1996), Bias plus variance decomposition for zeroone loss function, Machine Learning: Proc. 13th Int. Conf., Morgan Kaufmann, pp. 275-283. 29. Kuncheva L.I, Bezdek J., and Sutton M. (1998), On combining multiple classifiers by fuzzy templates, In: Proceedings of the 1998 Annual Meeting of the North American Fuzzy Information Processing Society, NAFIPS’98, Pensacola FL, pp. 193-197. 30. Kuncheva L.I. (2005), Diversity in multiple classifier systems, Information Fusion, 6, pp. 3-4. 31. Lam L. and Suen C.Y. (1997), Application of Majority Voting to Pattern Recognition: an Analysis of Its Behavior and Performance, IEEE Transactions on Systems, Man, and Cybernetics, vol. 27, No. 5. 32. Lam L. (2000), Classifier combinations: Implementations and theoretical issues, In Multiple Classifier Systems. First International Workshop, MCS 2000, Cagliari, Italy, vol.1857 of Lecture Notes in Computer Science, Springer-Verlag, pp. 77-86. 33. Lee Y. (1991), Handwritten Digit Recognition Using K-Nearest-Neigbor, Radial-Basis Functions, and Backpropagation Neural Network, Neural Computation, vol.3, pp. 440-449. 34. Moody J.E. and Darken C.J (1989), Fast Learning in Networks of LocallyTuned Processing Units, Neural Computation, vol. 1, pp. 281-294. 35. Munro P. and Parmanto B. (1997), Competition among networks improves committee performance, In Advances in Neural Information Processing Systems 9, MIT Press, Cambridge, pp. 592-598.


93

36. Perkins T.C. (2000), Remote sensing image classification and fusion for terrain reconstruction, MSc. Thesis, University of Louisville, KY. 37. Petrakos M., Benediktsson J. and Kanellopoulos I. (2001), The Effect of Classifier Agreement on the Accuracy of the Combined Classifier in Decision Level Fusion, IEEE Tran. Geosci. Remote Sensing, vol. 39, pp. 2539-2546. 38. Poggio T. and Girosi F. (1990), Networks for Approximation and Learning, Proceedings of the IEEE, vol. 78, pp. 1481-1497. 39. Roli F. and Giacinto G. (2002), Design of Multiple Classifier Systems, in H.Bunke and A Kandel (Eds.), Hybrid methods in Pattern recognition, World scientific Publishing. 40. Specht D. F. (1990), Probabilistic neural network, Neural Networks, vol. 3, pp. 109-118. 41. Taniguchi M. and Tresp V. (1997), Averaging regularized estimators, Neural Computation, vol. 9, pp. 1163-1178. 42. Tumer K. and Ghosh J. (1999), Linear and Order Statistics Combiners for Pattern Classification, In: Sharkey, A.J.C. (ed.): Combining Artificial Neural Nets, Springer, pp. 127-161. 43. Wolpert D. (1992), Stacked generalization, Neural Networks, vol. 5, pp. 241-259. 44. Xu L., Krzyzak A. and Suen C.Y. (1992), Methods of combining multiple classifiers and their application to handwriting recognition, IEEE Transactions on Systems, Man, and Cybernetics, vol. 22, pp. 418-435. 45. Lipnickas A. and Korbicz J. (2003), Adaptive selection of neural networks for a committee decision, IEEE International Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Application, Lviv, Ukraine, pp. 109-114. 46. Wilkinson G. (2005), Results and implications of a study of fifteen years of satellite image classification experiments, IEEE Tran. Geosci. Remote Sensing, vol. 43, no. 3, pp. 433-440. 47. Kumar A.S., Basu S.K. and Majumdar K.L. (1997), Robust classification of multispectral data using multiple neural networks and fuzzy integral. IEEE Tran. Geosci. Remote Sensing, vol 35, pp. 287-790. 48. Kumar A. S. and Majumder K. L. (2001), Information fusion in tree classifiers. Int. Journal of Remote Sensing, vol. 22, no. 5, pp. 861-869. 49. Tso B. C. K. and Mather P. M. (1999), Classification of multisource remote sensing imagery using a genetic algorithm and Markov random fields. IEEE Trans. Geosci. Remote Sensing, vol. 37, no. 3, pp. 1255-1260. 50. Seong J. C. and Usery E. L. (2001), Fuzzy image classification for continental scale multitemporal NDVI series images using invariant pixels and an image stratification method. Photogramm. Eng. Remote Sensing, vol. 67, no. 3, pp. 287-294. 51. Zhang J. and Foody G. M. (1998), A fuzzy classification of sub-urban land cover from remotely sensed imagery. Int. J. Remote Sensing, vol. 19, no. 14, pp. 2721-2738. 52. Giacinto G., Roli F., and Bruzzone L. (2000), Combination of neural and statistical algorithms for supervised classification of remote-sensing images. Pattern Recognition Letters, vol. 21, no. 5, pp. 385-397. 53. Giacinto G. and Roli F. (1997), Ensembles of Neural Networks for Soft Classification of Remote Sensing Images. Proc. of the European Symposium on Intelligent Techniques, Bari, Italy, pp. 166-170, March 20-21.

94


54. Petersen M. E., de Ridder D. and Handels H. (2002), Image processing using neural networks - a review. Pattern Recognition, Vol. 35, No. 10, pp. 2279-301. 55. Gamba P. and Houshmand B. (2001), An efficient neural classification chain of SAR and optical urban images. Int. J. Remote Sensing, vol. 22, no. 8, pp. 1535-1553. 56. Yoshida T. and Omatu S. (1994), Neural network approach to land cover mapping. IEEE Trans. Geosci. Remote Sensing, vol. 32, no. 5, pp. 1103-1109. 57. Bischof H., Schneider W., and Pinz A. J. (1992), Multispectral classification of Landsat images using neural networks. IEEE Trans. Geosci. Remote Sensing, vol. 30, no. 3, pp. 482-490. 58. Heerman P. D. and Khazenie N. (1992), Classification of multispectral remote sensing data using a backpropagation neural network. IEEE Trans. Geosci. Remote Sensing, vol. 30, no. 1, pp. 81-88. 59. Atkinson P. M. and Tatnall A. R. L. (1997), Neural networks in remote sensing. Int. J. Remote Sensing, vol. 18, no. 4, pp. 699-709. 60. Paola J. D. and Schowengerdt R. A. (1995), A review and analysis of backpropagation neural networks for classification of remotely-sensed multispectral imagery. Int. J. Remote Sensing, vol. 16, no. 16, pp. 3033-3058. 61. Serpico S. B. and Roli F. (1995), Classification of multisensor remote-sensing images by structured neural networks. IEEE Trans. Geosci. Remote Sens., vol. 33, no. 3, pp. 562-578. 62. Kanellopoulos I. and Wilkinson G. G. (1997), Strategies and best practice for neural network image classification. Int. J. Remote Sensing, vol. 18, no. 4, pp. 711-725. 63. Ji C. Y. (2000), Land-use classification of remotely sensed data using Kohonen self-organizing feature map neural networks. Photogramm. Eng. Remote Sensing, vol. 66, no. 12, pp. 1451-1460. 64. Dreyer P. (1993), Classification of land cover using optimized neural nets on SPOT data. Photogramm. Eng. Remote Sensing, vol. 59, no. 5, pp. 617-621. 65. Breiman L. (1996), Bagging Predictors. Machine Learning, vol. 24, pp. 123-140. 66. Freund Y. and Schapire R. (1996), Experiments with a new boosting algorithm. Proc. the 13th Int. Conf. on Machine Learning, pp. 148-156. 67. Partridge D. and Griffith N. (1995), Strategies for improving neural network generalization. Neural Computing and Applications, vol. 3, pp. 27-37. 68. Ho T. K. (1998), The random subspace method for constructing decision forests. IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 20, pp. 832-844.

Part II

Applications in Image Retrieval

Chapter 4 Linguistic Expression Based Image Description Framework and its Application to Image Retrieval Qingyong Li1,2 , Zhiping Shi1 , and Zhongzhi Shi1 1

2

Institute of Computing Technology Chinese Academy of Sciences 100080, Beijing, China {liqy,shizp,shizz}@ics.ict.ac.cn School of Computer and Information Technology Beijing Jiaotong University 100044, Beijing, China

1 Introduction Content-based image retrieval (CBIR) has been researched for many years, but there are quite a few problems affecting the development of the CBIR. Most of the disappointments with early CBIR systems come from the lack of recognizing the existence of the semantic gap and its consequences for system set-up. The semantic gap is the lack of coincidence between the information that one can extract from the visual data and the semantics that the same data have for a user in a given situation. Systematically describing and automatically obtaining the image semantic features are open issues [1]. We are not based on the digital feature vectors, but on the concepts describing the target images in natural language when we search images in the database or internet. But natural language is imbued with imprecision, vagueness and other forms of uncertainty in its syntactic structure and semantic content. It becomes a difficult task for computers to describe images with natural or natural-like language. How to utilize languages to describe the semantic features of an image becomes a significant but difficult problem. In this chapter, we propose a linguistic expression-based image description (LEBID) framework to provide images with semantic expression based on linguistic variable in fuzzy theory [2]. LEBID systematically demonstrates how to depict the image semantic features with natural language based on linguistic variable and how to extract the image semantic features with semantic rule. Furthermore, it also shows how to manipulate the linguistic values and linguistic expression with the syntax rule. At the end of LEBID processing, each image is associated with a semantic-vector, and each semantic component in Q. Li et al.: Linguistic Expression Based Image Description Framework and its Application to Image Retrieval, StudFuzz 210, 97–120 (2007) c Springer-Verlag Berlin Heidelberg 2007 www.springerlink.com

98

Q. Li et al.

the vector is assigned a confidence factor (or degree of membership). For instance, an image semantic-vector (coarse:0.90, regular:0.82, , harmony:0.76 ) for an art image means that the image is believed to be coarse, regular, and harmony with 0.90, 0.82 and 0.76 confidence value, respectively. When a textbased search is issued with such concepts, images are retrieved and ranked based on their combined confidence factors in the matching semantic concepts. As pointed out by [3], automaticly image object semantics extraction may not attain high accuracy with the present state of the computer vision and image processing technology. However, providing images with some global semantic descriptions based on the global visual features is believed to be feasible and effective to image retrieval [4]. LEBID aims to provide images with global semantic descriptions based on linguistic expression. Our application in texture image retrieval shows that even though the image linguistic expressions are not object-based descriptions, LEBID assists users to find some relevant images rapidly via keywords. More specifically, LEBID has two salient advantages: 1. LEBID provides a flexible semantic description scheme with natural language. The linguistic expression in LEBID is specified by the PhraseStructure grammar rule, not by roster. Moreover, LEBID models the vagueness and ambiguity for vision perception with fuzzy set and fuzzy logic, which is more accordant with the human perception. 2. LEBID construct the semantic rule, mapping from visual feature to semantic feature, by supervised machine learning method. It is simple and effective since it does not involve in segmenting of images. The rest of the chapter is organized as follows: The following section introduces the related work about image retrieval. Section 3 demonstrates the details of linguistic expression based image description(LEBID) framework. In the next we illustrate the application in texture image retrieval. The discussion is given in Section 5. Last we draw a conclusion in section 6.

2 Related work We discuss the related work in content-based image retrieval and image semantic description in this section. 2.1 Content-based image retrieval Content-based image retrieval (CBIR) was started in the 1990s, which made use of the automatic extraction of lower level image features, such as texture, color, shape and structure. Much research has been done to develop advanced

Linguistic Expression Based Image Description Framework

99

techniques about extracting image features and measuring the similarity between a pair of images based on their feature vectors. The overview of CBIR can be seen in [1]. Visual feature extraction is the basis of any content-based image retrieval system. Widely used features include color, texture, shape and spatial relationships. Because of perception subjectivity and the complex composition of visual data, there does not exist a single best representation for any given visual feature. Multiple approaches have been introduced for each of these visual features and each of such approaches characterized the feature from a different perspective. Color is one of the most widely used visual features for content-based image retrieval. It is relatively robust and simple to represent. Various studies of color perception and color spaces have been proposed [1, 32]. The color histogram is the most commonly used representation technique, statistically describing the combined probabilistic property of the three color channels. Swain and Ballard proposed the histogram intersection measure that has been a fairly standard metric for analyzing histogram based features [33]. Texture refers to the patterns in an image that present the properties of homogeneity that do not result from the presence of a single color or intensity value. It is a powerful discriminating feature, and presents almost everywhere in nature. However, it is almost impossible to describe texture in words, because it is virtually a statistical and structural property. There are three major categories of texture-based techniques [34], namely, probabilistic/statistical, spectral, and structural approaches. The well known Tamura features were introduced in [21], which included coarseness, contrast, directionality, linelikeness, regularity and roughness. Shape representation is normally required to be invariant to translation, rotation, and scaling. In general, shape representations can be categorized into either boundary-based or region-based. The former uses only the outer boundary characteristics of the entities while the latter uses the entire region. Well known methods include Fourier descriptors and moment invariants [32]. A number of representative generic CBIR systems have been developed in the last ten years. QBIC,and Photobook are the most prominent examples in business and academia. In addition, GIFT is popular free CBIR software. QBIC was developed at the IBM Almaden Research Centre [5, 6]. It was the first commercial CBIR application and played a vital role in the evolution of CBIR systems. The QBIC system supported low level image features of average color, color histogram, color layout, texture and shape. Moreover, users could provide pictures or draw sketches as example images in query. The visual queries could also be combined with textual keyword predicates. Photobook, developed at the MIT Media Lab, was a set of interactive tools for searching and querying images [8]. It was divided into three specialized systems, namely Appearance Photobook, Texture Photobook, and Shape Photobook, which could be used in combination. The features were compared by using one of the matching algorithms. These included Euclidean, Maha-

100

Q. Li et al.

lanobis, divergence, vector space angle, histogram, and wavelet tree distances, as well as any linear combination of those previously discussed. GIFT, a GNU Image Finding Tool, is available free of charge [41]. GIFT provided with the function of query by example, furthermore, it also provided the opportunity to improve query results by relevance feedback. Though CBIR showed great advantages in image retrieval compared with traditional text-based image retrieval, it also encountered some challenges, such as “semantic gap” between the lower visual image features and image semantic descriptions, “page zero” problem which means the difficulty to find the initial relevant images when browsing the image database. After all, human prefer using keywords to conduct search, so semantic based image retrieval becomes the more attractive and effective approach. 2.2 Semantic based image retrieval Semantic-based image retrieval (SBIR) makes use of the text descriptors to retrieve relevant images. Some recent studies [1, 9] found that text descriptors such as objects, events of image content, and topical terms are most helpful to users. The advantage of this approach was that it enabled widely approved text information retrieval systems to be used for visual information retrieval systems. Most multimedia systems that employ semantic information for retrieval are lexicon-based, requiring a framework to represent and extract the semantics of a multimedia object. A major challenge faced by such systems is that of describing and extracting image semantics. There are three approaches to obtain the image semantics. 1. Manual labeling: Long before digital images were available, access to image collections was provided by librarians, and archivists through the manual assignment of text descriptors [10]. However, manual assignment is time consuming and costly while automatic assignment may not be possible if the image collections do not have accompanied text. Besides, the descriptions of images are often subjective. Different people have different interpretations for an image, even though the same people have different interpretations for an image in different time. 2. Semantics Extraction: Semantic information extraction can be divided into two ways as to the source of semantics: from image itself and from the context (or external). Image-Based Methods: Image segmentation techniques are employed to detect some predefined objects in the image. An approach proposed by Chang et al. [11] used the semantic visual templates (SVTs), a collection of regional objects within a video shot, to express the semantic concept of a users query. The templates could be further refined by a two-way interaction between the user and the system. Saber and Tekalp [12] proposed a method to extract objects from an image based on color and edge detection. Recently, Wang et al. [13] proposed SIMPLIcity, a system that


101

captures semantics using the robust Integrated Region Matching metric. The semantics were used to classify images into two broad semantic categories ‘textured/nontextured ’ and ‘graph/photograph’. Then the classification was used to support semantics-sensitive image retrievals. Context-Based Methods: The text surrounding the images was analyzed and the system extracted those that appeared to be relevant [14]. Benitez and Chang [9] presented a method to extract semantic concepts by disambiguating words’ senses with the help of the lexical database WordNet. In addition, the relationships between keywords can be extracted using relations established in WordNet. 3. Semantic Propagation: A small subset of images is first manually annotated with semantics. Then the annotation is propagated to the rest of the database using some machine learning or statistical methods. Picard and Minka [17] used image texture to propagate annotation. After a user labeled a patch of an image, this label could be propagated to other images with similar patches of texture. Naphade et al. [18] presented a system which contains a lexicon learnt from users using annotated examples. The formulation of semantic relationships in the lexicon was modeled as a classification problem. The resulting retrieval system was able to support keyword-based queries, as well as to detect multimodal events. All the approaches mentioned above rely on local features, which in turn rely on high segmentation accuracy. However, segmentation can hardly be done reliably, especially on compressed images. LEBID performs image semantic description based on global features, and it uses linguistic variables to describe the image global semantic features without the need of segmentation. Our empirical results show that using LEBID, we can perform semantic based image retrieval reasonably well.

3 Linguistic expression based image description framework LEBID provides a scheme to describe the image semantics and to deal with the vagueness of the image semantics. We will demonstrate the detail of LEBID framework in this section. 3.1 Uncertainty in image content and linguistic variable Somebody (not only the opponents of fuzzy logic) may ask why we should use fuzzy techniques in image content description. There are many reasons to do this. The most important of them are as follows [19]: 1. Fuzzy techniques are powerful tools for knowledge representation and processing.

102

Q. Li et al.

2. Fuzzy techniques can manage the vagueness and ambiguity efficiently. In image content analyzing and understanding, we have to use expert knowledge to overcome some difficulties (e.g. object recognition, scene analysis). Fuzzy set theory and fuzzy logic offer us powerful tools to represent and process human knowledge in form of fuzzy rule. On the other side, many difficulties in image content analyzing arise because the data/tasks/results are uncertain. This uncertainty, however, is not always due to the randomness but to the ambiguity and vagueness. Besides randomness which can be managed by probability theory, there are three level imperfection in image content: pixel-level, region-level, and knowledge-level, as showing in Fig. 1. These problems are fuzzy in nature. The question whether a pixel should become darker or brighter (orange or red in color image), the question where is the boundary between two image segments, and the question how about the coarseness of a texture image, all of these and other similar questions are examples for such situations that a fuzzy approach can be a more suitable way to manage the imperfection. Especially, for some special images, such as art image, texture image, it is ambiguity to describe the characteristics of them. For example, it is very difficult and vague to distinguish whether an art image is harmonious or not; it is also inexplicit to define whether a texture image is coarse or not. On the contrary, we are used to describe the art image as “somewhat harmonious” or “very harmonious”, and texture image as “little coarse” or “not very coarse”. Zadeh [2] put forward linguistic variable to model such linguistic expression with fuzzy set, which allows a proposition to be neither fully true, nor fully false, but partly true and partly false to a given degree. We will demonstrate how to use linguistic variable to describe the image semantics in the next section.

Fig. 1. Three level uncertainty about image content


103

Linguistic variable [2] is a variable whose values are natural language expressions referring to some quantity of interest. Linguistic variable can be defined as below. Definition 1. A linguistic variable is characterized by a quintuple denoted by < x, T (x), U, G, M >. Here, x is the name of the variable, and T (x) is the set of the linguistic values of x, of which each element is a word or phrase describing the variable x and coupled with a fuzzy set on the universal U which represents the meaning of the linguistic value. G is the syntax rule responsible for generating some elements in T (x), and M is the semantic rule governing the generation of the meaning M (L, u) of each linguistic value L, L ∈ T (x), u ∈ U. Examined from another viewpoint, a linguistic variable consists of two parts: (1) a syntactic part which describes (a) the set of natural language expressions that are the values of the linguistic variable as well as (b) the structure of set, and (2) a semantic part that associates each natural language expression with a fuzzy subset. The syntactic parts of the above formal definition are T and G; the semantic parts are M and U. while M can assign fuzzy meanings to the elements of T in any manner, it is often useful that the meaning M assigned to a particular element of T (in the extended term set) is a function of its derivation (in the basic term set). In this case, X is said to be a structured linguistic variable (All of the linguistic variables used in this paper are structured). Thus, for a structured variable, X, we can algorithmically construct the meaning, M (t), for a particular t in the extended term set using synthesized and inherited attributes or a syntax-directed translation scheme. 3.2 Linguistic expression based image description framework Our previous work [20] showed that linguistic variable was a powerful tool to describe the image semantics and model the image vagueness. Next, we will demonstrate the detail of how to construct the semantic rule mapping from the image visual feature to image global semantics description and how to manipulate the description formed by linguistic expression. Linguistic name x There are many characteristics to describe the image semantics, for example, Tamura [21] put forward a texture model corresponding to visual perception, in which texture is characterized by six features: coarseness, contrast, directionality, regularity, line-likeness and roughness. And Itten [22] formulated a theory about the use of color in art and about the semantics it induces. Itten observed that color combinations induce effects such as harmony, disharmony, calmness, and excitement that artists consciously exploit in their paintings. So we can choose these characteristics as the name of the linguistic variable, such as coarseness for texture, harmony for color.

104

Q. Li et al.

Linguistic value set T(x) Definition 2. A basic term set is the linguistic value set in which each element can not be separated any more in semantic. Definition 3. An extended term set is the linguistic value set in which each element is created by the word in basic term set through the syntax rule G. T(X) is the set of the natural language terms which may be assigned to X as values. T (X) consists of the basic term set and extended term set. According to the linguistic variable name, we can designate the basic term set, as an instance, we take the word set ‘coarse’, ‘fine’ as the basic term set for linguistic variable coarseness, furthermore through the syntax rule G, we may generate the extend term set, such as ‘very coarse’, ‘not coarse’, ‘medium fine’. In conclusion, T (X) is a natural language expression set with certain grammar structure. Universe of discourse U The universe of discourse U is the whole input range allowed for a given fuzzy linguistic variable. In the domain of image retrieval, U always represents the image itself, or the image visual features approximating the image. Because the raw data of the image is always too difficult to use and understand, the image visual feature (such as color, texture) becomes a better choice than the image raw data. About the details for image visual features can be available in [1]. Semantic rule M M is the semantic rule mapping from U to a fuzzy subset about a specific linguistic value, that is to say ,for u ∈ U , t ∈ T (X), assuming a value in [0, 1] called degree of membership to u, which represents the confidence score that u satisfies the linguistic value t. We can formally describe the task as the following: through sample set {(V1 , y1 ), · · · , (Vn , yn )}, in which Vi ∈ U (i = 1, · · · , n) and yi (i = 1, · · · , n) is the degree of membership in [0, 1] regarding to a special linguistic value. We learn to construct a function (or computable program): Mt : V → y

(1)

where t is the linguistic value. It is very important how to construct the semantic rule M. In other viewpoint, it also can be regarded as image understanding or image semantic feature extraction. Fortunately, there are many machine learning methods to construct such rule, such as neural network, genetic programming. We will show the details about how to construct the semantic rule M with genetic programming in the next section.


105

Syntax rule G The set of linguistic expressions is not an unrestricted set. Rather it is a rich finite set that is carefully structured by the system designer. Because the set is often quite large, it is not usually specified by roster, but rather by rule. The most natural form of the rule is that of a phrase-structure grammar, but here we use a slightly more restricted form, Backus-Naur form (BNF), to specify the set of possible natural language values for a linguistic variable [23, 24]. Figure 2 addresses the BNF of the example ‘coarseness’:

Fig. 2. BNF notation for the set of Natural Language Expressions of the linguistic variable “coarseness”

The terms ‘Extended’, ‘Basic’ play the roles that ‘subject’, and ‘verb’ do in the construction of sets of English sentences. The basic terms are the fundamental notions from which all the other elements of the set are built and the extended terms allow for finely tuning of these basic terms. In principle, every linguistic value has an individual semantic rule M. However, the semantic rules of extended terms are not modeled by learning method as the basic terms are, but rather are modeled as operators on the fuzzy set of the basic terms, since the extended terms themselves are generated by rules. Generally, each element of the extended terms has an operator respectively. Since different domains have different characteristics, the operators are somewhat empirical [25]. Next we define some operators for the extended terms about image semantic description. Notation t stands for linguistic value, and Mt stands for semantic rule of linguistic value t. For ‘very ’, we define V operator as below: V (t) = {Mt (x)2 /x|x ∈ U }

(2)

Because, Mt (x) ∈ [0, 1], then Mt (x) ≥ Mt2 (x), so, the V operator reduces the degree of membership for all elements, except those with degrees of 0 or 1. In additional, it has the property that the decrease for such elements which have low degree of membership is greater than that of elements with high values. For example, if the membership degree for image x corresponding to linguistic value ‘coarse’ is 0.99, than the membership degree corresponding to ‘very coarse’ is square of 0.99, equal to 0.98. But for image y with value 0.8, its degree of membership for ‘very coarse’ is 0.64. For ‘somewhat’, we define Soperator as below: S(t) = {sin(Mt (x) × π)/x|x ∈ U }

(3)

106

Q. Li et al.

Because 0 ≤ Mt (x) ≤ 1, obviously S(t) ∈ [0, 1], especially, the S operator increases the degree of membership for these elements with medium degree of membership toward original linguistic value, but it decreases the degree of membership for those elements with high or low values. In a word, S operator stands out the medium elements. Obviously, for image x, with membership degree 0.5 corresponding to ‘coarse’, its membership degree for ‘medium coarse’ get the greatest value sin(π/2) = 1. For ‘not’ we define N operator as below: N (t) = {1 − Mt (x)/x|x ∈ U }

(4)

The definition of the N operator is intuitive. For each element with high degree of membership, its negative will have low value, so the fuzzy complement is reasonable. The operators work well not only in individual also in concatenation. It seems very natural and also allows for an easy implementation of the concatenation. An expression like “not very coarse” can be represented by the fuzzy set obtained as follows: the operator V is applied to the fuzzy semantic rule for “coarse”, and then the operator N is applied to that result. Syntax rule gives a way to enlarge the expression capacity for linguistic variable, it also helps to organize the linguistic values and to form a structural set with structure grammar. Proposition of linguistic variables In an image retrieval system, there are always more than one characteristic to describe the image semantics, so it is necessary to combine certain linguistic variables. Definition 4. If X1 , X2 , · · · , Xn are linguistic values, describing certain characteristics of image, M (X1 ), M (X2 ), · · · , M (Xn ) are the corresponding semantic rules, and µk (I) refers to the degree of membership generated by M (Xk ) for image I, where 1 ≤ k ≤ n, then we define the vector < (µ1 (I), X1 ), (µ2 (I), X2 ), · · · , (µn (I), Xn ) > as the n-dimension linguistic vector. So image semantics according to n characteristics can be described by n-dimension linguistic vector. Every component corresponds to a fuzzy set for a linguistic value: A = {(µ(I), I)}, where I represents the image, and 0 ≤ µ(I) ≤ 1, represents the membership degree. Because the fuzzy set is not adequate and convenient in the semantic based image retrieval, we have to change the fuzzy value into determinate value. we can define a logic function to further restrict the values with true and false. Φx (α, µ, I, ξ) =

true if µα ≥ ξ f alse if µα < ξ

(5)


107

where x is name of a linguistic variable, α ∈ T (x), µα (I) is the degree of membership, and ξ is the threshold assumed by the user in the range of (0, 1). Φx (α, µ, I, ξ) is a predicate function with four variables, and estimates whether the image I is accordance with the linguistic value α for linguistic variable x. If the membership degree is greater than the threshold ξ the image is assumed as the linguistic values, on the contrary, if the membership degree is smaller than ξ the image is not assumed as such linguistic value. Furthermore, the whole image semantics can be expressed as a first order predicate expression. All of the operation in the first order predicate logic are available here, such as AND (∧), OR (∨), and NOT (∼). So the image semantic description and image query statement can be expressed by a simple predicate expression. For example, natural language description ‘coarse and high contrast or good directionality texture image’ can be transformed into following logic expression: Ψ = Φ1 ( coarse , µ, I, ξ) ∧ Φ2 ( high contrast , µ, I, ξ) ∨ Φ3 ( good directionality , µ, I, ξ).

4 Application in texture image retrieval In this section, we will demonstrate the implementation of this LEBID framework in texture image retrieval, and also show the performance of semanticbased image retrieval. Tamura [21] put forward a texture model corresponding to visual perception, in which texture is characterized by six features: coarseness, contrast, directionality, regularity, line-likeness and roughness. Actually, the former three features are the most frequently used, so we select the coarseness, contrast and directionality to be our semantic features. In this chapter, texture semantic features are described by three linguistic variable < X1 , X2 , X3 > ,X1 represents the texture’s coarseness feature, its basic term set is ‘coarse’, ‘fine’; X2 represents the contrast, its basic term set is ‘high’, ‘low ’; and X3 represents the directionality, its basic term set is ‘linear ’, ‘irregular ’. Next, we will show the details about universe of discourse for texture images and the semantic rule. 4.1 Universe of discourse with texture spectrum feature The universe of discourse U is the whole input range allowed for a given fuzzy linguistic variable. In the domain of image retrieval, it is always represented by the image itself, or the image visual features approximating the image. As discussed in Sec.3, the image visual features, instead of raw data of image, are regarded as the universe of discourse. Texture is one of the most important features for digital image. We here introduce a texture spectrum feature, which is proved to have good accordance with human perception and successfully used in content-based image retrieval [35, 29].

108

Q. Li et al.

Image texture means a kind of pixels’ intensity change (or gray) in a neighborhood, which is spatially statistically relative. It consists of two elements, texture unit and its arrangement. According to this idea, incorporating the concept of texture spectrum proposed by He [26], we present a new method to describe image texture feature. In order to get the local texture information of image pixel, we take its 3*3 neighbor into consideration, shown as Fig. 3, here Ii is the intensities of the pixels. As for color image, Ii can be calculated according to following formula: I = 0.299R + 0.587G + 0.114B

Fig. 3. 3*3 neighbor of pixels, the location of I0 is the central pixel

We record the salient intensity changes of the neighboring pixels using binary sequence V1 V2 · · · V8 , here Vi (i = 1, · · · , 8) is denoted as: Vi =

0 if |Ii − I0 | ≤ T 1 if |Ii − I0 | > T

(6)

Here, T is a predefined constant. We keep eyes on the salient intensity changes, which is consistent with human vision perception for texture. If we transform the sequence V1 V2 · · · V8 from binary to decimal, obviously the value range of the sequence is [0, 1, · · · , 255], that is to say, the total number of texture model of image pixels is 256. The texture unit T U of image pixel can be obtained according to the follow formula,

TU =

8

2i Vi

(7)

i=1

Obviously,the value range of T U is [0, 1, · · · , 255]. By calculating the distribution of image texture unit in the value range, we can get the image texture spectrum. Suppose we denote the texture unit value of pixel (i, j) as T (i, j) , and denote image texture spectrum as {h[k]}(k = 0, 1, · · · , 255) , then


m−1 n−1 h[k] =

i=0

j=0

f (i, j, k)

m×n

109

(8)

1 if T (i, j) = k and m, n are image height and width 0 otherwise 255 respectively. Apparently, k=0 h[k] = 1. Such spectrum texture feature is an interwoven distribution of the intensity change of the pixels. Compared with texture descriptions [21, 26], our texture visual feature has following characters: (1) It describes the intensity changes of the neighboring pixels, not the absolute intensity, and it indicates that texture is a kind of changes about pixel intensity, (2) the texture unit obtained with our method is a local texture, which makes it possible to get texture spectrum. Figure 4 illustrates that the texture spectrum image holds on the salience feature of the origin image, and has a good accordance with the human visual perception since it captures the high contrast pixels just like the biologic visual system. Furthermore, compared with the method proposed in literature [26], our method decreases the dimension of texture spectrum from 6561 to 256. where f (i, j, k) =

Fig. 4. Texture spectrum image. Left) original image; Right) the texture spectrum image in which the value of each pixel is the texture unit value

4.2 Semantic rule learned by genetic programming It is important for the linguistic variable how to construct the semantic rule. M assigns the elements of T with a fuzzy set which measures the confidence score that the texture image can be believed to be consistent with the linguistic value. In other viewpoint, it can be regarded as image comprehension- simple and basic comprehension. Thus, how to simulate the function of the simple visual comprehension becomes the task of the semantic rule M. As we well know, Genetic Programming (GP) [27, 29], has potential as modeling and optimization techniques for complex problems. GP, based on the powerful principle of “survival of the fittest”, models some natural phenomena of genetic inheritance and Darwinian strife for survival. The GP is also an example of a robust method, which makes few assumptions about

110

Q. Li et al.

problem domain. In this subsection, we present our GP-based algorithm for constructing the semantic rule M. Just as discussed in section3, semantic rule can be learnt through sample set {(V1 , y1 ), · · · , (Vn , yn )}, in which Vi ∈ U (i = 1, · · · , n) and yi (i = 1, · · · , n) is the degree of membership in [0, 1] regarding to a special linguistic value. We select 200 images from Brodatz [28] texture database as the training sample set, and label the degree of membership by five people according to special linguistic value, and designate the average as the final degree of membership. Next, we shall address the most important three issues about GP: encoding scheme, fitness function and genetic operations. • Encoding scheme. GP utilizes tree structures to represent the function as Fig. 5. The intra-node is always constituted by the basic functions such as plus, minus; and the leaf-node exclusively includes the terminate symbols, which is comprised of variable expression (composed of variable, coefficient, and power, for example x, 3x4 ) or constant. Obviously, each tree corresponds to a computable function.

Fig. 5. Tree structure of a function sin(2h1 ) + 0.3h23 − h232

• Fitness function. Fitness function influences the quality of the result. We define the fitness function as the root of the square sum of the error between the labeled degree of membership and evolution function result. Suppose that F (v) represents the genetic individual, and the i-th training sample’s degree of membership is σi , and the number of sample is n, then, we can get the fitness function: n (9) F it(F (v)) = (F (vi ) − σi )2 i=1

• Genetic operations. The tree structure allows easily define a closed crossover operator by swapping sub-trees between two valid trees. The scheme is the steady state genetic algorithm: parents are selected by tournament


111

(of size of 2 to 7 typically), and an offspring is generated by crossover. The offspring is then put back in the population using a death-tournament: the individual with the worse fitness gets replaced by the new born offspring. We also use mutation operation: random replacement of a sub-tree or random change of a node or a leaf. The mutation probability is 0.02 in our experiment. Generally speaking, each linguistic value of the linguistic variable has a corresponding semantic rule. But we can get the extend term’s semantic rule M through corresponding syntax rule G since the extend term is generated by the basic term and syntax rule G (As discussed in Section 3). 4.3 Experiments We have presented an integrated LEBID framework based on linguistic variable. In this section, we will describe the linguistic expression based image retrieval system (LEBIRs) that we have implemented using this framework for texture images and show some experiment results. Our experiment is divided into two major parts: 1. Image semantic description and LEBIRs: We demonstrate the semantic features based on linguistic variable, also report the detail of LEBIRs. 2. Performance on image retrieval and influence of parameters: We do some experiments about the retrieval precision of LEBIRs, compared with other state-of-the-art image retrieval approaches. Moreover, we analyze the influence of the parameter in GP. In our experiments, we use a 12 000-image dataset that contains images accumulated from the Brodatz [28] dataset and internet. Brodatz texture images have been widely used by the computer vision and texture analysis. We select 200 images from Brodatz texture database as the training sample set to train the semantic rule M(as discussed in 4). Image semantic description Traditional image classification systems work by classifying an image into one of many predefined categories. Usually, a probability is associated with each category, and the highest one is chosen as the image’s category, moreover, the concept corresponding to the category is assumed as the semantics of the image. Obviously, the description ability for such method is limited since the predefined categories are limited, and the semantics is crisp, not flexible. However, in LEBIRs, we describe the image semantics with a more abundant and flexible linguistic expresses set, which is generated by basic term set and syntax rule G(as discussed in Section 3). The basic term set is designed by system designer, and the basic term set here is ‘coarse’, ‘fine’, ‘high’, ‘low ’, ‘linear ’, ‘irregular ’. The extended term set, ‘very coarse’, ‘somewhat high’, , ‘not

112

Q. Li et al.

low ’, , ‘very irregular ’, is generated by syntax rule G just as Fig. 2; opposite to utilizing the crisp probability in image classification systems, LEBIRs utilizes fuzzy value (degree of membership) to label the image’s semantics according to a certain linguistic value, and such fuzzy value is calculated by the semantic rule M. The rationale is that for every image, LEBIRs assigns a fuzzy value to it according to every linguistic value in linguistic term set. After an image is annotated, it is associated with a K-dimension linguistic-vector, where K refers to the number of linguistic value. Each element in the linguistic-vector includes a keyword, and a value which represents the degree of membership. A typical vector may look like < (coarse, 0.9), (irregular, 0.8), · · · , (very coarse, 0.81) >. Considering the storage cost, we only keep the basic term’s fuzzy values in the feature database, and the extended term’s values will be calculated through the syntax rule G and operators as discussed in Section 3. So every image semantic feature comprises of a 6-dimension vector < v1 , v2 , v3 , v4 , v5 , v6 >, the corresponding semantic description is < coarse, f ine, high, low, linear, irregular > .

Fig. 6. Example of image semantic feature description

The images in Fig. 6 are randomly selected from our feature database. From Fig. 6, we can see that: for each semantic term, if the corresponding value or degree of membership is bigger, the image is more accordant with the semantic in perception. For example, the first component of left image is 0.9, and that of the right is 0.02, in the same time we can feel that the left one is greatly coarser than the right one. Moreover, we can expand the semantic values for the extended linguistic terms using the operators (see Section 3) in the syntax rule G, such us operator V for “very”, operator S for “somewhat”. The left image semantic value for “very coarse” is 0.81 and “somewhat coarse” is 0.309. In the viewpoint of perception, they are also consistent with the semantic concept.


113

LEBIRs LEBIRs implements the semantic-based image retrieval, that is, when the user enters a linguistic express, constructed by a linguistic value or logic combination of linguistic values, the system returns the images which are accordant with the description. In the LEBIRs, Users can retrieval the database through the linguistic expression, such as ‘coarse’, ‘very coarse’, ‘high contrast’ and their logic combination. Following Fig. 7 illustrates the semantic retrieval interface and the retrieval result of the request ‘coarse and somewhat high contrast or somewhat linear ’.

Fig. 7. Retrieval result of ‘coarse and somewhat high contrast or somewhat linear’

Compared with the content-based image retrieval systems which retrieve images by example, LEBIRs allows users to query images by concepts which is represented by linguistic variables. Obviously, it is more accustomed for the human, especially for browsing the image database. Furthermore, the semantic feature is tidy with low dimension. For example, the dimension of semantic feature here is 6 whereas the dimension of the texture spectrum feature is 256. Image retrieval performance Here are some experimental results that we gathered from our system to demonstrate its effectiveness. To our best knowledge, LEBIRs is the first semantic based image retrieval system on fuzzy set, they don’t have the common ground to be directly compared to our approach. In this paper, we compare LEBIRs with the traditional content-based image retrieval system (briefly as CBIRs) and feedback image retrieval system(briefly as FIRs).

114

Q. Li et al.

In order to evaluate the image retrieval performance, precision criterion is used here, which is popular used in the field of information retrieval. Precision can be defined as: precision =

Number of satisfied images Number of returned images

In our experiments, the texture images are selected from Brodatz and internet. The 6-dimension fuzzy vector, corresponding to linguistic value vector (coarse, fine, high, low, linear, irregular), is regarded as the image semantic feature. So we can retrieve the image database with linguistic values or logic expression of linguistic values (as demonstrated in Fig. 7); For CBIRs, the 256-dimension texture spectrum feature is regarded as the image visual feature. We retrieve image database by example image and evaluate the precision according to semantics, that is, we subjectively estimate if the result images are relevant to the query image in semantic description; For FIRs, 4 times feedback are integrated in the CBIRs to refine the retrieval result according to semantics. Noted that the evaluation was constructed by three people and the average precision was regarded as the retrieval performance at last. Because precision is influenced by the number of returned images, we test the precision values in different number of returned images. Table 1. Average retrieval precision comparisons. Here, N refers to the number of returned images N=15 N=30 N=60 LEBIRs 0.823 0.804 0.726 CBIRs 0.634 0.529 0.407 FIRs 0.715 0.632 0.511

Table 1 shows the retrieval precision of LEBIRs, CBIRs and FIRs for the test set. Obviously, the precision of LEBIRs is greatly better than CBIRs and FIRs with all number of returned images. FIRs improves the semantic precision through four times of feedback which can make the system learn the query concepts by interaction with users, compared with CBIRs [30, 31]. Because of ‘semantic gap’ between visual feature and semantic feature, the semantic precision of CBIRs is very low. As demonstrated in Table.1, the precision is 0.407 when N equals to 60. However, LEBIRs outperforms the FIRs without any user feedback. Furthermore, the dimension of semantic feature in LEBIRs is greatly lower than that in CBIRs and FIRs. Note that lower feature dimension means less store space and more effective index structure. The relationship between retrieval performance and GP In our semantic-based image retrieval system, the semantic rule mapping the texture spectrum feature to the global texture semantic feature directly


115

influences the retrieval performance measured by precision here. In GP, the quality of the semantic rule is measured by the fitness(as defined in Equ.9). When we construct the semantic rule using GP we assume that less the individual’s fitness is, better its quality will be. But in effect, the relation between the individual’s fitness and retrieval performance is showed in Fig. 8 with x axis being the GP result’s fitness and y axis being the system’s retrieval precision.

Fig. 8. The relation between the GP individual fitness and retrieval precision. Taken from [29], with permission

Figure 8 shows that the retrieval precision increases as the fitness decreases in the case of fitness greater than 0.5; But when it is less than 0.5, the precision doesn’t always ascend, on the contrary, it decreases in the point x=0.1. It is because that the fitness represents the training error in GP. In some extent, less the fitness is, better the GP is trained, so we can achieve the high retrieval precision. But the visual texture feature is an approximate of the image, and the training set and label semantic value used in GP are also subjective. Obviously, there are some limitations in precisely representing an image. So when the fitness is less than some value, the retrieval precision can not achieve absolute increase as the fitness decreases. In our experiments, when the fitness equals 0.5 we can get the best retrieval precision.

5 Discussion It has been widely recognized that the family of image retrieval techniques should become an integration of both low-level visual features addressing the more detailed perceptual aspects and high-level semantic features underlying the more general conceptual aspects of visual data. Neither of these two types of features is sufficient to retrieve or manage visual data in an effective or efficient way. MPEG-7 attends to standardize the interface of multimedia content description between low-level features and high-level semantic concepts [36]. In

116

Q. Li et al.

the description interface, it was expected that textual descriptions, derived from image visual features, should be incorporated into the CBIR system. However, it doesn’t propose detailed ways about describing and extracting the image semantic content. Needless to say, human beings are much better than computers at extracting and making use of semantic information from images. We believe that complete image understanding should start from interpreting image objects and their relationships. Unfortunately, this goal is still beyond the reach of state-of-the-art in computer vision. As of now, most of the existing approaches are still based on manual annotations. In practice, content-based annotation is often treated as a multi-category classification problem. The initial input to annotation system is a training image set in which each image has been manually annotated with a single semantic label. The mission of annotation system is to propagate these labels to unlabeled images. The most frequently used scheme is the ‘One Per Class’ (OPC) approach. For each of the M classes, OPC constructs a binary classifier to make a ‘yes’ or ‘no’ prediction for it. Given an input image, each binary classifier will produce a prediction, and the category of the image is determined by such class that has the highest confidence for the ‘yes’ prediction. At last, the label of this class is assumed as the semantics of the image. For example, J. Li and J.Z. Wang put forward Markov model [37], and Goh et.al proposed an approach with SVMs [38], Furthermore, Chang et al put forward a content-based soft annotation method with BPM [39]. To measure the confidence, the output of binary classifiers was mapped into a posterior probability value, which represents the confidence in ‘yes’ prediction. In a word, the common ground of these methods is that they are based on classifier and local image features. Though classifier can map an image into certain class associating with a concept. In principle, these methods treat the image semantics as probability events. However, LEBID method directly modeled the image vagueness, and the semantic rule simulates the vision perception, constructing the map from image visual features to the semantic concepts in the form of fuzzy set. Vagueness and subjectivity are two salient characteristics affecting the information retrieval activity. As for image texture semantics, the criteria defining the image semantic characteristics is diverse for different people. For example, someone may think a texture image as ‘very coarse’, whereas someone else may take it as ‘medium coarse’. Our framework well suits to the vagueness and subjectivity with the fuzzy set. The image semantics are always complex and versatile just as the saying ‘a picture is worth a thousand words’. The current image semantic description schemes are lexicon-based [40, 37, 39, 38]. In these methods, all the semantics or keywords for classes are predefined and fixed. It has a great shortcoming: inability to recognize an instance of new semantics. When the label of an unlabeled image u does not belong to the label set C, u is forced to be classified into one of the categories in C. LEBID makes a big difference with these lexicon-based methods. LEBID provides a framework to describe the


117

image semantics based on linguistic variable, and the linguistic expression was constructed by phrase-structure grammar (see section 3). So we can not only describe the image semantics with basic terms, but also we can refine the description with the extended terms. For example, we can depict a texture image with ‘coarse’, it is convenient to concretely depict with ‘very coarse’ or ‘not coarse’. The image semantics formed by the LEBID will do good to image database index. In the early image retrieval systems, the image visual features are always high dimension vectors, more than tens even to hundred, so how to construct an efficient index is another difficult problem. Compared with the visual feature, the semantic features formed from LEBID are more compact with low dimension, and have explicit concepts which are valuable and convenient to cluster and classify images. So the system can organize the image database more efficiently. Obviously, the drawback of this description framework is the restriction for domain. That is to say, we have to, according to the specific domain, prior designate the linguistic variable name and basic term set for linguistic variables. LEBID is an open and extensible framework. According to different specific domain, the systems can take different visual feature, which approximately represents the image, as the universe of discourse. Of course, genetic programming algorithm used in the semantic rule M is also replaceable. Any nonlinear, adaptive machine learning approach can be tried here, such as neural network.

6 Conclusion One of the great challenges for the advancement of content-based image retrieval is the semantic gap. The aim of image retrieval systems is to provide maximum support for bridging the semantic gap between the simplicity of visual features and the specific user semantics. The image semantic description framework in this chapter exploits a new approach to reduce the semantic gap. LEBID framework, using linguistic variable, enables the image retrieval systems to describe image semantics with natural language and to provide the function of semantic-based image retrieval which is more accustomed and convenient for user than content-based image retrieval system. We also implement the framework for image texture feature, and validate the feasibility of this framework. Our experiments show that the semantic feature can effectively capture image texture semantics and the semantic-based image retrieval is also effective and convenient. Furthermore, the framework can deal with vagueness and subjectivity thanks to the advantage of fuzzy theory. Further work will include applications on other domain, such as color feature for art paintings. We will also probe into the utilization of such semantics

118

Q. Li et al.

formed from this framework for image index and image management, other than image retrieval.

Acknowledgements This paper is supported by the National Natural Science Foundation of China No. 60435010 and the National Basic Research Priorities Programme No. 2003CB317004.

References 1. Smeulders A., Worring M., Santini S., Gupta A., and Jain R. (2000) Contentbased image retrieval at the end of the early years. IEEE Transaction on PAMI, 22 (12), pp. 1349-1379 2. Zadeh L.A. (1975) The Concept of linguistic variable and its application to approximate reasoning I II III. Information Sciences,vol. 8, pp. 199-249, 301357, vol. 9, pp. 43-80 3. Barnard K. and Forsyth D. (2000) Learning the semantics of words and pictures. In Proc.Int.Conf.Computer vision, vol.2, pp. 408-415 4. Navon D. (1977) Forest before trees: The precedence of global features in visual perception. Cognitive Psychology, 9 (3), pp. 353-383 5. Flickner M., Sawhney H. and Niblack W. (1995) Query by image and video content: the QBIC system. IEEE Computer 5 (9), pp. 23-32 6. Hafner J., Sawhey H., Equitz W., Flicker M. and Niblack W. (1995) Efficient color histogram indexing for quadratic form distance function. IEEE Transaction on PAMI 17 (7), pp. 729-736 7. Ma W.Y. and Manjunath B.S. (1996) NeTra:a toolbox for navigating large image databased. Multimedia Systems 7 (3), pp. 184-198 8. Pentland A., Picard R. and Sclaroff S. (1994) Photobook:content-based manipulation of image databases. Storage and retrieval for Image and Video Databases, no. 2185, pp. 239-247 9. Benitez A.B. and Chang S.F. (2002) Semantic knowledge construction from annotated image collection. in Proc. IEEE Int. Conf.Multimedia pp. 189-195 10. Goodru A., Rorvig M.E., Jeong K. and Suresh C. (2001) An open source agenda for research linking text and image content features. Journal of the American Society for Information Science and Technology 52 (11), pp. 948-953 11. Chang S.F., Chen W., and Sundaram H. (1998) Semantic visual templates: linking visual features to semantice. In Proc. of IEEE Int. Conf. Image Processing, Chicago, IL, pp. 256-261 12. Saber E. and Tekalp A.M. (1997) Region-based affine shape matching for automatic image annotation and query-by-example. J. Vis. Commun. Image Repres., 8 (1), pp. 3-20 13. Wang J., Li J., and Wiederhold G. (2001) Simplicity: semantics-sensitive integrated matching for picture libraries. IEEE Trans. Pattern Anal. Machine Intell., vol. 23, pp. 947-963


119

14. Shen H.L., Ooi B.C., and Tan K.L. (2000) Giving meaning to www images. In Proc. ACM Multimedia, pp. 39-48 15. Paek S., Sable C., Hatzivassiloglou V., Jaimes A., Schiffman B., Chang S., and McKeown K. (1999) Integration of visual and text-based approaches for the content labeling and classification of photographs. In Workshop on Multimedia Indexing and Retrieval, pp. 324-330 16. Naphade M.R. and Huang T.S. (2000) A probabilistic framework for smeantic indexing and retrieval in video. In Proc. of IEEE Int. Conf. Multimedia, New York, pp. 475-478 17. Picard R.W. and Minka T.P. (1995) Vision texture for annotation. J. Multimedia Syst., pp. 3-14 18. Naphade M.R., Lin C.Y., and Smith J.R. (2002) Learning semantic multimedia representations from a small set of examples. In Proc. of IEEE Int.Conf.Multimedia, pp. 259-265. 19. Kerre E., Nachtegael M. (2000) Fuzzy Techniques in Image Processing 20. Li Q., Shi Z., Shi J., Shi Z. (2005) An image semantic description framework based on linguistic variable In Proc. Fuzzy Logic, Soft Computing and Computational Intelligence, pp. 1035-1039 21. Tamura H., Mori S.J., and Yamawaki T. (1978) Texture features corresponding to visual perception. IEEE Trans. On Sys. Man, and Cyb, MC8 (6), pp. 460-473 22. Itten J. (1961) Art of Color (Kunst der Farbe), Otto Maier Verlag, Ravensburg, Germany 23. Knuth D.E. (1964) Backus normal form vs. Backus Naur form. Communications of the ACM, 7 (12), pp. 735-736 24. Naur P., Backus J.W. (1960) Revised report on the algorithmic language ALGOL 60. Communications of the ACM, 3 (5), pp. 299-314 25. Kurt J.S. (1984) Fuzzy sets, natural language computations, and risk analysis. Computer Science Press 26. He D.C and Wang L. (1991) Texture features based on texture spectrum, Pattern Recognition, 24, pp. 391-399 27. Koza J.R. (1994) genetic programming II: automatic discovery of reusable programs. Cambridge, MAMIT Press 28. Brodatz P. (1966) Textures: A photographic album for artists designers, Dover, NY 29. Li Q., Shi Z., Shi J. (2004) Semantic feature extraction using genetic programming in image retrieval. In Proc. International Conference on Pattern Recognition, pp 648-651 30. Rui Y., Huang T.S. and Mehrotra S. (1997) Content-based image retrieval with relevance feedback in MARS. In Proc. IEEE Int. Conf. on Image Processing, pp. 244-248 31. Rui Y., Huang T.S. (1999) A novel relevance feedback technique in image retrieval. ACM Multimedia, pp. 67-70 32. Rui Y., Huang T.S., Chang S.F. (1999) Image retrieval: past, present, and future. Journal of Visual Communication and Image Representation Vol. 10, pp. 1-23 33. Swain M.J., Ballard D.H. (1991) Color indexing. International Journal of Computer vision, 7 (1), pp. 11-32 34. Gimel F., Jain G.L. (1996) On retrieving textured images from an image database. Pattern Recognition, 29 (9), pp.1461-1483

120

Q. Li et al.

35. Ojala T., Pietikinen M., Harwood D. (1996) A comparative study of texture measures with classification based on feature distributions. Pattern Recognition 29, pp. 51-59 36. Hunter J. (2001) An Overview of the MPEG-7 Description Definition Language (DDL) . IEEE Transactions on circuits and systems for video technology, 11 (6), pp. 765-773 37. Li J. and Wang J.Z. (2003) Automatic linguistic indexing of pictures by a statistical modeling approach. IEEE Transactions on Pattern Analysis and Machine intelligence, 25 (20), pp. 1075-1088 38. Goh K., Chang E., and Cheng K.T. (2001) Svm binary classifier ensembles for image classification. In Proc.of ACM CIKM, pp. 395-402 39. Chang E., Goh K., Sychay G., and Wu G. (2003) Content-based soft annotation for multimodal image retrieval using bayes point machines. IEEE Trans.on Circuits and Systems for Video Technology Special Issue on Conceptual and Dynamical Aspects of Multimedia Content Description, 13 (1), pp. 26-38 40. Mojsilovic A. et al (2000) Matching and retrieval based on the vocabulary and grammar of color patterns. IEEE transactions on image processing, 9 (1), pp. 189-194 41. Squire D.M., Muller W., Muller H., Pun T. (2000) Content based query of image databases: inspirations from text retrieval, Pattern Recognition Letters (Selected Papers from The 11th Scandinavian Conference on Image Analysis SCIA ‘99) 21, pp. 13-14

Chapter 5 Focusing Keywords to Automatically Extracted Image Segments Using Self-Organising Maps Ville Viitaniemi and Jorma Laaksonen Adaptive Informatics Research Centre Helsinki University of Technology P.O.Box 5400, FIN-02015 TKK, Finland [email protected], [email protected]

1 Introduction In this chapter we consider the problem of keyword focusing. In keyword focusing the input data is a collection of images that are annotated with a given keyword, such as “car”. The problem is to attribute the annotation to specific parts of the images. There exists plenty of suitable input data readily available for this data mining type of problem. For instance, parts of the pictorial content of the World Wide Web could be considered together with the associated text. We propose an unsupervised approach to the problem. Our technique is based on automatic hierarchical segmentation of the images, followed by statistical correlation of the segments’ visual features, represented using multiple Self-Organising Maps. The performed feasibility study experiments demonstrate the potential usefulness of the presented method. In most cases, the results from this data-driven approach agree with the manually defined ground truth for the keyword focusing task. In particular, the algorithm succeeds in selecting the appropriate level of hierarchy among the alternatives available in the segmentation results. The rest of the text is organised as follows. Section 2 reviews related fields of image analysis and conceptually introduces the proposed keyword focusing technique. In Section 3 we present an overview of some of the relevant principles and techniques in image content analysis. Section 4 discusses the role of the image segmentation subsystem in image content analysis, along with reviewing some image segmentation principles. In Section 5 we define and discuss the keyword focusing problem. We also propose and conceptually describe the statistical correlation method for keyword focusing on the conceptual level. Section 6 describes in detail the proposed technical implementation of the statistical correlation method. Section 7 reports the results of feasibility studies we performed with our implementation in two databases. In Section 8 we present our conclusions and future views.

V. Viitaniemi and J. Laaksonen: Focusing Keywords to Automatically Extracted Image Segments Using Self-Organising Maps, StudFuzz 210, 121–154 (2007) c Springer-Verlag Berlin Heidelberg 2007 www.springerlink.com

122

V. Viitaniemi and J. Laaksonen

2 Background In recent years the world around us has become increasingly visual. The technological evolution has made it possible to produce and store huge amounts of image data. The images can easily be indexed and searched according to some simple attributes or metadata, such as the date of archival. However, usually it is the content of the image that is more important for the future use of the image. When we imagine us humans describing an image, quite often we would list what different parts the image contains, and then possibly describe the major parts in more detail. In this light, partitioning the image to disparate parts and describing it in terms of the content and relationship of these parts appears to be a promising approach. Indeed, image segmentation is often a crucial part in computerised image understanding systems, e.g. [42, 34, 1]. Part-based object representations also provide the basis for many theories of object recognition in biological vision systems [43, 5, 31]. The traditional approach to image indexing has been to annotate images textually according to their contents. Then the database of annotations can be managed with conventional methods developed for textual databases. Unfortunately, the manual annotation method has severe shortcomings. In many cases the data volume is so huge that manual annotation is plainly impossible. In addition, the manual annotations can capture only a small subset of descriptions and interpretations of the images. The subjectivity and language specificity of manual annotations makes their use nontrivial. Against this background it is easy to see that automatic methods for content-based image characterisation would be highly desirable. The aim of content-based image retrieval (CBIR) research is to produce systems that automatically analyse the contents of given images, after which image databases can be queried by visual content. An important subproblem of CBIR is the automatic content analysis. Here one of the major challenges is the bridging of the large semantic gap between low-level image descriptors traditionally used in computer vision and the user’s desire to query the systems with high-level semantic concepts. The natural language, i.e. words, readily offers a symbolic representation of semantic concepts. Recently, the problem of matching words and images has attracted considerable research interest. Using textual annotations as proxy might offer a helpful approach to the semantic similarity assessment problem in CBIR. In the usual setting the word–image correspondence is regarded as a machine learning problem. The correspondence is learned from a set of annotated training images, after which the learned correlations can be used to automatically annotate any images. The correspondence between entities on different semantic levels can be seen as an instance of emergence [44]. Emergence is a process where a new, higher-level phenomenon results from co-operation of a large number of elementary processes. Thus, understanding and modelling emergence provides an

Focusing Keywords to Image Segments Using Self-Organising Maps

123

approach to overcoming the semantic gap. One essential element of emergence appears to be the involvement of large amounts data or processes. As there are lots of example data available where images are described with words, the image–keyword correspondence is a good example of a problem that can be naturally studied from the viewpoint of emergence. In this chapter, we show how soft computing techniques, more specifically the Self-Organising Map (SOM) [26], can be used in modelling the phenomenon of emergence. In the following we propose a mechanism for the emergence of semantic concepts from low-level features of image parts. We consider a learning problem that differs somewhat from the typical automatic annotation setting. We set the goal of the learning system to be keyword focusing, i.e. we consider the problem of identifying the areas in an image that correspond to the keyword when we already know that the keyword must correspond to some parts of the image. In this chapter we approach the keyword focusing problem using an unsupervised, data-driven method. We propose a two-stage mechanism that consists of a feedforward image representation front end, followed by an inference stage. The front end consists of image segmentation and feature extraction components, and a neural vector space quantizer utilising the SOM. The inference stage is based on finding statistical correlations between the annotating keywords and the representation formed by the front end. In addition to this baseline approach, we also consider interlinking the image segmentation step with the inference algorithm. In this case the unsupervised segmentation component produces a hierarchy of alternative tentative segmentations. The final selection among these segmentations is made during the keyword focusing process.

3 Image content analysis Large part of computer vision research can be regarded to revolve around the analysis of image content. In this presentation we consider some aspects of the content analysis problem starting from the needs of content-based image retrieval (CBIR), e.g. [41, 20]. In content-based image retrieval image databases are indexed with descriptors derived from the visual content of the images. Image content analysis is thus an essential subproblem of CBIR, even though it only seldom is addressed explicitly. Most CBIR systems are concerned with approximate queries where the goal is to find images visually similar to a specified target image. CBIR thus requires methods for both characterisation of image content and meaningful similarity assessment of the characterisations. A popular method of combining these two aspects is to employ the vector space model depicted in Figure 1. The idea behind the vector space model is to represent each image as a point or more generally, a collection of points, in a vector space. When a distance metric is defined for the vector space, the similarity assessment problem reduces

124


USER QUERY

IMAGE DATABASE FEATURE SPACE 1

Image 1

Query processing 2

Image 2

3

Image 3

Image representation Distance

comparison

SEARCH RESULT 2

1

3

Image 2 Image 1 Image 3

Fig. 1. In the vector space model of image retrieval both the database images and user queries are mapped into points in the feature (vector) space. The proximity of points in the feature space is taken as an indication of image similarity

to a geometric problem, e.g. finding points closest to the point that is the representation of a given example image or otherwise specified query. 3.1 Semantic levels of image content characterisation A central problem in content-based image retrieval is the semantic gap between the high-level semantic concepts we humans use in our reasoning and the low-level visual features directly available to computers for their information processing. This question can be elaborated by a categorisation of the depth of image content characterisation into three semantic levels, as proposed in [12]: Level 1 contains the characterisations by primitive features, such as colour, shape or spatial location of image regions. The features used on this level are both objective and directly obtainable from the image itself, without the need to resort to external knowledge bases. Most of the existing systems for general-purpose CBIR operate on this level. Level 2 uses derived (sometimes called logical) attributes, involving some sort of logical inference about the identity of objects in the image. A useful distinction can be made between two sublevels: a) identification of objects of a given type, and b) identification of individual objects or persons. Usually one needs to use some kind of an outside knowledge store to answer queries on this level.


125

Level 3 uses abstract attributes for image content characterisation. This means that significant high-level reasoning about the meaning and purpose of the objects in the image must be carried out. Also this level can usefully be divided into two: a) identification of named events or types of activity (e.g. finding pictures of Finnish folk dancing), and b) identification of pictures with emotional or religious significance (e.g. finding pictures of suffering). A need for image similarity queries on this level is often encountered in practice. Often (e.g. [21]) image retrieval based on the similarity on the semantic levels 2 and 3 is collectively termed as semantic image retrieval . In the practical systems of the present, the most significant semantic gap exists between the levels 1 and 2. 3.2 Feature types by spatial extent In the vector space model the visual content of images is represented by feature vectors. The feature calculation component of such an image analysis system thus considers the pixel representation of an image and forms a corresponding fixed-length feature vector according to some rule. In the keyword focusing application considered here we are interested in differentiating between the parts of the images. We therefore extend the model and allow an image to be represented with a set of feature vectors, each one corresponding to a part of the image. Approaches to feature calculation can be classified according to the way the image is segmented, i.e. how its pixels are divided among the set of feature vectors. The following characterisations are an augmented version of [41]: 1. Strong segmentation denotes the segmentation of an image according to real world objects. 2. Weak segmentation is the term used for data-driven grouping of an image into homogeneous regions. 3. Interest point detection identifies visually salient locations in an image. 4. Sign spotting reveals whether objects of (almost) fixed shape and a known semantic interpretation are present in an image and where they are located. 5. Partitioning divides images into geometrical areas, regardless of the image data. Grouping of pixels leads to various types of feature representation: 1. 2. 3. 4.

local features (correspond to sign spotting and interest point detection) object features (correspond to strong segmentation) region features (correspond to weak segmentation or partitioning) global features (a special case of partitioning where all pixels belong to the same region)

126


These categories vary in the extent they combine feature values from different parts of an image. In one extreme is the local feature approach which describes each found object and its feature values separately. The other extreme is the global feature approach, which combines feature values from the whole image. In the current context we are interested in correlating the semantic interpretations with locations in the images. The most relevant feature types to this end are local, object and region features. We limit ourselves into the automatic analysis of the image content. This rules out object features, for the bare reason that in a general setting, strong segmentation is still an unsolved problem (cf. also Section 4). On the other hand, by definition the connection between the features resulting from sign spotting and the semantics is already known by definition and is therefore uninteresting. The local features resulting from interest point detection are the first of the two remaining feature types. Such local descriptors are calculated for neighbourhoods of the interest points and are usually made invariant to some image transformations, such as rotation and scaling. Examples of local features include steerable filters [18], SIFT descriptor [32] and shape context [4]. For a review on the subject see [33]. Local features have proven to be successful in object recognition tasks, for example in the recent PASCAL Visual Object Classes challenge [13] many of the best performing approaches used the SIFT representation. When correlating the image locations with semantic interpretation of the content, the local feature based approach must overcome two challenges. Firstly, one interest point is usually not enough to robustly define an object, a configuration of multiple interest points must be considered simultaneously. The typical solutions to that problem do not conform easily to the vector space model. The second challenge is that it is desirable to attach semantic interpretations to whole image regions, not only to some isolated points such as corners. When using local features, a separate processing stage must be devised to perform this inference. Region features are the other remaining feature type that can be used for keyword focusing. In the subsequent sections we explore the application of region features, calculated for automatically obtained image segments, to the keyword focusing task. For the experiments we have used a large variety of different low-level features. A detailed description of them is given in Section 6.2.

4 Image segmentation for image content analysis In this section we will briefly review some of the key issues related to the use of image segmentation as a processing stage in an image content analysis system. We first discuss the requirements the segmentation subsystem should fulfill in order to be useful in this context. Then we review the principles behind


127

the existing segmentation methods. Finally, we shortly discuss the use of the segmentation results for image content similarity assessment. 4.1 Requirements for segmentation subsystem Generic, complete and to-the-pixel accurate unsupervised segmentation is virtually impossible [11]. Fortunately, less-accurate segmentations are useful enough for applications of image content analysis. Often it is enough to be able to locate salient regions of the image and calculate robust descriptions for them. In contrast, some types of visual descriptors, e.g. shape features, are more dependent on the segmentation accurately identifying the outline of objects or regions. Usually the desired result of image segmentation is a classification of each pixel as having exactly one segment label, i.e. a partitioning of the image. There exist some problems concerning such a disjoint labelling as it is quite unnatural and not in accordance with human perception. At the first glance more attention is certainly paid to larger regions, but subsequently also the smaller, salient image details are registered equally well. The “mental segmentation map”, that is, the decomposition of the image into its parts in the mind of the observer, is not a flat two-dimensional map, but some sort of hierarchy of image regions results. In that map, an image region may be a part of different objects on different levels of the hierarchy. As an example of hierarchically related regions we could think of a scene from the Orient in Figure 2. On the root level the image can be divided to three segments: sky, the fortress and the ground level (with the road and the people). The “fortress” segment is marked with a white outline in the leftmost figure. The fortress is further divided into wall and tower parts, the tower consists of two levels, and if we look closer at the upper level we can distinguish columns and two horisontal fascias. These steps are depicted in the rightmost column of Figure 2. There is no question whether the use of this kind of hierarchy information could potentially be beneficial. The issue is more like whether we are able to find a practical way to incorporate this kind of information into an image content analysis system. In Section 6 we will present our implementation of a keyword focusing system that utilises results of hierarchical segmentation. 4.2 Types of segmentation methods Local methods Local segmentation methods try to detect local discontinuities in image attributes. These local indications of segment boundaries can further be grouped together to form contours of objects. The problem with local methods is that information occurring on a larger scale is not used. The local

128


Fig. 2. Segment hierarchy of an image from the Orient. Subfigures display nested segments with increasing detail

neighbourhoods—often just a single pixel or a couple of pixels—are considered independently of each other and the presence of boundaries and their attributes are determined separately for each neighbourhood. After edge detection, contours—possibly closed—are identified by linking the found edge segments with each other. Often the information contained in the local neighbourhoods is not enough to reliably locate edges. In addition, the edge information from different local neighbourhoods may be contradictory or ambiguous. Area-based methods Area-based segmentation methods try to locate homogeneous areas in the images. One of the problems of this approach is to determine what is homogeneous enough to be considered as a single region. Another problem is the complete negligence of local edge information, resulting in difficulties in locating reasonable region boundaries even in the case of unambiguous edge information. Region growing, region merging and region splitting are widely-used areabased segmentation methods. In region growing new image pixels are appended to a seed region as long as the newly-created region still fulfills a homogeneity criterion. Region merging tends to merge adjacent regions if the resulting region stays homogeneous enough. Region splitting starts from in-


129

homogeneous regions. They are split until the homogeneity region is fulfilled. Often a combination of these area-based methods is used together. Global methods In traditional area-based methods the regions are considered separately or pair-wise at time. In contrast the global optimisation methods used in image segmentation aim at partitioning the image into disjoint homogeneous regions in an optimal way. When determining the optimality measure all the regions in the image are taken into account simultaneously. There are, however, three major difficulties in the global optimisation approach: 1. localisation of boundaries, 2. the tradeoff between the size and the number of regions, 3. the hierarchy of regions. Boundary localisation problems are similar to those with area-based methods. The tradeoff between the size and number of regions means that if the number of regions is chosen to be small, the average size of regions must be large. Then it is likely that the segmentation misses small but salient or homogeneous regions due to undersegmentation. On the other hand, if the number of regions is large, the image becomes oversegmented into many small regions so that large regions and structures are completely fragmented. As mentioned earlier, it would be desirable to have the segmentation algorithm to output the complete region relation hierarchy instead of only one level of it. In principle, there seems to be no reason why this could not be combined with the idea of global optimisation. However, it may not be straightforward to find a formulation that would both be rigorous and produce reasonable results. In methods with simple local decision rules the hierarchy would be easier to incorporate in some form. An issue also arises concerning whether the segmentation hierarchy should be optimised as a whole or whether single levels of the hierarchy could be optimised separately with some constraints forcing the adjacent levels to be compatible. 4.3 Use of segmentation in image content analysis Let us now assume that our images have been segmented and the contents of the individual image segments described. How should the content of the images then be characterised, and how should the characterisations be compared? The latter problem arises particularly in CBIR, where the similarity of the characterisations needs to be assessed. As long as the interest lies in individual segments of images, no special techniques are needed. Often, however, the goal of the system is not to assess similarity of individual segments but of images in whole. Then a method must be devised to propagate the segment similarity assessments to the image level. A straightforward alternative is to treat the segments as independent images. In that

130


way the information about relationships between the segments—even about co-occurrence in the image—is lost. Furthermore, the user might be interested in whole images instead of segments. Another approach is to let the user define her interest in terms of example segments, e.g [7]. This, however, could be laborious, as example images contain several segments. A solution to this is to form an image similarity measure automatically from the segment similarities without segment-level user intervention [46, 8, 24]. A more rigorous alternative is to use generative models such as in [3, 6] to describe the joint distribution of semantic concepts and a blob representation of the image segments and then relegate the query to the concept level. In the present application of keyword focusing, half of the issue becomes redundant as the objects of interest are indeed image segments. Still, we are left with the issue of converting image level examples to the realm of image segments.

5 Focusing keywords in image segments The main topic of our present work, focusing of keywords to image segments, is addressed in this section. In Section 5.1 the keyword focusing task is defined and its potential application areas discussed. Section 5.2 reviews the use of related techniques in some of the literature. In Section 5.3 we propose a specific technique, statistical correlation method, for accomplishing the task of keyword focusing. Finally, some characteristics of the statistical correlation method are discussed in Section 5.4. 5.1 Task of keyword focusing In the keyword focusing problem the input is a set of images, all of which are annotated with a single keyword. The goal is to find the areas of the images that correspond to the keyword. This can be considered as an unsupervised machine learning problem: no labeled data is given that directly pinpoints the appropriate image locations. In our solution to the problem we additionally allow the learning system to use unlabeled auxiliary image data that is nonspecific to any given keyword. The auxiliary data can be considered as part of the system in the sense that it remains the same regardless of the particular keyword at hand. Related to the intra-image keyword focusing, where the task is to compare different parts of the same image, is the problem concerning the database-wide identification of regions corresponding to a keyword. We denote this databasewide ranking of locations according their likelihood to correspond to a specific keyword with the term database-level keyword focusing. Intra-image keyword focusing is a subproblem of the database-level keyword focusing in the sense that a solution to the latter problem gives also an answer to the first problem. One may also argue that solving the intra-image keyword focusing problem is


131

a prerequisite to solving the database-level counterpart. This, in turn, explains why keyword focusing can be regarded as a potential tool in solving a part of the CBIR problem. There are numerous collections of images available that can potentially be used as training data for the focusing problem with minimal preparation. One prototypical example is formed by the commercial illustration image databases that are annotated with image-level keywords, for example the Corel image gallery [10]. Many museums have annotated collections of digital images (e.g. [14]). Also any image collection that is partitioned into classes can be used by considering each class to represent a keyword. The images in the World Wide Web along with the text they appear next to form a more speculative and an overwhelmingly large instance of the keyword focusing problem. The focusing problem is defined for all keywords, but of course a sensible solution can be expected only for keywords that in reality can be localised into some part in an image. For example, focusing the keyword “evening” to a specific part of an image is usually senseless. 5.2 Keyword focusing in image content analysis Learning the image–word correspondence has attracted considerable research interest recently. Often the motivation has been to learn the correspondence from a set of training images and then apply it to the automatic annotation of a new set of unlabeled images. For the automatic annotation research the keyword focusing is mostly a by-product, whose results are not explicitly stated or analysed beyond its effect on the annotation performance. This is reasonable since pairing images with keywords is somewhat different problem than focusing the keyword further down to a certain location inside the image. On the image level the prediction is often easier as the various objects in the images are correlated. For instance, airplanes often appear together with sky. Yet the location of sky in the images should not be called “airplane”. The automatic annotation of images is more directed to database-level relevance keyword problem than to intra-image keyword focusing. Some of the models for automatic image annotation consider the image as a whole (e.g. [38]) or use a rough geometrical division of the image (e.g. [35]). In [30] a geometrically formed multi-resolution two-dimensional hidden Markov model is used for modelling the images, but concepts are correlated with the model as whole, not with its individual constituent parts. A more popular approach, however, is to formulate the models more directly in terms of image segment– keyword correspondence (e.g. [36, 3, 37, 16, 15, 6, 23, 19, 45]). In a broader sense, any image analysis or classification task can be seen as involving kind of keyword focusing if the problem solution includes the identification of relevant image locations. In such settings, the existence of a keyword is not explicit but implicit. A straightforward approach to the focusing problem is a bottom-up or feedforward process where the input images are first segmented and the sub-

132


sequent focusing is then reduced to selecting the one or ones among these segments that correspond to the keyword. An attempt to interlink the segmentation and focusing phases is given in [6] where a Markov random field (MRF) model combines the subtasks of segmentation on a coarse grid and assigning the keywords to segments. On the other hand, even this approach can be considered feedforward by regarding the coarse grid as a fixed segmentation result and the MRF algorithm as the focusing stage. In Sections 6.1 and 6.5 we propose an approach that also takes a step into this direction. In the proposed method we consider producing a hierarchical segmentation, among whose levels the focusing stage selects the most appropriate one of the alternative segmentations. 5.3 Statistical correlation method Our proposed approach to the keyword focusing problem is based on statistically correlating the keywords and image segments. For this we need a set of training image data that consists of example images of the particular keyword class and an auxiliary image collection. The outline of the approach is the following: 1. Automatically segment the studied images. 2. Form feature space representations for the image segments. 3. Identify feature space regions that are more densely populated by the example image segments than by the auxiliary image segments. 4. Find the example image segments with feature representations in the relatively dense regions of the feature space and associate them with the keyword. As we see, two mechanisms are responsible for the working of the focusing strategy: (1) the effect of concentration of the example image segments in certain regions of the feature space, and (2) the negative influence of the auxiliary data in regions where it concentrates. The approach is thus qualitatively similar to the term frequency – inverse document frequency (TF-IDF) formula [39] successfully used in natural language (text and speech) retrieval. Also the TF-IDF formula awards high correlation scores to terms that appear often in the relevant documents (example images), but such terms are punished that appear often also in the reference corpus (the auxiliary images). Different from the natural language processing, in image processing the representation of documents is not readily available in the form of words, but must be constructed. In this sense the segment boundaries produced by the segmentation algorithm correspond to the set of words in a document on the syntactical level, and the feature representation of the segments corresponds to the actual identity of the words, i.e. the semantic level. The segments and their feature representations form together a blob representation [7] of the image. Blob representations are often used in image analysis, for example in automatic image captioning [3, 6].


133

The purpose of the auxiliary data in the focusing algorithm is to give a basis of comparison for determining whether a certain region of feature space is more commonly populated by the actual example data than by images in general. The discussion in [13] gives some perspectives to the issue of selecting appropriate auxiliary data. In principle, a collection of general domain images could be used as auxiliary data regardless of the specific focusing task. However, if the example images are known to come from a restricted domain, using a narrower domain data set as auxiliary data as well will probably result in better focusing performance. 5.4 Discussion In the preceding we have approached the keyword focusing problem as a data mining problem. We are interested in discovering new, previously unknown relationships in the studied data. In this case, the novel discoveries are the correlations of specific parts of the studied images with the keywords. This approach can be contrasted with a pattern recognition approach, where the goal is to learn a model from a labeled training data set and generalise it to an unlabeled test set. This distinction corresponds to the one between unsupervised and supervised learning. A shortcoming of the statistical correlation principle is that inadequate example data is likely to lead to spurious correlations. For instance, if the example images are such that the sun and buildings always coincide, the focusing algorithm can not be expected to tell these apart. When the correlations between objects are artefacts of the example set, increasing the number of example images can remove the ambiguous situations. Still, for genuine correlations such as “sky” and “airplane”, the required amount of example data could get impractically high before enough separate occurrences of the keywords would be observed. Especially serious this problem can be in object–part hierarchies. For instance, motorbike wheels without a whole motorbike are rare in images, although also such images exist. It would seem that the statistical correlation learning should be augmented in these cases with some other learning principle that could take such object hierarchies more directly into account.

6 System implementation This section describes in detail the implementation of the system we propose for the unsupervised keyword focusing task. The system is implemented inside the PicSOM1 CBIR software framework [27, 28, 29]. As an input, the system takes two sets of images: a set of images annotated with a certain keyword 1

http://www.cis.hut.fi/projects/cbir/

134


(positive examples), and a set of auxiliary background images (negative examples) As the result of the processing the system produces a segmentation of the positive example images and ranks the segments according to their relevance to the keyword, i.e. the likelihood of the segments to correspond to the given keyword. The proposed system consists of a feedforward pre-processing stage, followed by an inference stage. In the preprocessing stage, both sets of images are first hierarchically segmented and statistical visual features extracted from the segments. The features are grouped into multiple feature spaces that are finally quantized using a variant of the Self-Organising Map (SOM) [26]. The inference stage implements the statistical correlation method for keyword focusing simultaneously for parallel quantized feature spaces. As a post-processing step the produced ranking of the segments is re-ordered in an additional relevance propagation step so that the hierarchy information in the segmentation results is explicitly taken into account. As a result, the system is able to automatically select the most appropriate level of hierarchy in the hierarchical segmentations. The rest of the section is organised as follows. Sections 6.1 and 6.2 discuss the hierarchical image segmentation and feature extraction methods, respectively. Section 6.3 describes the use of SOM in quantizing the feature spaces. In Section 6.4 we delineate the implementation of the statistical correlation principle. Section 6.5 describes the algorithm propagating the relevance within the segment hierarchy. 6.1 Automatic image segmentation For the current experiments we have used a generic image segmentation method which is simple and somewhat rudimentary. Referring to the taxonomy of Section 4.2, the method is in essence a hybrid of area-based region merging combined with a local edge heuristics. The method partitions the images to a fixed number of segments that are homogeneous in terms of average colour in the CIE L*a*b* colour space [9]. The images in the database are segmented in two steps. In the first step ISODATA variant of K-means algorithm [40] with a K value 15 is used to compute an oversegmentation based on the colour coordinates of the pixels. This step typically results in a few thousand separate segments. In the second step the segments are merged. The difference dLab (r1 , r2 ) in the average CIE L*a*b* colour of regions r1 and r2 is used as the basis for the merging criterion. In addition, the multi-scale edge strength e(r1 , r2 ) between the regions is also taken into account. The final merging criterion C is weighted with a function s of the sizes |ri | of the to-be-merged regions ri : C(r1 , r2 ) = s (r1 , r2 ) dLab (r1 , r2 ) + γe (r1 , r2 ) , (1) where


135

s(r1 , r2 ) = min(|r1 |/|I|, |r2 |/|I|, a) + b

(2)

is the size-weighting function, |I| is the number of pixels in the image and γ, a and b are parameters of the method. The values for the parameters have been selected to give visually feasible results for photographs and other images in earlier applications. The same values (γ = 40, a = 0.02, b = 0.002) have been used also in the current experiments. The merging is continued until the desired number of regions are left. In addition to these leaf segments, we also record the hierarchical segmentation that results from running the region-merging algorithm on the leaf segments until only one region remains. Such composite segments are considered in our last experiments alongside with the leaf segments. Figure 3a shows an example of a segmented image and Figure 3b the corresponding segmentation hierarchy. 0,1,2,3,4,5,6,7 0

1,2,3,4,5,6,7 1,2,5,6,7

7

3,4

1,2,5,6 6

3

4

1,2,5 1,5 1

2

5

(b)

Fig. 3. Example of a segmented image. Subfigure (a) displays the eight leaf segments found by the segmentation algorithm. Subfigure (b) shows the segmentation hierarchy resulting from the continued region merging. Leaf segments are circled in the tree

6.2 Statistical image features The PicSOM system implements a number of methods for extracting different statistical visual features from images and image segments. These features include a set of MPEG-7 content descriptors [22, 29] and additionally some non-standard descriptors for colour, shape and texture. Colour Of the used MPEG-7 descriptors Color Layout, Dominant Color and Scalable Color describe the colour content in image segments. In addition to the

136


MPEG-7 colour descriptors, both the average colour in the CIE L*a*b* colour space [9] and three first central moments of the colour distribution are used as colour features. Shape Besides the MPEG-7 Region Shape, the shape features include two nonstandard descriptors. The first consists of the set of the Fourier descriptors for the region contour [2]. Fourier descriptors are derived from the following expansion of the region contour: z(s) =

∞

zn e

2πins L

.

(3)

n=−∞

Here the Cartesian coordinates of the contour are represented by the real and the imaginary parts of the complex function z(s), parametrized by the arc length s. The resulting feature vector includes a fixed number of low-order expansion coefficients zn . The coefficients are then normalised against affine image transformations. In addition, the high-order coefficients are quadratically emphasized. The second non-standard shape descriptor is formed from the Zernike moments [25] of the region shape. The Zernike polynomials are a set of polar polynomials that are orthogonal in the unit disk. The Zernike moments Anm are given by the expansion coefficients when the polar representation of the region shape is represented in the basis of Zernike polynomials: Anm =

n + 1 I(x, y)Vnm (ρ(x, y), θ(x, y)) , n − |m| even. π x y

(4)

Here n is the order of the moment, m the index of repetition, i, j are the rectangular image coordinates, and ρ, θ the corresponding polar coordinates. I(x, y) is the binary representation of the region shape and Vnm is the Zernike polynomial: Vnm (ρ, θ) = Rnm (ρ)eimθ n−|m| 2

Rnm (ρ) =

s=0

(5) (−1)s (n − s)!

s!( n+|m| 2

−

s)!( n−|m| 2

− s)!

ρn−2s .

(6)

The feature vector includes coefficients Anm up to order a selected order. The feature is normalised against translation and scaling by fitting the region inside the unit disk. Rotation invariance is achieved by taking the absolute values of the coefficients.


137

Texture We have used MPEG-7’s Edge Histogram descriptor to describe the statistical texture in image segments. For non-standard description of a region’s texture the YIQ colour space Y-values of the region pixels are compared with the values of their 8-neighbours. The feature vector describes the statistics of the resulting distribution. 6.3 Quantizing the features with self-organising maps The visual features are disjointly partitioned into feature spaces, each feature space corresponding to the components of one visual descriptor. For instance, the components of MPEG-7 Color Layout descriptor form one feature space. The feature spaces are quantized using a variant of Self-Organising Map algorithm. The SOM is an unsupervised, self-organising neural algorithm widely used to visualise and interpret large high-dimensional data sets. The SOM defines an elastic net of points that are fitted to the distribution of the data in the input space. The SOM consists of a two-dimensional lattice of neurons or map units. A model vector mi ∈ Rd is associated with each map unit i. The map attempts to represent all the available observations x ∈ Rd with optimal accuracy by using the map units as a restricted set of models. During the training phase, the models become ordered on the grid so that similar models are close to and dissimilar models far from each other. When training a SOM, the fitting of the model vectors is carried out by a sequential regression process, where t = 0, 1, 2, . . . , tmax − 1 is the step index: For each input sample x(t), first the index c(x) of the best-matching unit (BMU) or the “winner” model mc(x) (t) is identified by the condition ∀i : x(t) − mc(x) (t) ≤ x(t) − mi (t) .

(7)

The distance metric used here is usually the Euclidean one. After finding the BMU, the vectors of map units constituting a neighbourhood centered around the node c(x) are updated as mi (t + 1) = mi (t) + h(t; c(x), i)(x(t) − mi (t)) .

(8)

Here h(t; c(x), i) is the neighbourhood function, a decreasing function of the distance between the ith and c(x)th nodes on the map grid. This regression is reiterated over the available samples and the value of h(t; c(x), i) is let decrease in time to guarantee convergence of the model vectors mi . Large values of the neighbourhood function h(t; c(x), i) are used in the beginning of the training for initialising the network, and small values on later iterations are needed for fine-tuning. After the training, any vector in the feature space can be quantized to a two-dimensional index by its BMU on the SOM.

138


6.4 Implementation of statistical correlation inference The implemented system performs statistical correlation separately in each of the feature spaces. For an image segment, this results in a relevance score for each of the feature spaces, as will be described below. A combined relevance score is then formed by summing the scores of all the feature spaces. For the computational implementation of the statistical correlation principle, all the positive image segments are projected to all the feature SOMs. For each unit the number of segments projected to that particular unit is counted. The counts form a sparse value field on the SOM surfaces. Due to the SOM’s property of mapping similar objects in nearby map units, we are motivated to spatially spread these sparse values by a low-pass filter, i.e. to convolve them with a smoothing kernel. The size and shape of the convolution kernel is selected in a suitable way in order to produce a smooth value map. In the resulting map each location is assigned a relevance value according to the number of positive objects mapped to the nearby units. This process of obtaining smooth relevance maps can be seen as nonparametric density estimation of the class of positive images in the latent spaces of the SOM grids. After forming the positive relevance map for each SOM surface, the same procedure is repeated with the negative examples. These negative examples are obtained from the auxiliary or background images. Then the estimates of the positive Pi+ (x, y) and negative Pi− (x, y) densities in SOM coordinates (x, y) of the feature SOM i are combined by map-wise weighting and subtraction: Pi (x, y) = Pi+ (x, y) − λPi− (x, y) .

(9)

Here λ is a free parameter of the method. The appropriate value of the parameter seems to be a function of the sizes of the positive and negative example image sets. At this point each of the SOMs has a relevance map associated with it. For each image segment, a final relevance score is then calculated by summing the relevance scores of the segment’s BMUs on all the feature SOMs. Those features that seem to distinguish well between positive and negative example images are in this process implicitly weighted more than the others. This is because the good distinction ability results in dense clusters of example images on the corresponding SOMs and high concentrations of relevance in the kernel estimation. If the images are scattered evenly on the map, the local peaks of relevance are much smaller in amplitude. 6.5 Propagating relevance within segment hierarchy To augment the implementation of the statistical correlation principle, we implement a mechanism for propagating relevance scores along the segmentation hierarchy within a single image. The propagation takes place after the


139

relevance of individual segments has been evaluated by the SOMs. The statistical correlation is only able to indicate whether an individual segment or segment combination is relevant to the keyword. In contrast, the propagation algorithm simultaneously considers the relevance of several of the segments and their combinations that appear in the hierarchical segmentation of an image. By explicitly using the hierarchical relationship of the segments, the propagation algorithm is able to identify the largest combination of segments that is likely to correspond to the keyword. Further motivation for the propagation step stems from the keyword focusing setting where we project both the leaf and the composite image segments, contained in the beforehand formed segmentation hierarchy (see Figure 3), to the same feature SOMs. We would then let the relevance scoring select the most appropriate scale of segmentation among the alternatives. Many of the useful low-level features are averaging, by which we mean that e.g. the average colour of a segment formed by combining two blue segments is still blue. We would like the relevance scoring mechanism to favour the merging of segments to a composite segment if they are similar in the sense of an averaging feature. However, some features are not averaging, e.g. the shape features of a composite region are not weighted averages of the constituent leaf regions. The composite shape will in some cases be characteristic to the positive example images while the shapes of the constituent regions are not. The scoring mechanism should favour also this type of composite segments for non-averaging features. We would want the scoring mechanism not to use any a priori information whether the individual features are averaging or not. We require that the scoring method favours combining of segments that are relevant and whose combination is also relevant. On the other hand, also such combined segments should be favoured whose children are not relevant, but the combined segments themselves are. Considering the two requirements, a non-linear score propagation mechanism is needed for post-processing the relevance scores of the segments in the hierarchy after the SOM-based relevance assessments. To this end, the proposed system implements a simple multiplicative model for propagating the relevance score pi of segment i upwards in the segmentation hierarchy like the one seen in Figure 3. The propagation is initialised by setting pi = ri for the leaf segments. Here ri denotes the score obtained by normalising the relevance scores ri within the image Ii containing the segment i: ri − r¯i maxIj =Ii rj − r¯i 1 r¯i = rj |{j|Ij = Ii }| ri =

Ij =Ii

For composite segments, the propagation rule is given by

(10) (11)

140


pi =

+ri (ri + j∈children(i) p+ j ) , if ri > 0 − −ri (ri + j∈children(i) pj ) , otherwise.

(12)

The set children(i) refers to all segments j to whom i is the immediate parent. Furthermore, p+ i = max(pi , 0) − pi = min(pi , 0).

7 Experiments We have tested the feasibility of our approach to keyword focusing with two image databases. As the input we use, along with the images of the databases, the knowledge about each image being either annotated or not annotated with a specific keyword. The method is unsupervised: it does not learn to reproduce the image-level training labels but extracts qualitatively different labelling—labels for the image segments. The method’s performance is evaluated by comparing the data-driven segment labels with manually defined ground truth. In these experiments, the parameters of the proposed method, most importantly the used visual features, have been selected to optimise the performance in the test tasks. For a quantitative study, this would not be an adequate procedure for parameter selection. Some other means—such as optimisation with an independent data set—should be used. However, the used procedure serves to demonstrate the viability of our approach if the parameters are chosen appropriately. The rest of the section is organised as follows. In Section 7.1 the methods for performance evaluation are described. Section 7.2 describes the two databases used. In Section 7.3 we gain insight to the inner workings of the proposed method by looking the SOM surface distributions corresponding to quantized representations of different feature spaces. Section 7.4 presents more quantitative results as the method’s outputs are compared to ground truth via Receiver Operating Characteristic (ROC) curves. 7.1 Performance evaluation To evaluate the system’s performance in the focusing tasks, we have manually defined a ground truth against which the system’s performance is compared. To this end, we have first applied the automatic segmentation algorithm for the images. Then we have manually annotated the segments in those images which have been annotated with the studied keyword. In this annotation, we have marked which of the segments cover the object the keyword refers to. This annotation has been used only as a ground truth for the performance evaluation.


141

In the last experiments we also evaluate the system’s ability to select the appropriate level in the segmentation hierarchy among all the alternatives produced by the hierarchical segmentation algorithm. To this end we define another set of segment-level ground truth annotations—the best segment. The best segment is defined to be the segment combination contained in the automatically produced hierarchical segmentation that best resembles the annotating keyword. Of course, choosing the best alternative is usually somewhat subjective, especially as the results of the hierarchical segmentation are often quite far from ideal. However, the task of selecting the best segment defined in this manner still captures an important aspect of the performance evaluation problem. After having defined the ground truth classes for performance evaluation, we measure the system’s performance with receiver operating characteristic curves. For the shown figures, we have first generated separate ROC curves for each database image annotated with the particular keyword. The curves have then been averaged by considering the true positive rate to be a function of the false positive rate. Thus, the curves are averaged graphically along the vertical direction. Generating a separate curve for each image is motivated by the fact that here we consider the task of focusing the keyword inside individual images. The ordering of the segments in different images is not regarded as relevant. The averaging procedure may not correspond to any actual classification scenario, but is adequate to get a measure of average performance for different images. For the linear combination coefficient λ in Eq. (9) the value λ = 1/30 was found to produce good results in the current experiments. This means that the effect of an individual negative example segment is considerably smaller than the effect of a positive segment. However, since the negative example images were much more numerous, the resulting weight of the distribution of negative segments is much larger than that of the positive examples. 7.2 Databases The first of the two databases has been used to provide an easily understandable lightweight testbed for the framework. For this purpose, we have selected a 900 image subset of the commercial Corel database [10]. The images depict people, most of them models. We thus call this database the models database from here on. As the keyword to be focused we chose the word “red”. We have manually annotated the database for this keyword by defining the keyword for an image if it contained a salient red object captured by the automatic segmentation algorithm. 107 images of the database were judged to portray red objects. The chosen keyword has straightforward connections to both image segmentation and the visual features, and we thus hope the results to be straightforward to interpret. This way we could focus our attention solely to the keyword focusing mechanism, not the image segmentation. As the visual features for this database we used one colour feature, colour moments, and

142


one local texture feature, the MPEG-7 Edge Histogram (cf. Section 6.2). We expect to observe a different behaviour of these two features in the keyword focusing task as colour is directly related to the target keyword whereas the edge histogram is not. The second database we used is the 101 Object Categories database [17] of the PASCAL Visual Object Classes Challenge 2 . The database contains 9197 images divided into 101 semantic categories, each containing between 31 and 800 images, and a background or auxiliary class of 520 miscellaneous images. The database has been created mostly for object recognition purposes and therefore does not contain detailed image-wise annotations. For the experiments, we chose one of the categories, “lobster”, as the keyword to be focused. A lobster is portrayed in 41 of the database images. The keyword does not have a direct connection to the image segmentation algorithm or to any specific feature representations as in the case of the first database. For this database the set of visual features is selected by considering the concerted ROC curves for the focusing tasks. In these curves all the segments of the images corresponding to the keyword are ordered in a common list according to their relevance scores. Features are added to the set until further addition of features no longer improves the equal error rate (EER) of the ROC curve. 7.3 Feature Distributions on SOM Surfaces Models Database Figure 4 displays how the feature distributions of the example segments become projected on the feature SOM surfaces in the case of the models database. The map surfaces have toroidal topology, i.e. the top edges are connected to the bottom edges and the left edges to the right edges. This way the visual analysis is more straightforward as we do not have to consider the edge effects that can be significant in rectangular SOMs. The actual focusing performance of the both topologies is according to our observations approximately the same. Distribution of the colour feature is shown in the left column and that of the edge histogram feature in the right column. The input densities to the algorithm are densities on the rows (a) and (b). The row (c) is the outcome of the algorithm, and row (d) can be seen as the desired outcome. However, note that outside the feature space regions where there are segments on row (a), the final qualification values on row (c) have no significance to the outcome of the keyword focusing task. As expected, the colour feature captures well the keyword “red”, as indicated by the dense concentration of the positive example segments to specific map surface regions. The segments are more widely spread on the edge histogram map surface. Furthermore, by comparing the distributions of true “red” segments and all the keyword segments, we note that the distributions 2

http://www.pascal-network.org/challenges/VOC/


(1a)

(2a)

(1b)

(2b)

(1c)

(2c)

(1d)

(2d)

143

Fig. 4. Distributions of models database segments on two feature SOM surfaces. The left column (1) shows the distribution of colour feature, the right column (2) the distribution of MPEG-7 Edge Histogram feature. Dark colour corresponds to high density. Note that the distributions are normalised so that the maximum value always corresponds to black colour. Therefore the shades in different subimages are not comparable, only the shapes of the distributions. The top row (a) displays the distribution of all the segments in the images that are annotated with the keyword “red”. The second row (b) shows the distribution of all the segments in the models database. The third row (c) shows the linear combination of the first and the second row according to Eq. (9) which is used as the final qualification value of the segments with respect to that feature. The relevance is spread on the second and third row by the convolution mechanism discussed in Section 6.4. The fourth row (d) shows the distribution of manually confirmed “red” segments (true positives)

peak at approximately same locations corresponding to the truly “red” segments. This happens even though the majority of the keyword segments are false positives, i.e. they are not “red”. This is explained by the fact that the non-red segments in the example images are distributed throughout the colour feature space. Therefore, in any specific region of the feature space their con-

144


centration is still low and easily dominated by the locally peaking distribution of the true positives. 101 object categories Figures 5 and 6 display the projections of the segments on the feature SOM surfaces in a manner similar to Figure 4. In addition, however, the figures include the distribution of best segments (cf. Section 7.1) on row (e) as we also consider the problem of finding the most representative combined segment among the segments in the hierarchical segmentation. From the figures it can be seen that, in addition to the concentration of positive example segments, the application of Eq. (9) and subtraction of the background density due to the auxiliary images (row (b)) can be essential to the focusing performance. For instance, by comparing the distributions on the first and last rows of the Fourier feature (column (3) of Fig. 5), we notice areas of false positives in the lower part of the first-row map near the left and right edges. Successful keyword focusing requires suppression of these regions and the subtraction of the background relevance seems to offer an effective means for achieving this. On the other hand, by comparing rows (a) and (d) of the texture feature (column (4) of Fig. 5), we see that keyword focusing would require suppression of the lower right quadrant of the first-row SOM surface. However, the background density is quite low in that region and therefore the region gets amplified in application of Eq. (9). This leads one to expect the texture feature to perform poorly in keyword focusing, which is indeed confirmed to be the case. It can be seen that the shape features (columns (3) and (5) in Fig. 5 and columns (3) and (4) in Fig. 6) are promising candidates for favouring the best segments over other “lobster” segments. The large values of background relevance coincide with some regions that are more pronounced in the distribution of all “lobster” segments (row (d)) than in the distribution of the best “lobster” segments (row (e)). 7.4 Performance in keyword focusing Models database Figure 7 shows some examples of image segmentation and Figure 8 the ROC curve, when the keyword focusing experiment applied onto keyword “red” in the models database. In Figure 7c the focusing algorithm erroneously considers the segment 4 to be more “red” than the segment 5. This can be explained by the unusual shades of red and some dark areas in segment 5. The almost ideal ROC curve of Figure 8 indicates the performance of the system to be very satisfactory in general, with some rare exceptions. This is also confirmed by manual inspection of the focusing results of the individual


(1a)

(2a)

(3a)

(4a)

(5a)

(1b)

(2b)

(3b)

(4b)

(5b)

(1c)

(2c)

(3c)

(4c)

(5c)

(1d)

(2d)

(3d)

(4d)

(5d)

(1e)

(2e)

(3e)

(4e)

(5e)

145

Fig. 5. Projections of segments associated with keyword “lobster” on 101 Object Categories database to the feature SOM surfaces of the non-standard feature spaces. Columns: (1) colour moments, (2) average colour, (3) Fourier shape, (4) texture and (5) Zernike moments. Rows: (a) all segments of images annotated with keyword lobster, (b) all segments in the database, (c) the combination of (a) and (b) according to Eq. (9), (d) true “lobster” segments, (e) best “lobster” segments

images. We can thus confirm that when the feature spaces, image segmentation and the studied keyword are compatible, the statistical correlation method is an effective means for keyword focusing.

146


(1a)

(2a)

(3a)

(4a)

(5a)

(1b)

(2b)

(3b)

(4b)

(5b)

(1c)

(2c)

(3c)

(4c)

(5c)

(1d)

(2d)

(3d)

(4d)

(5d)

(1e)

(2e)

(3e)

(4e)

(5e)

Fig. 6. Projections of segments associated with keyword “lobster” on 101 Object Categories database to the feature SOM surfaces of MPEG-7 feature spaces. Columns: (1) Color Layout, (2) Dominant Color, (3) Edge Histogram, (4) Region Shape and (5) Scalable Color. Rows: (a) all segments of images annotated with keyword lobster, (b) all segments in the database, (c) the combination of (a) and (b) according to Eq. (9), (d) true “lobster” segments, (e) best “lobster” segments

101 object categories The system’s keyword focusing performance with keyword “lobster” in the 101 Object Categories was evaluated separately in two tasks: (1) identifying any lobster segments in the segmentation hierarchy, and (2) selecting the

Focusing Keywords to Image Segments Using Self-Organising Maps 7

6

6

3

4

8

3

2

8

7 2

5

147

5 3

4 6

4

1

1

2

(a)

1

5 7

8

(b)

(c)

Fig. 7. Examples of focusing the keyword “red” in the models database. The white borders in the image indicate the eight regions found by the segmentation algorithm. The number tags reflect the ordering of the segments produced by the focusing algorithm. The tags of truly red segments (defined for evaluation purposes only) are shown with double border Averaged ROC curve of focusing the keyword red 1

True positives

0.8

0.6

0.4

0.2

0 0

0.2

0.4

0.6

0.8

1

False positives

Fig. 8. The averaged ROC curve of focusing keyword “red” in the models database

single best segment combination from the hierarchy. Both of these tasks were performed with and without the intra-image relevance propagation mechanism (cf. Section 6.5). This gives four variants of the problem altogether. For each problem variant we state the results of using the set of features that was found to perform best. The optimal feature sets were found to be different for each problem variant, although the performance was not strongly

148


dependent on the choice of features. The robustness can be partially attributed to the PicSOM system’s ability to automatically emphasise the most useful features. Table 1 shows the optimal feature sets in the ROC EER sense, listed in order of decreasing significance. Table 1. The feature sets optimised for variants of the focusing task No propagation Any segment colour moments Fourier shape average colour MPEG-7 Edge Histogram MPEG-7 Color Layout

With propagation colour moments average colour Fourier shape MPEG-7 Dominant Color MPEG-7 Color Layout MPEG-7 Scalable Color MPEG-7 Edge Histogram Best segment colour moments average colour MPEG-7 Edge Histogram MPEG-7 Edge Histogram Zernike Moments Zernike moments MPEG-7 Scalable Color

Figure 9 provides some example cases of keyword focusing. In general, the performance of the system in this task is satisfactory, although there are cases where the system does not function as well as desired. In many cases of failure, the reason can be tracked down to the unsatisfactory segmentation of images. The lowermost row (c) of Figure 9 exemplifies such a situation. The white background and kitchen tool cause the lobster to divide into two parts and the segmentation algorithm does not even consider the merging of these regions. Comparison of columns (2) and (3) in Figure 9 shows the effect of the relevance propagation algorithm. On the rows (a) and (c) the typicality in the parallel feature spaces alone has been enough to capture the proper ordering of the “lobster” segments (marked with a +), even placing the best segments (marked with a *) on the top of the lists. On the row (b), however, the relevance propagation step is required for the correct re-ordering of the list. Figure 10 shows the ROC curves for three cases of the keyword focusing task. The subfigures (a) and (b) correspond to the four variants identified in the Table 1. It can be noted that the propagation of relevance along the segmentation hierarchy improves performance in finding the single best segment in (b), but does not significantly affect the performance in the task of finding any lobster segments in (a). This was to be expected, as the rationale for the relevance propagation is to re-order the segments that were found to be relevant so that the most representative segment combinations are favoured. Figure 10c shows the algorithm’s performance in finding the best segment (* in Fig. 9) among the true lobster segments (+ in Fig. 9). This way the the effect of the algorithm’s performance in finding any lobster segment among

Focusing Keywords to Image Segments Using Self-Organising Maps 2

0

3

1 4 5

6

(1a)

7

* 3+ 0,1,2,3,4,5,6,7 0,1,2,3,4,6,7 1,3,4,6,7 1,2,3,4,6,7 1,3,4,6 4 0 7 5 2 1 1,4 1,4,6 6

(2a) 1 1,4+ * 1,2,4+ 1,2,4,6+ 7 1,2,4,5,6,7 2+ 5,7 5+ 4+ 6+ 3 0,1,2,3,4,5,6,7 0,3 0

(2b) * 1,2,4 + 2,4 + 3+ 6+ 2+ 1+ 4+ 0,3,5,6,7 0,3,5,7 0,5,7 0,1,2,3,4,5,6,7 5 0,7 0 7

(2c)

149

* 3+ 1,3,4,6 0,1,2,3,4,5,6,7 0,1,2,3,4,6,7 1,3,4,6,7 1,2,3,4,6,7 4 0 7 5 2 1 1,4 6 1,4,6

(3a) * 1,2,4+ 1,2,4,6+ 1,4+ 1,2,4,5,6,7 1 7 5,7 2+ 5+ 4+ 6 3 0 0,1,2,3,4,5,6,7 0,3

(3b) * 1,2,4+ 2,4+ 3+ 6+ 2+ 1+ 4+ 0,3,5,6,7 5 0,1,2,3,4,5,6,7 0 7 0,3,5,7 0,7 0,5,7

(3c)

Fig. 9. Examples of focusing the keyword “lobster” in the 101 Object Categories database. The white borders in the images in column (1) indicate the eight regions found by the segmentation algorithm. The numbers in the tags are arbitrarily chosen segment labels. Columns (2) and (3) list the ordering of the segments output by the focusing algorithm. Column (2) shows the algorithm results without relevance propagation along the segmentation hierarchy. In column (3) the propagation is included. The segments more likely to be associated with the keyword “lobster” are on the top of the lists. In the lists segments marked with + have been manually judged to consist of mostly lobsters. The asterisk (*) beside a segment label indicates that the segment has been manually judged as the best segment, i.e. the segment most representative of the keyword lobster. Note that we have considered only the alternatives generated by the hierarchical segmentation algorithm. Therefore, for instance, in the figure of row (b) the combined segment segment 1,2,4 is chosen as the best segment as the combination 1,2,4,5 is not offered as an alternative by the segmentation stage

150


all the segments is excluded. Figure 10c can thus be regarded as a residual performance that remains when the effect of the good performance in the easier subtask (a) is eliminated from the task of subfigure (b). In Figure 10c the relative ordering of the algorithms with and without relevance propagation is similar to that in subfigure (b). This happens because the performance in finding any lobster segment is practically the same for the two algorithm alternatives, as shown by subfigure (a). However, from the absolute magnitude of the curves we see that also without relevance propagation the algorithm performs considerably better than random selection. Thus the principle of typicality in the selected feature spaces partly manages to favour the appropriate composite segments over their constituent parts. Nonetheless, in a significant proportion of cases the ordering is improved by augmenting the typicality assessment with the relevance propagation step.

Finding the best segment

True positives

Finding any lobsters segments

Ordering lobster segments

1

1

1

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0

0 0

0.2

0.4

0.6

0.8

False positives without propagation with propagation

(a)

1

0 0

0.2

0.4

0.6

0.8


(b)

1

0

0.2

0.4

0.6

0.8

1


(c)

Fig. 10. The averaged ROC curves of focusing keyword “lobster” in the 101 Object Categories database. Solid lines correspond to the focusing algorithm without the relevance propagation along segmentation hierarchy, the dashed line with the propagation. Subfigure (a) measures the focusing accuracy of finding any “lobster” segments. Subfigure (b) measures the accuracy of pinpointing the segment combination that is manually judged to be the best. Subfigure (c) measures in which order the true lobster segments are found. Images with only one lobster segment are excluded from this subfigure

8 Conclusions and future views In the light of the experiments, it is evident that the proposed statistical correlation principle offers a viable approach to the keyword focusing problem. However, it is clear that to function as a part of a real-world application, this technique should be augmented with other learning principles in order


151

to produce keyword focusings that utilise the information contained in the data more efficiently. This is kind of self-evident: a priori information makes the learning problem easier. A lower-level, more general learning principle is necessarily more laborious in the case where the a priori assumptions hold. The presented experiments demonstrate the potential of a system architecture, where image data is first pre-processed by a feedforward type region segmentation and description front end. The inference algorithms are subsequently applied to the representations generated by the front end. Parallel Self-Organising Maps provide a feasible means for constructing such a front end. An analogy can be drawn between this and the cortical maps of the human visual system. The straightforward image segmentation algorithm and low-level visual features in our current implementation are by no means optimal. The framework, however, is useful and we have tried to make it easy to improve, add or replace the individual components. There exists inevitably a limit to the performance of low-level feedforward image segmentation, which can be overcome only by interlinking the image segmentation with higher-level image understanding. A low-level preprocessing algorithm cannot be expected to connect parts of objects that are visually sufficiently dissimilar. The borderline where the feedforward front end should yield up the processing to the higher-level inference algorithms, in this case the focusing algorithm, is quite vague. It might be better to let the image segmentation interact more closely with the focusing procedure and not consider the segmentation as a part of the front end. One could also include more information about the alternative segmentations in the image representation by using a more versatile data structure than a tree. The data structure could be equipped with the probabilities of region merges. On the other hand, the data structure could include just more alternative segmentations, resembling the data structures, e.g. lattices, used in automatic speech recognition. All in all, in the feasibility studies of this chapter we have demonstrated that soft computing methods can be successfully used to explore the image– word correspondence and the emergence of semantical concepts. More rigorous experiments would have to be performed for more quantitative analysis. On the other hand, it has been elucidated that the subproblems of content-based image retrieval offer a challenging application area for soft computing techniques.

References 1. S. Agarwal, A. Awan, and D. Roth. Learning to detect objects in images via a sparse, part-based representation. IEEE Transactions on Pattern Analysis Analysis and Machine Intelligence, 26(11):1475–1490, November 2004.

152


2. K. Arbter. Affine-invariant Fourier descriptors. In J. C. Simon, editor, From Pixels to Features, pages 153–164. Elsevier Science Publishers B.V.(North-Holland), 1989. 3. K. Barnard, P. Duygulu, N. de Freitas, D. Forsyth, D. Blei, and M. I. Jordan. Matching words and pictures. Journal of Machine Learning Research, Special Issue on Machine Learning Methods for Text and Images, 3:1107–1135, February 2003. 4. S. Belongie, J. Malik, and J. Puzicha. Shape matching and object recognition using shape contexts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(4):509–522, April 2002. 5. I. Biederman. A theory of human image understanding. Psychological Review, 94:115–147, 1987. 6. P. Carbonetto, N. de Freitas, and K. Barnard. A statistical model for general contextual object recognition. In Proceedings of the Eight European Conference on Computer Vision, Prague, May 2004. 7. C. Carson, S. Belongie, H. Greenspan, and J. Malik. Blobworld: Image segmentation using expectation-maximization and its application to image querying. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(8):1026– 1038, August 2002. 8. Y. Chen and J. Z. Wang. Looking beyond region boundaries: Region-based image retrieval using fuzzy feature matching. In Multimedia Content-Based Indexing and Retrieval Workshop, September 24-25, INRIA Rocquencourt, France, September 2001. 9. Supplement No. 2 to CIE publication No. 15 Colorimetry (E-1.3.1) 1971: Official recommendations on uniform color spaces, color-difference equations, and metric color terms, 1976. 10. The Corel Corporation WWW home page, http://www.corel.com, 1999. 11. A. Dimai. Unsupervised extraction of salient region-descriptors for content based image retrieval. In 10th International Conference on Image Analysis and Processing (ICIAP), September 27-29, pages 686–691, Venice, Italy, September 1999. 12. J. P. Eakins. Automatic image retrieval — are we getting anywhere? In Third International Conference on Electronic Libraries and Visual Information Research (ELVIRA3), April 30 - May 2, pages 123–135, Milton Keynes, UK, 1996. De Montfort University. 13. M. Everingham, A. Zisserman, and C. K. I. Williams et al. The 2005 PASCAL Visual Object Classes Challenge. In F. d’Alche Buc, I. Dagan, and J. Quinonero, editors, Selected Proceedings of the first PASCAL Challenges Workshop. Springer, 2006. 14. The Fine Arts Museum of San Francisco http://www.thinker.org, 2005. 15. J. Fan, Y. Gao, and H. Luo. Multi-level annotation of natural scenes using dominant image components and semantic concepts. In Proceedings of the 12th annual ACM international conference on Multimedia, pages 540–547, New York, NY, October 2004. 16. J. Fan, Y. Gao, H. Luo, and G. Xu. Automatic image annotation by using concept-sensitive salient objects for image content representation. In Proceedings of the 27th annual international conference on Research and development in information retrieval, pages 361–368, Sheffield, England, July 2004. 17. L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental Bayesian approach tested on 101 object


18.

19.

20.

21. 22. 23.

24.

25.

26. 27.

28.

29.

30.

31. 32. 33.

153

categories. In Proceedings of the Workshop on Generative-Model Based Vision, Washington, DC, June 2004. W. Freeman and E. Adelson. The design and use of steerable filters. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(9):891–906, September 1991. H. Glotin and S. Tollari. Fast image auto-annotation with visual vector approximation clusters. In Proc. of IEEE EURASIP Fourth International Workshop on Content-Based Multimedia Indexing (CBMI2005), June 2005. L. Guan, P. Muneesawang, J. Lay, I. Lee, and T. Amin. Recent advancement in indexing and retrieval of visual documents. In Proceedings of the Ninth International Conference on Distributed Multimedia Systems / The 2003 Conference on Visual Information Systems (VIS’2003), pages 375–380, Miami, FL, USA, September 2003. V. N. Gudivada and V. V. Raghavan. Content-based image retrieval systems. IEEE Computer, 28(9):18–22, 1995. ISO/IEC. Information technology - Multimedia content description interface Part 3: Visual, 2002. 15938-3:2002(E). J. Jeon, V. Lavrenko, and R. Manmatha. Automatic image annotation and retrieval using cross-media relevance models. In Proceedings of 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 119–126, Toronto, Canada, July-August 2003. F. Jing, M. Li, L. Zhang, H. Zhang, and B. Zhang. Learning in region-based image retrieval. In Proceedings of International Conference on Image and Video Retrieval, volume 2728 of Lecture Notes in Computer Science, pages 198–207. Springer, 2003. A. Khotanzad and Y. H. Hong. Invariant image recognition by Zernike moments. IEEE Transaction on Pattern Analysis and Machine Intelligence, 12(5):489–497, 1990. Teuvo Kohonen. Self-Organizing Maps, volume 30 of Springer Series in Information Sciences. Springer-Verlag, third edition, 2001. J. Laaksonen, J. Koskela, S. Laakso, and E. Oja. PicSOM – Content-based image retrieval with self-organizing maps. Pattern Recognition Letters, 21(1314):1199–1207, December 2000. J. Laaksonen, M. Koskela, S. Laakso, and E. Oja. Self-organizing maps as a relevance feedback technique in content-based image retrieval. Pattern Analysis & Applications, 4(2+3):140–152, June 2001. J. Laaksonen, M. Koskela, and E. Oja. PicSOM—Self-organizing image retrieval with MPEG-7 content descriptions. IEEE Transactions on Neural Networks, Special Issue on Intelligent Multimedia Processing, 13(4):841–853, July 2002. J. Li and J. Z. Wang. Automatic linguistic indexing of pictures by a statistical modeling approach. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(9):1075–1088, September 2003. N. K. Logothetis and D. L. Sheinberg. Visual object recognition. Annual Review of Neuroscience, 19:577–621, 1996. D. G. Lowe. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2):91–110, November 2004. K. Mikolajczyk and C. Schmid. A performance evaluation of local descriptors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(10):1615– 1630, October 2005.

154


34. A. Mohan, C. Papageorgiou, and T. Poggio. Example based object detection in images by components. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(4):349–361, April 2001. 35. F. Monay and D. Gatica-Perez. On image auto-annotation with latent space models. In Proceedings of the eleventh ACM international conference on Multimedia, pages 275–278, Berkeley, CA, 2003. 36. Y. Mori, H. Takahashi, and R. Oka. Image-to-word transformation based on dividing and vector quantizing images with words. In Proceedings of First International Workshop on Multimedia Intelligent Storage and Retrieval Management, 1999. 37. J.-Y. Pan, H.-J. Yang, P. Duygulu, and C. Faloutsos. Automatic image captioning. In Proceedings of the 2004 IEEE International Conference on Multimedia and Expo, Taipei, Taiwan, June 2004. 38. J.-Y. Pan, H.-J. Yang, C. Faloutsos, and P. Duygulu. GCap: Graph-based automatic image captioning. In Proceedings MDDE ’04, 4th International Workshop on Multimedia Data and Document Engineering, Washington, DC, USA, July 2004. 39. G. Salton and M. J. McGill. Introduction to Modern Information Retrieval. Computer Science Series. McGraw-Hill, 1983. 40. R. J. Schalkoff. Pattern Recognition: Statistical, Structural and Neural Approaches. John Wiley & Sons, Ltd., 1992. 41. A. W. M. Smeulders, M. Worring, S. Santini, A. Gupta, and R. Jain. Contentbased image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(12):1349–1380, December 2000. 42. M. Sonka, V. Hlavac, and R. Boyle. Image Processing, Analysis and Machine Vision. International Thomson Computer Press, 1993. 43. S. Ullman. High-Level Vision: Object recognition and cognition. MIT Press, 1996. 44. A. Ultsch. Data mining and knowledge discovery with emergent self-organizing feature maps for multivariate time series. In E. Oja and S. Kaski, editors, Kohonen Maps, pages 33–45. Elsevier, 1999. 45. V. Viitaniemi and J. Laaksonen. Keyword-detection approach to automatic image annotation. In Proceedings of 2nd European Workshop on the Integration of Knowledge, Semantic and Digital Media Technologies (EWIMT 2005), pages 15–22, London, UK, November 2005. 46. J. Z. Wang, J. Liu, and G. Wiederhold. SIMPLIcity: Semantics-sensitive integrated matching for picture libraries. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(9):947–963, September 2001.

Part III

Applications in Image Analysis

Chapter 6 A Fuzzy Edge-Dependent Interpolation Algorithm Piedad Brox, Iluminada Baturone, and Santiago S´ anchez-Solano Instituto de Microelectr´ onica de Sevilla (IMSE-CNM) [email protected]

1 Introduction Interpolation of images to achieve a higher resolution is required in many applications such as medical and satellite imaging and video format conversion. The resolution improvement provides more image details which are critical to make diagnosis, to distinguish an object or to generate high definition television (HDTV) signals. The most widely used image interpolation techniques such as bilinear, bicubic and spline based methods [1]-[4] assume that the images behave locally like a polynomial. They perform acceptably in homogeneous areas of the images but introduce blurring and annoying artifacts (e.g. staircase effect) around edges. Many algorithms [5]-[11] have been proposed to improve the quality of the interpolated images by enhancing edges and the overall image sharpness. Some of them introduce adaptive interpolation techniques to adjust the interpolation coefficients [5, 6]. Other algorithms add image details in the wavelet domain by using exponential decay or training data [9, 10] while other authors resort to the use of neural networks such as radial basis function (RBF) networks or multilayer perceptrons [11]. Fuzzy logic has also been used for image interpolation [12]-[15]. In [12] a fuzzy edge-direction detector is introduced to orient a vertico-temporal interpolation filter. The algorithms proposed in [13]-[14] use a fuzzy inference approach to determine the contribution of each pixel in the neighborhood. The decision taken by the fuzzy system depends on the local properties of the pixel, such as the luminance value and the distance between the interpolated pixel and the pixel in the original image. In [15] a codebook-based fuzzy interpolation technique is proposed to realize image enlargement. The problem is that these approaches feature a high computational cost. A very simple algorithm which considers edges to increase the resolution of images is the edge-based line average algorithm (ELA) [16]. It was proposed to convert de-interlaced into progressive images. To find the local edge

P. Brox et al.: A Fuzzy Edge-Dependent Interpolation Algorithm, StudFuzz 210, 157–185 (2007) c Springer-Verlag Berlin Heidelberg 2007 www.springerlink.com

158

Piedad Brox et al.

direction, it uses the maximum correlation between 3+3 pixels of two adjacent scan lines within the same image and then interpolates along that direction. This algorithm performs well when the edge direction agrees with the maximum correlation but, otherwise, it introduces mistakes and degrades the image quality. These errors also appear in cases where the image is corrupted by noise. Several proposals have been presented in the literature to avoid these shortcomings [17]-[24]. Some of them focus on increasing the edge detection consistency. For example, in [17], a linear interpolation depending on a directional correlation similar to ELA is used for the signal in the low horizontal frequency region whereas the line doubling method is used for the signal with high horizontal frequency components. An adaptive ELA technique to also enhance the reconstruction of horizontal edges is presented in [18]. Other authors resort to the use of a larger neighborhood to get more information about the possible edge direction. The algorithm developed in [19] is composed by a modified ELA module and a contrast enhancement module. The ELA module increases the processing window up to 5+5 pixels whereas the second module reduces detail losses due to the interpolation. The neighborhood is enlarged up to 6+6 taps in [20], 7+7 taps in [21]-[22], 11+11 taps in [23], and 34+34 taps in [24]. The problem is that a high number of pixels increases the complexity of the resulting technique. The algorithm proposed herein aims at an efficient tradeoff between performance and low computational cost. On one side, it is inspired by the simple ELA scheme and, on the other side, uses a simple fuzzy system, which models heuristic knowledge, to ensure the presence of edges. This is why it has been named Fuzzy-ELA algorithm. The Chapter is organized as follows. The proposed Fuzzy-ELA algorithm and its possible enhancement are described in Section 2. The following sections illustrate two important applications related to image interpolation: deinterlacing of video sequences (Section 3) and enlargement of images (Section 4). The superior performance of the novel interpolation algorithm has been proven by extensive simulations on standard images and video sequences. Simulations on noisy images are also included to illustrate the better robustness of our technique against noise. The large amount of data involved in image processing applications as well as the real-time requirements demand hardware implementation. Section 5 describes the FPGA implementation of the proposed algorithm, showing its low computational cost and high speed performance. Finally some conclusions are given in Section 6.

2 Fuzzy-ELA algorithm 2.1 Description Since the proposed algorithm is inspired by the conventional ELA, let us describe firstly how this algorithm interpolates the pixel value, X, of a new row

A Fuzzy Edge-Dependent Interpolation Algorithm

159

Fig. 1. 3+3 taps for the ELA algorithm

from 3+3 pixels in the upper and lower rows (Fig. 1). The ELA algorithm doubles the resolution of an image in the vertical direction since it was proposed to convert interlaced into progressive formats. In the same way, this section describes how the Fuzzy-ELA algorithm interpolates new rows (interpolation of new columns is quite symmetrical). The pseudo code of the ELA algorithm is as follows: a = |A − F | b = |B − E| c = |C − D| if min(a, b, c) = a → X = (A + F )/2 elseif min(a, b, c) = c → X = (C + D)/2

(1)

else → X = (B + E)/2 ELA performs well if the edges are clear, i.e. the maximum correlation coincides with the presence of an edge. However, problems arise because the maximum correlation does not always indicate the direction of an edge, as shown in Fig. 2. In the case of non-clear edges, ELA usually introduces errors, as shown in Fig. 3b. In the case of ambiguous situations, ELA may also fail (Fig. 3c) depending on the particular gray level of the pixels. These problems increase when the image is corrupted by noise. Our approach has been to apply heuristic knowledge to overcome these limitations. The following knowledge is employed to estimate correctly the edge direction without enlarging the processing window: 1. An edge is clear in direction a not only if a is small but also if b and c are large. 2. An edge is clear in direction c not only if c is small but also if a and b are large. 3. If there is a strongly small difference in directions a and c and a large difference in direction b, neither there is an edge nor vertical linear interpolation performs well; the best option is a linear interpolation between the neighbors with small differences: A, C, D, F . 4. In other cases, a vertical linear interpolation would be the most adequate. This heuristic knowledge is fuzzy since the concepts of “small”, “strongly small”, and “large” are not understood as threshold values but as fuzzy ones. Hence, our proposal is to model this knowledge by a fuzzy system.

160

Piedad Brox et al.

Fig. 2. Examples of (a) clear, (b) non-clear, and (c) ambiguous edges in a progressive image

Fig. 3. Interpolating the middle row of examples in Fig. 2 with the ELA algorithm

The rule base of the fuzzy system is described in Table 1. Using fuzzy logic, the concepts of “SMALL” and “LARGE” are represented by fuzzy sets whose membership functions change continuously instead of abruptly between 0 and 1 membership values, as shown in Fig. 4a and Fig. 4b. The linguistic hedge “strongly” acting upon the concept of “SMALL” modifies its membership function as illustrated in Fig. 4c [25]. Table 1. Fuzzy rule set for the Fuzzy-ELA algorithm if

antecedent

1) (a is SMALL) and (b is LARGE) and (c is LARGE) 2) (a is LARGE) and (b is LARGE) and (c is SMALL)

then

consequent (A+F)/2 (C+D)/2

3)

(a is strongly SMALL) and (b is LARGE) and (c is strongly SMALL)

(A+F+C+D)/4

4)

otherwise

(B+E)/2


161

Fig. 4. Membership functions for the fuzzy sets (a) “LARGE”, (b) “SMALL”, and (c) “strongly SMALL”

The minimum operator is used as connective “and” of antecedents and the activation degrees of the rules, αi , are calculated as follows: α1 = min(µSM ALLa (h), µLARGEb (h), µLARGEc (h)) α2 = min(µLARGEa (h), µLARGEb (h), µSM ALLc (h)) α3 = min(µstronglySM ALLa (h), µLARGEb (h), µstronglySM ALLc (h))

(2)

α4 = 1 − α1 − α2 − α3 Since the consequents, ci , of the rules are not fuzzy, the global conclusion provided by the system is calculated by applying the Fuzzy Mean defuzzification method (which corresponds to a weighted average), as follows: 4 αi ci α1 c1 + α2 c2 + α3 c3 + α4 c4 X = i=1 = (3) 4 α1 + α2 + α3 + α4 i=1 αi Substituting the consequents, ci , by their values in Table 2, and applying that α1 + α2 + α3 + α4 is equal to 1, the above expression can be given as: + α2 C+D + α3 A+F +C+D + α4 B+E (4) X = α1 A+F 2 2 4 2 Fuzziness makes all the rules could be activated simultaneously, contrary to what happens in the ELA algorithm. Hence, the Fuzzy-ELA algorithm applies a linear interpolation in the directions a or c if there is a clear edge (α1 or α2 take the value 1 and the others are 0). Otherwise several rules are active and the interpolation is non-linear. As a consequence, the proposed method works better than ELA with non-clear and ambiguous edges. This is illustrated in Fig. 5, where the same examples of Fig. 3 are now processed with fewer errors. 2.2 Determination of membership function parameters for the fuzzy system Having defined the rules of the fuzzy system from heuristic knowledge, what remains to be defined are membership functions of the concepts “SMALL”,

162

Piedad Brox et al.

Fig. 5. Interpolating the middle row of examples in Fig. 2 with the Fuzzy-ELA algorithm

“LARGE”, and “strongly SMALL”, that is, the parameters H1, H2, and H3 in Fig. 4. Heuristic knowledge does not provide much information on this issue, except that H3 should be smaller than H2 to make the concepts of “SMALL” and “strongly SMALL” meaningful. The use of the minimum operator as antecedents’ connective imposes a stronger constraint on H2 and H3: since it has not sense that rules’ activation degrees take negative values (in particular, the activation degree α4 ), the parameter H3 should be smaller or equal to H2/2. Among the lot of values that meet this constraint, some of them will surely provide better results than others. If a set of data is available from progressive images, an error function can evaluate the difference between using some parameters or others, and supervised learning algorithms can be applied to minimize the error. This is the approach we have followed to select good values for the membership function parameters. Automatic tuning has been applied by using the CAD tool xfsl integrated into the Xfuzzy 3.0 development environment [26]. Xfuzzy 3.0 is a whole environment for designing fuzzy systems that is composed of a set of CAD tools covering the different stages of description, verification, simplification and synthesis of inference systems based on fuzzy logic [27]. They are integrated into a graphical user interface, and employ an internal representation scheme that allows them to work in parallel on a fuzzy system definition. The fuzzy system employed in the Fuzzy-ELA algorithm has been described within Xfuzzy 3.0 as a first-order Takagi-Sugeno system, since it can be considered that the rule consequents are linear functions of the inputs. To meet the choice of calculating the activation degree α4 as 1 − α1 − α2 − α3 , the rule set in Table 2 has been translated into the equivalent one shown in Table 2, where the consequents are defined as follows: c1 = 0, 5(A + F ) − 0, 5(B + E) c2 = 0, 5(C + D) − 0, 5(B + E) c3 = 0, 25(A + F + C + D) − 0, 5(B + E)

(5)

c4 = 0, 5(B + E) the defuzzification is implemented as a weighted sum (instead of a weighted average), and the membership functions “LARGE”, “SMALL”, and “strongly


163

Table 2. Description of the Fuzzy-ELA rule set with Xfuzzy 3.0 if

antecedent

then consequent

1) (a is SMALL) and (b is LARGE) and (c is LARGE)

c1

2) (a is LARGE) and (b is LARGE) and (c is SMALL)

c2

3) 4)


c3

a is DUMMY

c4

SMALL” are described as was shown in Fig. 4 (Fig. 6 illustrates the graphical user interface of the CAD tool xfedit of Xfuzzy 3.0 which eases the descriptions of these functions). The membership function called “DUMMY” returns a value of one independently of the input value. It is employed in rule 4 to always include the value (B + E)/2 in the output of the system. The final conclusion calculated by Xfuzzy is given by the following expression: + α2 C+D + − B+E − B+E X = α1 A+F 2 2 2 2 A+F +C+D B+E B+E + (6) − 2 α3 4 2 which is equivalent to the expression in (4), as desired. The CAD tool xfsl allows to apply different supervised learning algorithms to fuzzy systems described in Xfuzzy 3.0. In particular, the MarquardtLevenberg algorithm has been employed. Exploiting that the xfsl tool allows

Fig. 6. Membership function description of the input variables using xfedit

164

Piedad Brox et al.

the user to configure the tuning process, only the parameters involved in the description of the membership function “SMALL”, “LARGE” and “strongly SMALL” are enabled to participate in the tuning process. The set of input/output training patterns have been generated from a set of progressive video sequences. Figure 7 shows the evolution of three error functions along the learning process. The result of the learning process is that the parameter H3 of the “strongly SMALL” membership function (see Fig. 4) is close to 8; the parameter H2 of “SMALL” function is close to 16; and H1 is approximately 64 for the “LARGE” membership function. Since initially the definition of H1 and H2 was the same for “SMALL” and “LARGE” functions, a significant variation has been introduced by the tuning process. The values of 64, 16 and 8 for H1, H2, and H3, respectively, are also very efficient from a hardware point of view, and, hence, they have been chosen to complete the definition of the Fuzzy-ELA algorithm.

Fig. 7. Evolution of the tuning process

2.3 Enhanced fuzzy-ELA algorithms Two extensions of the Fuzzy-ELA algorithm have been explored in order to improve the obtained results. They also employ a fuzzy inference system to decide a non-linear interpolation but they work with a larger neighborhood of pixels. The first one considers 5+5 taps so as to detect more edge directions while the second one uses samples from the previously interpolated rows to ensure the presence of an edge.


165

2.3.1 The 5+5 tap-Fuzzy-ELA algorithm The Fuzzy-ELA (3+3 taps) algorithm described previously searches for edges in only three directions: a, b, and c. The 5+5 tap-Fuzzy-ELA considers the larger neighborhood shown in Fig. 8 so as to include the two new directions a and c (corresponding to 153, 4o and 26, 6o ): a = |A − F | c = |C − D |

(7)

As a consequence, the fuzzy inference system now applied has 6 instead of 4 rules, whose antecedent parts have 5 instead of 3 antecedents, as shown in Table 3. The concepts of “SMALL” and “LARGE” are again represented by the membership functions previously defined. The activation degrees of the rules, αi , are calculated as follows: α1 = prod(µSM ALLa (h), µLARGEa (h), µLARGEb (h), µLARGEc (h), µLARGEc (h)) α2 = prod(µLARGEa (h), µLARGEa (h), µLARGEb (h), µLARGEc (h), µSM ALLc (h)) α3 = prod(µLARGEa (h), µSM ALLa (h), µLARGEb (h), µLARGEc (h)), µLARGEc (h) α4 = prod(µLARGEa (h), µLARGEa (h), µLARGEb (h), µSM ALLc (h)), µLARGEc (h) α5 = prod(µstronglySM ALLa (h), µLARGEb (h), µstronglySM ALLc (h)) α6 = 1 − α1 − α2 − α3 − α4 − α5 (8)

Although it is more costly in hardware than the minimum operator, the product operator is now used to represent the connective “and” so as to ensure that the activation degrees of the rules (in particular α6 ) never take negative values. The final conclusion is also obtained using the Fuzzy Mean defuzzification method as follows: C +D + α + α3 A+F + α4 C+D X = α1 A +F 2 2 2 2 2 + α6 B+E (9) +α5 A+F +C+D 4 2

Fig. 8. Taps employed by the 5+5 tap-Fuzzy-ELA algorithm

166

Piedad Brox et al. Table 3. Fuzzy rule set for the 5+5 tap-Fuzzy-ELA algorithm

if

antecedent

then

consequent

1) (a’ is SMALL) and (a is LARGE) and (b is LARGE) and (c is LARGE) and (c’ is LARGE)

(A’+F’)/2

2) (a’ is LARGE) and (a is LARGE) and (b is LARGE) and (c is LARGE) and (c’ is SMALL)

(C’+D’)/2

3) (a’ is LARGE) and (a is SMALL) and (b is LARGE) and (c is LARGE) and (c’ is LARGE)

(A+F)/2

4) (a’ is LARGE) and (a is LARGE) and (b is LARGE) and (c is SMALL) and (c’ is LARGE)

(C+D)/2

5)

(a is strongly SMALL) and (b is LARGE) and and (c is strongly SMALL)

(A+F+C+D)/4

6)

otherwise

(B+E)/2

2.3.2 Recursive Fuzzy-ELA algorithm The other improvement of 3+3 tap-Fuzzy-ELA algorithm that has been explored regards increasing not the number of possible edge directions but the consistency of the detected direction. Estimation of the dominant direction is now enhanced by using further information from the neighborhood. Figure 9 shows the evaluated pixels, where Ai and Ci are samples from the previously de-interlaced line (the first de-interlaced line is calculated with the 3+3 tap-Fuzzy-ELA). The new differences considered are given by the expressions: ai = |Ai − A| aii = |Ai − F | ci = |Ci − C|

cii = |Ci − D|

(10)

The new fuzzy rule set is described in Table 4. It also contains 4 rules but with more complex antecedents. In particular, the first and second rules are modified to ensure the presence of an edge in their corresponding orientations (angle 45o or 135o ). The activation degree of the rules are calculated using

Fig. 9. Taps employed by the Recursive Fuzzy-ELA algorithm


167

Table 4. Fuzzy rule set for the Recursive Fuzzy-ELA algorithm if

antecedent

then consequent

1)

((a is SMALL) or (ai is SMALL) or (aii is SMALL)) and (b is LARGE) and (c is LARGE) and (ci is LARGE)

2Ai+A+F 4

2)

(a is LARGE) and (ai is LARGE) and (b is LARGE) and ((c is SMALL) or (ci is SMALL) or (cii is SMALL))

2Ci+C+D 4

3)


A+F +C+D 4

4)

otherwise

B+E 2

the minimum operator as connective “and” and the maximum as connective “or” (since there is no problem with negative activation degrees in this rule set): max(µSM ALLa (h), µSM ALLai (h), µSM ALLaii (h)) α1 = min µLARGEb (h), µLARGEc (h), µLARGEci (h) max(µSM ALLc (h), µSM ALLci (h), µSM ALLcii (h)) (11) α2 = min µLARGEb (h), µLARGEa (h), µLARGEai (h) α3 = min(µstronglySM ALLa (h), µLARGEb (h), µstronglySM ALLc (h)) α4 = 1 − α1 − α2 − α3 The final conclusion is also obtained by applying the Fuzzy Mean defuzzification method, as follows: + α2 Ci+Ci+C+D + X = α1 Ai+Ai+A+F 4 4 A+F +C+D B+E + α4 (12) α3 4 2 In order to analyze these two modifications as well as the Fuzzy-ELA algorithm, many simulations on de-interlacing images and video sequences have been carried out, as described in the next Section.

3 De-interlacing 3.1 Review of algorithms for de-interlacing De-interlacing or interlaced-to-progressive (IPC) conversion is one of the operations which is usually required by video format conversion techniques to display different image/video resources on different display systems. Interlaced formats are used in standard television systems (e.g. PAL in Europe and NTSC in USA and Japan) to halve the required video bandwidth because the whole frames are not transmitted but fields with the half of the

168

Piedad Brox et al.

lines. This means that after the interlacing process, the “odd fields” in the sequence order only contain the odd lines while the “even fields” only contain the even lines. As an example, Fig. 10 illustrates an interlaced field of a video sequence and the corresponding progressive frame from which it has been generated. The increasing demand for progressive scan formats with the advent of HDTV system, high quality monitors and projectors requires that the missing lines of the interlaced format have to be interpolated by applying what is known as a de-interlacing algorithm. A good de-interlacing process is achieved when realizing a good interpolation between the pixels of the fields.

Fig. 10. (a) Interlaced field of a video sequence. (b) The corresponding progressive frame

Severals proposals have been reported in the last years to de-interlace video sequences [28]-[33]. They can be classified into motion-compensated (MC) and nonmotion-compensated (non-MC) algorithms. Two categories are distinguished in non-MC algorithms regarding the pixels which participate in the interpolation process. The first one contains linear techniques that always use the same pixels of the neighborhood while the second one attempts to improve the de-interlacing process with non-linear strategies. Spatial (or intra-field), temporal (or inter-field), and spatio-temporal algorithms can be found in both categories. Among the linear techniques, temporal de-interlacing methods (such as field insertion) exploit the correlation in the time domain. These techniques realize a perfect de-interlacing in stationary areas of the images but introduce annoying artifacts in moving objects such as the serration effect. Figure 11a illustrates how this effect appears in the moving edges of the calendar and the diagonal black and white lines when de-interlacing the field in Fig. 10a. Spatial de-interlacing techniques (such as line doubling and line average) exploit the correlation between vertically neighboring samples in a field. These techniques are the most advantageous considering hardware implementation since no memories are required to store previous fields. However, they cause blurring especially in moving areas and regions with edges, as shown in Fig. 11b.


169

Fig. 11. De-interlaced frame obtained with: (a) field insertion, (b) line average, (c) VT-filtering 2 fields and (d) VT-filtering 3 fields

Theoretically, the best linear approach is VT filtering since it combines the linear spatial and temporal methods. The obtained results are, again, good in stationary areas but not in moving ones. This can be seen in Fig. 11c and Fig. 11d, where the de-interlaced image has resulted from applying a VT filtering with two [29] and three fields [30]. The ELA algorithm described in the previous section belongs to the group of non-linear de-interlacing techniques. Figure 12a and Fig. 12b show, respectively, the result obtained by this algorithm when employing 3+3 and 5+5 taps in the neighborhood. The errors introduced by this algorithm when processing non-clear edges or ambiguous situations can be seen in the number, letters, and diagonal lines of the calendar. Neither ELA nor the above commented linear algorithms take into account motion in the video sequence. The most advanced de-interlacing algorithms which consider motion are the MC techniques. They estimate motion by calculating motion vectors and realize the interpolation along the motion trajectories. Figure 12c and Fig. 12d show the de-interlaced images obtained by applying two of these MC methods, namely, temporal backward projection [31] and time-recursive de-interlacing [32], respectively. The results obtained with these methods are usually very good. Their principal drawback is the high computational cost involved in the calculation of motion vectors.

170

Piedad Brox et al.

Fig. 12. De-interlaced frame obtained with: (a) ELA 3+3 taps, (b) ELA 5+5 taps, (c) temporal backward projection and (d) time-recursive

The Fuzzy-ELA algorithms introduced in the previous section belong to non-linear non-MC techniques and, hence, their computational cost is much less than that of MC techniques. The de-interlacing they perform on the field in Fig. 10a are shown in Fig. 13a and Fig. 13b. The following subsection illustrates more in detail the performance of Fuzzy-ELA algorithms and compares the results with other linear and non-linear non-MC algorithms with less or similar computational cost. 3.2 The fuzzy-ELA algorithm for de-interlacing Extensive simulations have been carried out with several video sequences to evaluate the quality of the three Fuzzy-ELA algorithms introduced and compare their performance with other intra-field methods. In particular, the line doubling, line average, the conventional ELA [16], and the enhanced ELA reported in [19] with any additional processing have been considered. The 3+3 tap-Fuzzy-ELA with crisp instead of fuzzy descriptions of labels “SMALL”, “LARGE”, and “strongly SMALL” has also been considered in the analysis to evaluate the advantages of using fuzzy logic. The interlaced video sequences processed have been obtained from progressive ones in order to measure the difference between the obtained interpolated frames and the original ones. Many error measures have been proposed


171

Fig. 13. De-interlaced frame obtained with: (a) 3+3 tap-Fuzzy-ELA, (b) 5+5 tapFuzzy-ELA

as quality figure of merits for digital video. However, it is difficult to entirely describe the visibility of image distortions with one objective criterion because the human visual system is a very complex system. Nevertheless, some measures seem to have a higher correlation than others with the perceived quality. A very much popular quality criteria is the mean square error (MSE) given by the following expression: M SE =

1 (X − Xprogressive )2 M ×N

(13)

where the image has a resolution of M × N pixels, X is the value of the interpolated pixel, and Xprogressive is the pixel value in the original progressive image. Very much in relation with the MSE is the PSNR, which is defined as follows: P SN R = 40 log

255 255 √ = 20 log √ M SE M SE

(14)

Table 5 shows the average PSNR values obtained when de-interlacing the fields of “Claire” video sequence with the different algorithms considered. The robustness of the algorithms against noise has been also analyzed by adding artificial noise to original frames. Two kinds of noise (Gaussian and impulse) have been considered with different degrees of degradation. The results obtained with another typical video sequence (named as “Susie”) are included in Table 6. The PSNR results show that the proposed Fuzzy-ELA algorithms perform better than all the other algorithms with both noisy and non-noisy images. They not only reconstruct clear edges as well as conventional ELA algorithm, as corroborated by the zooms in Fig. 14 and Fig. 15 (which also shown the annoying artifacts of line doubling and line average methods), but also introduce

172

Piedad Brox et al.

Table 5. Average PSNR values (in dBs) when de-interlacing “Claire” sequence Noise No noise Impulse 5.5% 6.5% 7.5% 12% 15% Gaussian 0.003 0.009 0.015 0.021 0.027

Line Line (3+3) (3+3) Modified Fuzzy (5+5) Recursive Doubling Average ELA Crisp ELA[19] ELA F-ELA F-ELA 35.75 17.51 16.81 16.19 14.15 13.20 25.65 21.01 18.84 17.42 16.32

40.95 18.77 18.06 17.45 15.38 14.41 26.69 22.15 20.01 18.59 17.56

40.55 20.24 19.47 18.82 16.54 15.42 26.71 22.19 20.07 18.66 17.63

40.94 19.59 18.89 18.29 16.22 15.24 26.69 22.15 20.01 18.59 17.57

40.96 19.86 19.14 18.51 16.33 15.33 26.71 22.23 20.18 18.83 17.84

41.12 20.32 19.58 18.95 16.76 15.68 27.22 22.76 20.63 19.21 18.15

41.25 20.35 19.59 18.96 16.80 15.75 27.01 22.59 20.53 19.15 18.14

41.11 20.34 19.61 18.99 16.82 15.77 27.23 22.79 20.69 19.31 18.27

fewer errors in the rest of the image (it can be seen how the PSNR results of ELA algorithm are smaller than line average for non-noisy images while those of Fuzzy-ELA are always greater). Advantages of using fuzzy instead of crisp processing can be seen from comparing the PSNR results of Fuzzy-ELA algorithms with that of “(3+3) Crisp” in Tables 5 and 6. Regarding the three Fuzzy-ELA algorithms, the 5+5 tap-Fuzzy-ELA method works slightly better than the 3+3 tap-Fuzzy-ELA for non-noisy images and images corrupted by impulse noise, but performs slightly worse for images corrupted by Gaussian noise. The Recursive Fuzzy-ELA algorithm always improves the results as the degradation degree of the noisy images increases. Figure 16a and Fig. 16b show one progressive frame of the “Susie” video sequence corrupted by Gaussian and impulse noise, respectively. The correTable 6. Average PSNR values (in dBs) when de-interlacing “Susie” sequence Noise No noise Impulse 5.5% 6.5% 7.5% 12% 15% Gaussian 0.003 0.009 0.015 0.021 0.027

Line Line (3+3) (3+3) Modified Fuzzy (5+5) Recursive Doubling Average ELA Crisp ELA [19] ELA F-ELA F-ELA 32.71 17.76 17.45 16.86 14.84 13.20 25.65 20.23 18.23 16.76 15.75

36.16 19.45 18.74 18.11 16.09 14.41 26.69 21.58 19.45 18.06 17.03

35.66 20.82 20.06 19.38 17.12 15.98 27.43 21.51 19.43 17.99 16.96

36.31 20.12 19.44 18.81 16.78 15.79 27.61 21.57 19.45 18.06 17.03

36.53 20.55 19.79 19.17 17.07 16.06 27.65 21.69 19.63 18.82 17.31

36.39 20.97 20.23 19.58 17.42 16.34 28.08 22.19 20.04 18.63 17.58

36.73 21.03 20.29 19.65 17.51 16.46 27.91 22.05 19.98 18.61 17.58

36.38 20.99 20.26 19.62 17.48 16.43 28.09 22.28 20.18 18.79 17.76


173

Fig. 14. (a) Progressive frame of “Claire” sequence. (b) Zoom of the original frame. Zooms of the de-interlaced image applying: (c) line doubling, (d) line average, (e) ELA, and (f) Fuzzy-ELA algorithm

Fig. 15. (a) Progressive frame of “Susie” sequence. (b) Zoom of the original frame. Zooms of the de-interlaced image applying: (c) line doubling, (d) line average, (e) ELA, and (f) Fuzzy-ELA algorithm

sponding de-interlacing images applying 3+3 tap-Fuzzy-ELA are shown in Fig. 16c and Fig. 16d. Nevertheless, not only quality performance but also computational complexity should be evaluated. For this purpose all the algorithms have been programmed in Matlab and executed on the same PC (running the operating

174

Piedad Brox et al.

Fig. 16. Original progressive frame corrupted by (a) Gaussian noise and (b) impulse noise. The corresponding de-interlaced images with Fuzzy-ELA ((c) and (d))

system MS-Windows XP with a 2.0 GHz Pentium IV processor). The total CPU time as well as the computational time ratio according to the fastest algorithm are shown in Table 7. Taken into account all these results it can be concluded that the 3+3 tap-Fuzzy-ELA algorithm provides the best tradeoff quality/cost of all the algorithms. Table 7. Computation time required by de-interlacing algorithms Algorithm Line Doubling Line Average (3+3) ELA (3+3) Crisp Modified ELA [19] Fuzzy-ELA (5+5) Fuzzy-ELA Recursive Fuzzy-ELA

CPU time Computational (in seconds) time ratio 0.15 0.17 0.31 0.43 0.96 0.546 5.59 2.12

1 1.13 2.06 2.86 6.4 3.64 37.26 14.13


175

4 Image enlargement 4.1 Review of algorithms for image enlargement Image enlargement addresses the problem of magnifying the size of images aiming at obtaining higher resolution, that is, increasing the pixel density within the image. This is a growing demanding problem with the advent and proliferation of low resolution digital cameras, such as those found in today’s cell phones. In addition, high resolution images are very helpful in different application domains such as medical imaging (to help the doctor in making correct diagnosis) or computer vision (to improve the performance of pattern recognition methods). The developed algorithms for image enlargement interpolate between pixels from images with low resolution. In the basic case where the magnification factor is just two, the resizing scheme consists of two steps: the interpolation of new rows (as it is done in a de-interlacing process) and of new columns (Fig. 17). The different algorithms proposed in the literature can be grouped into two categories. One group focuses on using pixels of a single low-resolution image [1]–[4], [34]–[35] while the other contains multiframe methods which use pixels of multiple images. Recently, the latter techniques have been extensively used in super resolution image reconstruction [36]. The results they offer are very good but at expense of a high increase in memory requirements and computational costs because they employ complex motion compensation techniques which consider the relative motion between the frames. Among single frame algorithms those which calculate a global interpolation function using polynomial methods, such as Lagrange or Hermite, do not perform well for image data since they cannot reconstruct local image properties. However, researchers have applied piecewise polynomials approaches to the interpolation problem with successful results. The simplest one is the

Fig. 17. (a) Image with low resolution. (b) Image with high resolution applying an enlargement factor equal to two

176

Piedad Brox et al.

nearest neighbor interpolation (NNI) algorithm which sets the new pixel value to the nearest pixel. The underlying image model in this case is a polynomial spline of order zero (piecewise constant). It is extremely simple to implement but tends to produce images with a blocky appearance. Let us consider, for instance, the enlargement of the text image shown in Fig. 18a. This image has been obtained by removing every even row and column of a higher resolution image. A zoom of this higher resolution image is shown in Fig. 18b, that is, Fig. 18b illustrates the ideal result of an enlargement algorithm. The result given by the NNI algorithm is illustrated in Fig. 18c. The bilinear interpolation (BLI) algorithm, which uses an implicit first order spline model, evaluates the interpolated point as a combination of the values of the four closest points. It offers more satisfactory results, as can be seen in Fig. 18d. Several authors have proposed piecewise cubic interpolation methods such as the BCI and the CSI, which uses cubic spline functions [2], [34]-[35]. These interpolation algorithms are very useful in areas with homogeneous luminance values, that is, in the absence of edges, since they generate the smoothest interpolations. However, they introduce blurring and step-edge effects in borders and are considerably expensive in terms of computational cost. The results of these two algorithms when enlarging Fig. 18a are shown in Fig. 18e and Fig. 18f, respectively. Our proposal is to use the 3+3 tap-Fuzzy-ELA algorithm to realize an efficient interpolation under low complexity restrictions. As can be seen in Fig. 19, Fuzzy-ELA algorithm obtains better results than conventional ELA, which introduces more errors in non-clear and ambiguous edges. More details about the performance of Fuzzy-ELA for image enlargement and comparisons with other algorithms are given in the following subsection. 4.2 The Fuzzy-ELA algorithm for image enlargement Since the image enlargement can be understood as a problem of interpolating new rows and columns, the Fuzzy-ELA algorithm can be used as explained in Section 2 to interpolate new rows (by using 3+3 pixels from the upper and lower rows) and, similarly, to interpolate new columns (by using 3+3 pixels from the columns on the left and on the right). Extensive simulations with different types of images have been carried out to evaluate the performance of the proposed Fuzzy-ELA method and comparisons have been made with four piecewise interpolation algorithms, namely, NNI, BLI, BCI, and CSI. Having programmed all the algorithms in Matlab language and executed on the same PC, their computation times can be compared. Table 8 shows the total CPU time (in seconds) for each algorithm as well as the computational time ratio calculated according to the fastest algorithm. Analyzing the results, it is corroborated that the fastest method is NNI but it provides the worst results. BLI method considerably improves the results with a slight increase of the execution time. Unfortunately, the most


177

Fig. 18. Low-resolution text image obtained from the high-resolution one in (b). Zoom of enlarged text image in (a) obtained with (c) NNI, (d) BLI, (e) BCI, and (f) CSI

accurate methods, which are BCI and CSI, involve a huge amount of mathematical operations to compute the new pixel, which translates in a much longer computational time. The CPU time of the proposed Fuzzy-ELA algorithm is much shorter than that of cubic methods and slightly longer than that of BLI and NNI methods.

178

Piedad Brox et al.

Fig. 19. Zoom of enlarged text image obtained with ELA (a) and Fuzzy-ELA (b) Table 8. Computation time required by interpolation methods Algorithm

CPU time Computational (in seconds) time ratio

NNI BLI BCI SCI ELA FUZZY-ELA

0.516 0.671 5.511 12.089 0.995 1.548

1 1.3 10.68 23.43 1.93 3

To evaluate the quality of the proposed algorithm the images processed have been obtained by eliminating the even rows and columns of a high resolution image in order to measure the difference between the enlarged image (by a factor of two in both the horizontal and vertical directions) and the original (high resolution) image. The MSE has been selected as figure of merit. MSE results for different images are shown in Table 9. Table 9. Comparison of interpolation methods Image Cell * Xray Nematoide Lena * Elaine Boat * Tiffani

Resolution 128x128 128x128 256x256 256x256 256x256 256x256 256x256

to to to to to to to

256x256 256x256 512x512 512x512 512x512 512x512 512x512

NNI

BLI

BCI

CSI

77.45 382.82 193.5 76.91 88.18 189.44 54.66

32.55 132.31 80.23 21.14 43.56 80.07 25.21

32.03 119.46 84.58 18.45 46.37 78.19 25.53

33.38 123.91 91.11 18.49 49.73 81.59 27.09

ELA FUZZY-ELA 56.44 177.17 122.71 44.66 56.27 132.41 42.66

32.04 115.65 79.94 20.01 42.7 79.4 24.8


179

The obtained results agree with the above comments on the performance of piecewise interpolation methods. In particular, it can be seen how the cubic interpolation methods (specially the BCI) offer the best results among them. The proposed Fuzzy-ELA algorithm always works better than NNI, BLI, and ELA (at expense of a low increase in computational time). For images with an important number of edges, Fuzzy-ELA results are also better than that of BCI and CSI, as can be seen in Fig. 20 for the “xray” image. Even for images with a majority of homogeneous areas (images marked with * in Table 9) the results of Fuzzy-ELA are very slightly worse than those of BCI, as illustrated in Fig. 21.

Fig. 20. Zoom of enlarged medical image “xray” obtained with (a) NNI, (b) BLI, (c) BCI, (d) CSI, (e) ELA, and (f) Fuzzy-ELA algorithm

Performance of the different algorithms against Gaussian and impulse noise has been also tested. Table 10 shows the MSE results for the medical image “xray” corrupted by these two kinds of noise. It can be seen how our proposal is much more robust that all the other methods.

180

Piedad Brox et al.

Fig. 21. Zoom of enlarged image “Lena” obtained with (a) NNI, (b) BLI, (c) BCI, (d) CSI, (e) ELA, and (f) Fuzzy-ELA algorithm

5 Hardware implementation Hardware implementations are usually required in video and image processing applications to meet real-time specifications. For example, in the case of dealing with PAL standard TV sequences (25 images per second with 704 x 576 pixels), more than 10 mega pixels should be processed per second, that is, the time available to process one pixel is less than 100 ns. The software implementation of all the de-interlacing algorithms commented in Section 3


181

Table 10. MSE results when enlarging the “xray” image Noise No noise Impulse 2.5% 3.5% 4.5% 5.5% Gaussian 0.003 0.008 0.014 0.019

NNI

BLI

ELA

BCI

CSI

FUZZY-ELA

382.82 1020.3 1207.2 1510.8 1684.2 878.98 885.64 908.54 935.9

132.31 500.69 614.26 796.51 898.25 445.31 452.74 475.55 497.21

177.17 377.36 454.71 560.22 639.71 505.55 509.7 534.4 543.5

119.46 554.54 686.99 902.94 1024 478.48 486.71 510.25 533.44

123.91 619.98 771.65 1014.5 1155.3 478.58 534.76 558.37 582.81

115.65 332.53 407.98 512.63 584.03 405.31 411.6 430.11 448.78

is clearly unable to meet this speed (according to the results shown in Table 7). In order to analyze the performance achieved by hardware approaches, two architectures have been studied for the proposed Fuzzy-ELA algorithm. They differ in the parallelism level employed to calculate the fuzzy rules activation degrees. The parallel design, which is illustrated in Fig. 22, provides a new pixel value every clock cycle and requires three multipliers to implement the defuzzification method of the fuzzy system (4) (taking into account the expression of α4 in (2)). The blocks RO, R1, R2 and R3 in Fig. 22 represent the registers to store the pixels in the upper (Y) and lower scan lines (Y+1). Each register passes its data to the next register and to the “antecedent LARGE”, “antecedent SMALL” and “antecedent strongly SMALL” modules. These modules calculate each membership degree according to the functions described in Section 2. The activation degrees of each rule, which are obtained in parallel by using three minimum circuits, represent the weights of the weighted average implemented by the last block that contains the three multipliers. A sequential design has been also implemented. Its block diagram is represented in Fig. 23. Five two-input minimum circuits implement sequentially the computation of the rules’ activation degrees. These degrees subsequently weight the mean of its corresponding pixels and the result is accumulated. Hence, this architecture only requires one multiplier but three clock cycles to provide the processed pixel value. Both implementations have been developed employing a design methodology which combines the use of different CAD tools. In order to accelerate the design process, it has been especially interesting to generate the VHDL descriptions and testbench files for simulating the system behavior. The implementations have been done on an FPGA Virtex2 from Xilinx which provides a wide variety of flexible features (block RAMs and multipliers), 500k system gates and an internal clock speed of 420 MHz. The postsynthesis results in terms of number of slices occupied in the FPGA are: (a)

182

Piedad Brox et al.

Fig. 22. Block diagram of parallel architecture for the Fuzzy-ELA algorithm

Fig. 23. Block diagram of sequential architecture for the Fuzzy-ELA algorithm

283 (9.2% of the FPGA) for the sequential design and (b) 504 (16.4%) for the parallel approach. In terms of processing time invested per pixel, the results are: (a) 17.73 ns for the sequential design and (b) 7.14 ns for the parallel one. Operating at such speed, it can be seen how the processing of PAL standard TV sequences could be done in real time. Even more, since the maximum resolution per frame that could be provided for frame sequences displayed at a rate of 30 frames per second is 3.760.104 pixels in the sequential case and 9.337.068 pixels in the parallel one, both implementations satisfy in excess HDTV requirements.

6 Conclusions A novel interpolation algorithm has been presented in this Chapter. It employs a simple fuzzy system which models heuristic knowledge to adapt the interpolation to the presence of edges. Two applications have been considered to evaluate the performance of the algorithm: de-interlacing of video sequences


183

and enlargement of images. In both cases, the Fuzzy-ELA algorithm offers a very good tradeoff quality/cost in comparison with other traditional algorithms. It not only eliminates the blurring and the stair-step effect than others introduce but also reconstructs the edges of the images better than the original ELA algorithm, because it is more robust against ambiguous or non-clear edges. The improvements are more significant when the images are corrupted by impulse and Gaussian noise. This has been proven by extensive simulations with video sequences and still images. Two enhancements of the Fuzzy-ELA algorithm have been also analyzed. They improve slightly the quality of results but at expense of increasing the required computational cost. The different architectures for FPGA implementation that have been described show that real-time processing of high-resolution images can be achieved with the FuzzyELA algorithm.

References 1. T. C. Chen and R. J. P. de Figueiredo. Two-dimensional interpolation by generalized spline filters based on partial differential equation image models. IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-33, no.3, pp. 631-642, 1985 2. M. User, A. Aldroubi, and M. Eden. Fast B-spline transforms for continuous image representation and interpolation. IEEE Trans. Pattern Anal. Machine Intell., vol.13, no.3, pp. 277-285, 1991 3. E. Maeland. On the comparison of interpolation methods. IEEE Trans. Medical Imaging, vol. 7, pp. 213-217, 1988 4. T. M. Lehmann, C. Gonner, and K. Spitzer. Survey: Interpolation methods in medical image processing. IEEE Trans. Medical Imaging, vol. 18, pp. 1049-1075, 1999 5. S. W. Lee and J. K. Paik. Image interpolation using fast B-spline filtering. in Proc. IEEE Int. Conf. Acoustics, Speech, Signal Processing, ICASSP, vol. 5, pp. 177-180, 1993 6. G. Ramponi. Warped distance for space-variant linear image interpolation. IEEE Trans. Image Processing, vol. 8, pp. 629-639, 1999 7. K. Jensen and D. Anastasio. Subpixel edge localization and the interpolation of still images. IEEE Trans. Image Processing, vol. 4, pp. 285-295, 1995 8. J. Allebach and P. W. Wong. Edge-directed interpolation. in Proc. IEEE Int. Conf. Image Processing, ICIP, pp. 707-710, 1996 9. S. G. Chang, Z. Cvetkovic, and M. Vetterli. Resolution enhancement of images using wavelet transform extrema interpolation. in IEEE Proc. Int. Conf. Acoustics, Speech, Signal Processing, ICASSP, pp. 2379-2382, 1995 10. S. Baker and T. Kanade. Limits on super-resolution and how to break them. IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 9, pp. 11671183, 2002 11. N. Plaziac. Image interpolation using neural networks. IEEE Trans. Image Processing, vol. 8, no. 11, pp. 1647-1651, 1999 12. F. Michaud, C. T. Le Dinh and G. Lachiver. Fuzzy Detection of Edge-Direction for Video Line Doubling. IEEE Trans. on Circuits and Systems for Video Technology, vol.7, no.3, pp. 539-542, 1997

184

Piedad Brox et al.

13. H. C. Ting and H. M. Hang. Spatially adaptive interpolation of digital images using fuzzy inference. in Proc. SPIE, vol. 27, pt.3, pp. 1206-17, 1996 14. N. Shezaf, H. Abromov-Segal, I. Sutskoner and R. Bar-Sella. Adaptive low complexity algorithm for image zooming at fractional scaling ratio. in Proc. 21st IEEE Convention of the Electrical and Electronic Engineers, pp. 253-256, 2000 15. T. Aso, N. Suetake and T. Yamakawa. A code-reduction technique for an image enlargement by using a som-based fuzzy interpolation. in Proc. 9th Int. Conf. on Neural Information Processing (ICONIP02), vol.3, pp. 1281-86, 2002 16. T. Doyle and M. Looymans. Progressive scan conversion using edge information. in Proc. 3rd Int. Workshop on HDTV, pp. 711-721, Torino, Italy, 1989 17. M. H. Lee, J. H. Kim, J. S. Lee, K.K. Ryu and D. Song. A new algorithm for interlaced to progressive scan conversion based on directional correlations and its IC design. IEEE Trans. on Consumer Electronics, vol.40, no.2, pp. 119-129, 1994 18. C. J. Kuo, C. Liao and C. C. Lin. Adaptive interpolation technique for scanning rate conversion. IEEE Trans. on Circuits and Systems for Video Technology, vol.6, no.3, pp. 317-321, 1996 19. H. Y. Lee, J. W. Park, T. M. Bae, S. U. Choi and Y. H. Ha. Adaptive scan rate up-conversion system based on human visual characteristics. IEEE Trans. on Consumer Electronics, vol.46, no.4, pp. 999-1006, 2000 20. J. Salonen and S. Kalli. Edge adaptive interpolation for scanning rate conversion. in Signal Processing of HDTV IV, Elsevier, pp. 757-764, 1993 21. R. Simonetti, A.P. Filisan, S. Carrato, G. Ramponi and G. Sicuranza. A deinterlacer for IQTV receivers and multimedia applications. IEEE Trans. on Consumer Electronics, vol.39, no.3, pp. 234-240, 1993 22. G. De Haan and R.Lodder. De-interlacing of video data using motion vector and edge information. in Proc. IEEE Int. Conf. on Consumer Electronics (ICCE), pp. 70-71, Los Angeles, USA, 2002 23. Y. L. Chang, S. F. Lin and L. G. Chen. Extended intelligent edge-based line average with its implementation and test method. in Proc. IEEE Int. Symposium on Circuits and Systems (ISCAS), vol.2, pp. 341-344, Vancouver, Canada, 2004 24. H. Yoo and J. Jeong. Direction-oriented interpolation and its application to de-interlacing. IEEE Trans. on Consumer Electronics, vol.48, no.4, pp. 954-962, 2002 25. M. Sugeno and T. Yasukawa. A fuzzy-logic-based approach to qualitative modeling. IEEE Trans. Fuzzy Systems, vol.1, no.1, pp. 7-31, 1993 26. F. J. Moreno-Velo, I. Baturone, R. Senhadji and S. S´ anchez-Solano. Tuning complex fuzzy systems by supervised learning algorithms, in Proc. IEEE International Conference on Fuzzy Systems (FUZZ-IEEE2003), pp. 226-231, 2003 27. F. J. Moreno-Velo, I. Baturone, S. S´ anchez-Solano, A. Barriga. Rapid Design of Complex Fuzzy Systems with XFUZZY, in Proc. IEEE International Conference on Fuzzy Systems (FUZZ-IEEE2003), pp. 342-347, 2003 28. G. De Haan and E.B. Bellers. De-interlacing-An overview. Proc. of the IEEE, vol. 86, pp. 1839-1857, 1988 29. Genesis Microchip, Inc., Preliminary data sheet of Genesis gmVLD8, 8 bit digital videoline doubler, version 1.0, 1996 30. M. Weston. Interpolating lines of video signals. US-patent 4, pp. 789-893, 1998 31. J. W. Woods and S.-C.Han. Hierarchical motion compensated de-interlacing, in Proc. SPIE, vol. 1605, pp. 805-810, 1991


185

32. F. M. Wang, D. Anastassiou, and A. N. Netravali. Time-recursive deinterlacing for IDTV and pyramid coding, Signal Process.: Image Commun. 2, pp. 365-374, 1990 33. M. Zhao and G. De Haan. Content Adaptive Vertical Temporal Filtering for De-interlacing. in Proc. 9th International Symposium on Consumer Electronics, pp. 69-73, 2005 34. G. Chen and R. J. P. de Figueiredo. A unified approach to optimal image interpolation problems based on linear partial differential equation models. IEEE Trans. Image Processing, vol.2, no.1, pp. 41-49, 1993 35. R. G. Keys. Cubic convolution interpolation for digital image processing. IEEE Trans. Acoust., Speech, Signal Processing, vol. ASSP-29, no. 6, pp. 1153-1160, 1981 36. S. C. Park, M.K. Park and M.G. Kang. Super-Resolution Image Reconstruction: A technical overview. IEEE Signal Processing Magazine, vol.3, pp.21-36, 2003

Chapter 7 Fuzzy Image Segmentation Based on Triangular Function and Its n-dimensional Extension Vasile P˘ atra¸scu Department of Informatics Technology Tarom Company Calea Bucurestilor, 224F, Otopeni, jud. Ilfov, Romania [email protected]

Summary This chapter presents new fuzzy cluster algorithms within the image segmentation procedure. Firstly, we present the alternative algorithm for a onedimensional real space. This algorithm is based on triangular function. Secondly, the fuzzy clustering algorithms for the n-dimensional real space and the angular value set are presented, using an extension of the triangular function. We also explain the usage of these algorithms for gray level and color image segmentation. The color image segmentation is applied to the RGB space and to subspaces of an orthonormal color space called IJK.

1 Introduction The triangular function is frequently used for fuzzification methods due to its simplicity. The values of this scalar function are rapidly calculated. Hence, using the triangular function, it is very simple to construct fuzzy partitions for the real number set R. This chapter shows how the clustering algorithms are developed using this function. Moreover, it presents how this function can be extended from the real number set R to the n-dimensional real space Rn . In addition, the n-dimensional variant can be adapted to obtain an extension of the triangular function for the angular value set [0, 2π]. This result is of great significance because it is well known that the circular properties of the angular value set generate some difficulties in building the clustering algorithms. The chapter has the following structure: Section 2 provides the calculus formulae for the triangular function, its extension to the n-dimensional real space Rn and subsequently, the extension to the angular value set; Section 3 presents three fuzzy clustering algorithms: first for the real space R, second for the ndimensional real space Rn and third for the angular value set [0, 2π]; Section V. P˘ atra¸scu: Fuzzy Image Segmentation Based on Triangular Function and Its n-dimensional Extension, StudFuzz 210, 187–207 (2007) c Springer-Verlag Berlin Heidelberg 2007 www.springerlink.com

188

V. P˘ atra¸scu

4 presents an orthonormal system IJK and a perceptual system ISH for the color space followed by a method for the color hue visualization; Section 5 explains four image segmentation methods that use the clustering algorithms earlier presented. The first method is for grey level image segmentation. The next methods are each dedicated to a color image segmentation: the second uses the RGB space, the third uses the chromatic plane JK and the forth uses the color hue subspace H. The description of these four segmentation methods is accompanied by some experimental results. Finally, Section 6 outlines some conclusions.

2 The triangular function and its n-dimensional extension 2.1 The calculus formulae for the triangular function Let a = b be two real numbers. The open triangular function can be defined as: x−a ω(x, b; a) = max min 1, ,0 (1) b−a If a < b the formula (1) defines a right-open triangular function and if a > b the formula (1) defines a left-open triangular function. Let us note that the function ω verifies the next four properties: ω(a, b; a) = 0 ω(b, b; a) = 1

ω(

1 a+b , b; a) = 2 2

ω(x, b; a) + ω(x, a; b) = 1

(2)

Expression (2) yields that the functions ω(x, b; a) and ω(x, a; b) determine a fuzzy partition for the space R. Let a < b < c be three real numbers. The triangular function can be defined as: t(x, b; a, c) = ω(x, b; a) ∧ ω(x, b; c)

(3)

where ∧ refers to the function “min” or to the algebraic product “·”. In other words, the fuzzy set defined by the triangular membership function represents the intersection between two open fuzzy sets.

Fuzzy Image Segmentation Based on Triangular Function

189

It is possible to find other calculus formula for the triangular function. Thus, this function can have the following form: f (x) = α · |x − a| + β · |x − b| + γ · |x − c| The determination of the function parameters is completed by using the function values f (a), f (b), f (c) in the points a, b, c. It is necessary to solve this system prior to determination:   β · |a − b| + γ · |a − c| = f (a) α · |b − a| + γ · |b − c| = f (b)  α · |c − a| + β · |c − b| = f (c)

(4)

The system (4) has the following solution:  f (c)+f (a) f (b)−f (a) 1  α = · +  2 c−a b−a   (b) f (b)−f (a) + β = 12 · f (c)−f c−b b−a     γ = 1 · f (c)+f (a) − f (c)−f (b) 2 c−a c−b For the particular function values f (a) = 0, f (b) = 1, f (c) = 0, it results the solution:   α = β=   γ= and f (x) =

1 1 |x − a| · ·+ · 2 b−a 2

1 2 1 2 1 2

1 · b−a

1 · b−a − 1 · c−b

1 c−b

1 1 − b−a c−b

· |x − b| +

1 |x − c| · 2 c−b

or f (x) =

1 |x − a| − |x − b| 1 |x − c| − |x − b| · + · 2 b−a 2 c−b

(5)

Similarly, we can obtain an equivalent formula for the open triangular function. Thus, for two real numbers a = b the function: ω(x, b; a) =

1 |x − a| − |x − b| 1 · + 2 |b − a| 2

(6)

defines also, an open triangular function. This function is important because it can be extended in those spaces that have no order structure. At practical level, we can use instead of the formula (6) the following:

190

V. P˘ atra¸scu

ω(x, b; a) =

1 |x − a| − |x − b| + ε 1 · + 2 |b − a| + ε 2

(7)

where 1ε>0

(8)

2.2 The triangular function extension for the n-dimensional real space Let a, b ∈ Rn , with the representation a = (a1 , ..., an ) and b = (b1 , ..., bn ). The scalar product ·, · : Rn × Rn → R and the norm || · ||: Rn → [0, ∞) are defined by: a, b =

n

ai · bi

i=1

|| a ||2 =

n

a2i

i=1

The triangular function definition uses the following auxiliary function : h(x, b; a) =

x−a b−a

(9)

The function h(x, b; a) can not be used in the set Rn because there is no commutative division algebra structure defined. However, the formula (9) can be written in the following equivalent form: h(x, b; a) · (b − a) = x − a

(10)

Expression (10) can be considered as an equation with the known vectors a, b, x and the undetermined scalar h(x, b; a). Let us define: e = h(x, b; a) · (b − a) − (x − a)

(11)

The equation (11) will be solved in such a way to obtain the minimum value of the norm || e ||2 . The result is next solution: h(x, b; a) =

x − a, b − a || b − a ||2

(12)

We can now extend the open triangular function, from the set R to the set Rn . Thus, the following function results from the expressions (6), (9) and (12): ω(x, b; a) =

1 |x − a, b − a| − |x − b, a − b| 1 · + 2 || b − a ||2 2

(13)


191

It is necessary to prove that the function (13) has its values in the interval [0, 1]. Consequently, the following sequence of inequalities exists for ∀A, B ∈ R: | A − B |≥| A | − | B |≥ − | A − B | 1≥

1≥

|A|−|B| ≥ −1 |A−B |

1 |A|−|B| 1 · + ≥0 2 |A−B | 2

(14)

For A = x − a, b − a and B = x − b, a − b, (14) yields: 1 ≥ ω(x, b; a) ≥ 0 At practical level, we can use for the function ω(x, b; a) calculus the following formula: ω(x, b; a) =

1 |x − a, b − a| − |x − b, a − b| + ε 1 · + 2 || b − a ||2 +ε 2

(15)

where ε verifies (8). Also, the following equivalent formula of the function ω(x, b; a) results from (1), (9) and (12): x − a, b − a ω(x, b; a) = max min 1, ,0 (16) || b − a ||2 The extension of the triangular function t(x, b; a, c) will be done using the / {a1 , a2 , ..., ak }, we define the formula (3). For a1 , a2 , ..., ak ∈ Rn and b ∈ n-dimensional extension as: % t(x, b; a1 , a2 , ..., ak ) = ω(x, b; aj ) (17) j=1,k

2.3 The triangular function extension for the angular value set A particular case for the one-dimensional real spaces is the angular value set, namely the interval [0, 2π]. The formulae presented in section 2.1 for the real number set R can not be used due to the circular properties of the angular value set. However, an expression can be obtained on the same principles as defined for the n-dimensional real space Rn . Geometrically, the interval [0, 2π] can be represented by the unit circle, since each point (cos(x), sin(x)) from the unit circle can be associated to each angle x ∈ [0, 2π]. Consequently, the

192

V. P˘ atra¸scu

following vectors (cos(x), sin(x)), (cos(a), sin(a)), (cos(b), sin(b)) will associate to the angles x, a, b. Therefore, the function h(x, b; a) defined by (12) becomes: sin x−a · cos x−b 2 2 h(x, b; a) = sin b−a 2 The formulae (13) and (16) become: x−b · cos x−b · cos x−a 1 sin x−a 2 2 − sin 2 2 b−a ω(x, b; a) = + 2 2 · sin 2

ω(x, b; a) = max min 1,

sin

· cos x−b b−a 2 ,0

x−a 2

sin

2

The triangular function extension is calculated using (17). / {a1 , a2 , ..., ak }, the extension of Thus, for a1 , a2 , ..., ak ∈ [0, 2π] and b ∈ the triangular function is defined for the set [0, 2π] as: % ω(x, b; aj ) (18) t(x, b; a1 , a2 , ..., ak ) = j=1,k

The formula (18) can be simplified in contrast to the n-dimensional case. Without losing generality, but assuming that a1 < a2 < ... < ak , let us denote: a0 = ak The intervals [a0 , a1 ], [a1 , a2 ], . . . , [αk−1 , ak ] form a partition on the unit circle and let be the index j so that b ∈ (aj−1 , aj ). In this case, the formula ( 18) can be simplified: t(x, b; aj−1 , aj ) = ω(x, b; aj−1 ) ∧ ω(x, b; aj )

(19)

The formula (19) is similar to (3).

3 The fuzzy clustering algorithms 3.1 The fuzzy clustering algorithm for the real number set Let there be the real set X = {x1 , x2 , ..., xm } ⊂ R and k real numbers µ1 , µ2 , ..., µk in ascending order, namely: µ1 < µ2 < ... < µk A fuzzy partition A1 , A2 , ..., Ak for the set X may be organized using the triangular function. Firstly, we consider the membership functions:


193

w1 (x) = ω(x, µ1 ; µ2 ) for i = 2, 3, ..., k − 1 wi (x) = ω(x, µi ; µi−1 ) ∧ ω(x, µi ; µi+1 ) wk (x) = ω(x, µk ; µk−1 ) The functions w1 , w2 , ..., wk define a unit partition owing to the following equality: w1 + w2 + ... + wk = 1 Secondly, we define the defuzzifier operator τ (w, γ) that will be applied to the fuzzy partition A1 , A2 , ..., Ak [12, 13]: τ (w, γ) = (τ1 , τ2,..., τk )

(20)

wiγ (x) k wjγ (x)

(21)

where τi (x, w, γ) =

j=1

The real number γ ≥ 1 was used as defuzzification parameter. The operator τ (w, γ) is similar to the BADD operator defined by Filev and Yager [5]. Let νi and Fi defined by νi (x) = wi (x) ∧ τi (x, w, γ) m

Fi (µi ) =

νi (xj ) j=1 m

(22)

· xj (23)

νi (xj )

j=1

We now consider the following constraint for the parameters µ1 , µ2 , ..., µk : µi = Fi (µi )

(24)

The fuzzy set Ai is determined by the equation (24). The center µi belongs to the convex hull of the set X and it is a fixed-point for the function Fi . The existence of the solution results from the Shauder fixed-point theorem [16, 8]. This equation does not have an analytical solution and may be solved using numerical methods. The result is the subsequently fuzzy clustering algorithm.

194

V. P˘ atra¸scu

Step 1. There are initialized the number of clusters k, the defuzzification parameter γ, the procedure stopping parameter δ, the iteration index l = 0, (0) (0) (0) and the cluster centers µ1 , µ2 , ..., µk . Next, the fuzzy membership func(0) (0) (0) (0) (0) (0) tions w1 , w2 , ..., wk , the defuzzifier components τ1 , τ2 , ..., τk and the (0) (0) (0) functions ν1 , ν2 , ..., νk are calculated. Step 2. The iteration index is incremented, i.e. l → l + 1. We calculate (l) (l−1) (l) (l−1) (l) (l−1) ), µ2 = F2 (µ2 ), ..., µk = Fk (µk ), the cluster centers µ1 = F1 (µ1 (l) (l) (l) the fuzzy membership functions w1 , w2 , ..., wk , the defuzzifier components (l) (l) (l) (l) (l) (l) τ1 , τ2 , ..., τk and the functions ν1 , ν2 , ..., νk . Step 3. Calculate d =

k (l) (l−1) µi − µi . If d > δ then return to step 2,

i=1

otherwise go to step 4. Step 4. Save the data and stop. 3.2 The fuzzy clustering algorithm for the n-dimensional real space Let be the set X = {x1 , x2 , ..., xm } ⊂ Rn and the vectors µ1 , µ2 , ..., µk ∈ Rn . The fuzzy sets A1 , A2 , ..., Ak will be defined using the n-dimensional extension of triangular function (17). Firstly, we consider the membership functions: % ω(x, µi ; µj ) (25) wi (x) = j=1,k ; j=i

Secondly, we define the defuzzifier operator components τ1 , τ2 , ..., τk for the fuzzy sets A1 , A2 , ..., Ak using (21) and the functions ν1 , ν2 , ..., νk using (22). We now consider the following constraint for the parameters µ1 , µ2 , ..., µk : µi = Fi (µi )

(26)

where Fi (µi ) is defined by (23). The fuzzy set Ai is determined by the equation (26). The problem can be solved using the same algorithm steps that were described in the section 3.1.

3.3 The fuzzy clustering algorithm for the angular value set Let X = {x1 , x2 , ..., xm } ⊂ [0, 2π] be a set and the angles µ1 , µ2 , ..., µk ∈ [0, 2π], in ascending order, namely: µ1 < µ2 < ... < µk


195

A fuzzy partition A1 , A2 , ..., Ak for X can be organized using the extension of the triangular function defined by (19). Firstly, we consider the membership functions: wi (x) = ω(x, µi ; µi−1 ) ∧ ω(x, µi ; µi+1 )

(27)

where µ0 = µk and µk+1 = µ1 . The functions w1 , w2 , ..., wk define a unit partition, owing to the following equality: w1 + w2 + ... + wk = 1 Secondly, we define the defuzzifier operator components τ1 , τ2 , ..., τk for the fuzzy sets A1 , A2 , ..., Ak using (21) and the functions ν1 , ν2 , ..., νk using (22). We can not apply the formula (23) to compute the center of a fuzzy angular set. The required real number algebra becomes inadequate due to the circular properties of the angular values. Thus, there were defined special functions and operations [7, 14, 19]. Geometrically, the unit circle is very well capable of the angular value set representation. Therefore, the point (cos(x), sin(x)) may associate to each angle x ∈ [0, 2π]. Given this property and using the function atan2 , as defined by the international standard ECMA-262 [20], we can define the following constraint for the the parameters µ1 , µ2 , ..., µk : µi = Fi (µi )

(28)

where  m

m

νi (xj ) · sin(xj )

 j=1 Fi (µi ) = atan2  m 

, νi (xj )

j=1

νi (xj ) · cos(xj )

j=1 m

νi (xj )

   

(29)

j=1

The fuzzy partition is determined by the equations (28). Again, these equations will be solved using the algorithm that was described in the section 3.1.

4 The color systems IJK and ISH 4.1 The orthonormal color system IJK The best known system for color representation is the RGB (red, green, blue). Its components have values in the interval [0, 255]. RGB belongs to the color primary representation class. Munsell has defined a color specification system based on luminosity, saturation and hue [3, 9]. These three attributes form a

196

V. P˘ atra¸scu

perceptual representation system that is obtained by some transforms applied to the RGB system [2]. Next, the calculus procedure for an orthonormal coordinates system, called IJK, will be presented [11]. Every color q from the RGB system is described by a vector having three scalar components: q = (R, G, B) Let u, v and w be three real and positive numbers verifying the equality: u+v+w =1 The following scalar product will be defined in the RGB space: for q1 = (R1 , G1 , B1 ) and q2 = (R2 , G2 , B2 ) q1 , q2 = u · R1 R2 + v · G1 G2 + w · B1 B2

(30)

From (30) it results the norm: || q ||=

uR2 + vG2 + wB 2

Let there be the following basis in the RGB space: f1 = (1, 1, 1), f2= (1, 0, 0), f3 = (0, 1, −1). Using the Gram-Schmidt procedure [15], it results the following orthonormal basis: e1 = (1, 1, 1) , e2 =

, , 1−u u u ,− ,− u 1−u 1−u

, e3 = 0,

, w v ,− v(1 − u) w(1 − u)

The coordinates I, J, K in the basis e1 , e2 , e3 are computed by: I = q, e1 = uR + vG + wB J = q, e2 =

vG + wB u (1 − u) · R − v+w ,

K = q, e3 =

vw (G − B) 1−u

Also, knowing the I, J, K components, the R, G, B components can be computed using the following expressions:


, R=I+ , G=I− , B=I−

1−u J u

u J+ 1−u u J− 1−u

,

,

197

(31)

w K v(1 − u)

(32)

v K w(1 − u)

(33)

If R, G, B ∈ [0, 255] then I, J, K verify the inequalities: 0 ≤ I ≤ 255 | J |≤ 255 ·

u(1 − u) ,

| K |≤ 255 ·

vw 1−u

This chapter uses the following value for the constants u, v, w: u=v=w=

1 3

As a result, the components I, J, K have the following particular expressions: I=

R+G+B 3

(34)

J=

2R − G − B √ 3 2

(35)

G−B √ 6

(36)

K=

4.2 The perceptual color system ISH Using the IJK system there will be defined a perceptual color system called ISH. Given that the basis e1 , e2 , e3 is an orthonormal one, the next equality follows: || q ||= I 2 + J 2 + K 2 We observe that I is the achromatic component of the color and J, K are the chromatic components. The saturation can be defined as:

198

V. P˘ atra¸scu

S=

J 2 + K2

Saturation S has the following equivalent expressions: S = uR2 + vG2 + wB 2 − I 2 S= S=

u(R − I)2 + v(G − I)2 + w(B − I)2

uv(R − G)2 + vw(G − B)2 + wu(R − B)2

If R, G, B ∈ [0, 255] then S verifies the inequality: S ≤ 255 · max(u, v, w) · (1 − max(u, v, w)) If S > 0, we define the hue H as the angle that verifies the expressions: cos (H) = SJ sin (H) =

K S

(37)

Applying the function atan2 [20], the two expressions (37) are equivalent with: H = atan2(K, J)

(38)

From (31), (32), (33) and (37) it results: , 1−u R=I+ S cos (H) u G=I−√

, √ w 1 sin (H) u cos (H) − ·S· v 1−u

, √ 1 v B=I−√ sin (H) u cos (H) + ·S· w 1−u For u = v = w = 13 the components S, R, G and B have the following particular expressions: (R − G)2 + (G − B)2 + (R − B)2 S= 3 R=I+ G=I+

B=I+

√ 2S cos (H)

(39)

√ 2π 2S cos H − 3

(40)

4π 2S cos H − 3

(41)

√


199

4.3 The hue visualization using color images Due to the circular properties of the angular values, the hue visualization using a gray level image is not adequate. For example, 0 and 2π represent the same hue. In the gray level visualization method, black is associated to 0 and white to 2π. Thus, two gray levels are associated to the same hue. The color images are more adapted to hue visualizations. Let H ∈ [0, 2π] be a hue. For S = √I2 and I = 127.5, from (39), (40), and (41) it results: H 2 R = 255 · cos (42) 2 G = 255 · cos

2

B = 255 · cos

2

H π − 2 3

H 2π − 2 3

(43) (44)

Within the image of color hue visualization, each hue H is associated to a color (RH , GH , BH ); the scalar components being calculated using the expressions (42), (43) and (44). Figure 1 shows the image ,,flowers” and its hue component visualization.

a)

b)

Fig. 1. The image “flowers” (a) and its hue component visualization (b)

5 The fuzzy image segmentation methods 5.1 The gray level image segmentation A gray level image is described by a function f : Ω → [0, 255], where Ω ⊂ R2 is the image support. The clustering algorithm is applied to the image

200

V. P˘ atra¸scu

value set f (Ω) ⊆ [0, 255], in order to obtain the image segmentation [6, 10]. Next, the fuzzy partition A1 , A2 , ..., Ak of f (Ω) is obtained; and afterwards, Ω1 , Ω2 , ..., Ωk a fuzzy partition of image support Ω, having the membership functions u1 , u2 , ..., uk defined by: ∀(x, y) ∈ Ω, ui (x, y) = wi (f (x, y)) We may say that the fuzzy sets Ω1 , Ω2 , ..., Ωk define a fuzzy segmentation of the image support Ω [6]. The functions u1 , u2 , ..., uk verify the equality: ∀(x, y) ∈ Ω, k

ui (x, y) = 1

i=1

The fuzzy cardinality of Ωi represents a measure for its area [6], namely: area (Ωi ) = ui (x, y) (x,y)∈Ω

After the clustering, the segmented image can be visualized using the formula defined in the rule-based contrast enhancement method [4, 17, 18]: k

T (f (x, y)) =

ui (x, y) · vi

i=1 k

(45) ui (x, y)

i=1

where v1 , v2 , ..., vk are associated to the fuzzy regions Ω1 , Ω2 , ..., Ωk representation. There are two choices of settings for these values: the first: vi = µi and the second:



 i−1 1 255 vi =  area (Ωi ) + area (Ωj ) · 2 area (Ω) j=1

The formula (45) can be generalized using the defuzzifier operator τ (u, λ): T (f (x, y)) =

k

τi ((x, y), u, λ) · vi

(46)

i=1

The fuzzy clustering algorithm was applied to image “Lena” (Fig. 2(a)). The segmented image is shown in Fig. 2 (b) and the graphs of membership functions are shown in Fig. 3.


a)

201

b)

Fig. 2. The image “Lena” (a) and its segmentation for k = 3 (b)

Fig. 3. The graphs of membership functions

5.2 The color image segmentation in the space RGB A color image f is described by three scalar functions fR : Ω → [0, 255], fG : Ω → [0, 255], fB : Ω → [0, 255] where Ω ⊂ R2 is the image support. The clustering algorithm is applied to the image value set f (Ω) ⊆ [0, 255]3 , in order to obtain the image segmentation. We then obtain the fuzzy sets A1 , A2 , ..., Ak having the centers µ1 , µ2 , ..., µk ∈ [0, 255]3 and the membership functions w1 , w2 , ..., wk . Next, we can define the fuzzy sets Ω1 , Ω2 , ..., Ωk having the following membership functions u1 , u2 , ..., uk : ∀(x, y) ∈ Ω, ui (x, y) = wi (fR (x, y) , fG (x, y) , fB (x, y)) Following the clustering procedure, we find the segmented image by using the transformation function defined by (46):

202

V. P˘ atra¸scu

T (f (x, y)) =

k

τi ((x, y), u, λ) · µi

(47)

i=1

The clustering algorithm was applied to images “pool” and ,,parrots” (Fig. 4 a, b). The segmented images obtained using the proposed algorithm and the fuzzy c-means algorithm [1] are shown in Fig. 4 (c, d, e, f).

a)

b)

c)

d)

e)

f)

Fig. 4. The image “pool” (a) and the image ,,parrots” (b); The image segmentation using the proposed algorithm (c, d ) and using the FCM algorithm (e, f )


203

5.3 The color image segmentation in the subspace JK In order to avoid the influences of nonuniform illumination, we can apply the clustering procedure only for the JK chromatic subspace. Thus, for a color image f having the scalar components fR , fG , fB , we calculate the IJK components using the expressions (34), (35), (36): fI =

fR + fG + fB 3

fJ =

2fR − fG − fB √ 3 2

(48)

fG − fB √ 6

(49)

fK =

The clustering algorithm is applied to the image value set situated in the chromatic plane JK, namely (fJ (Ω) , fK (Ω)) ⊆ (−121, 121) × (−105, 105). After that, the fuzzy sets A1 , A2 , ..., Ak of (fJ (Ω) , fK (Ω)) are obtained, having the membership functions w1 , w2 , ..., wk . Then, the fuzzy sets Ω1 , Ω2 , ..., Ωk result, having the membership functions u1 , u2 , ..., uk defined by: ∀(x, y) ∈ Ω, ui (x, y) = wi (fJ (x, y) , fK (x, y)) Following the clustering procedure, we find the segmented image by using the transformation function: T (f (x, y)) =

k

τi ((x, y), u, λ) · vi

(50)

i=1

where vi is the mean of f computed for the fuzzy region Ωi , namely: vi =

ui (x, y) · f (x, y)

(x,y)∈Ω

ui (x, y)

(x,y)∈Ω

The algorithm was applied to image “hand” (Fig. 5). The segmented image using the JK space and the RGB space can be seen in Fig. 6. 5.4 The color image segmentation in the hue subspace Another possibility to eliminate the influence of a nonuniform illumination is the hue clustering [2]. For a color image f having the scalar components

204

V. P˘ atra¸scu

Fig. 5. The image “hand”

a)

b)

Fig. 6. The segmentation using the JK space (a) and the RGB space (b)

fR , fG , fB we calculate fJ , fK using (48), (49) and then, the hue fH using (38): fH = atan2 (fK , fJ ) In order to obtain the image segmentation, the clustering algorithm is applied to the angular value set fH (Ω) ⊆ [0, 2π]. After the clustering procedure there results the partition A1 , A2 , ..., Ak having the membership functions w1 , w2 , ..., wk , the centers µ1 , µ2 , ..., µk and the fuzzy partition Ω1 , Ω2 , ..., Ωk of the support Ω having the membership functions u1 , u2 , ..., uk defined by: ∀(x, y) ∈ Ω, ui (x, y) = wi (fH (x, y)) Finally, the segmented image is obtained using the transformation function defined by (46): T (f (x, y)) =

k i=1

τi ((x, y), u, λ) · vi

(51)


205

where vi = (ri , gi , bi ) are computed using (42), (43), (44), namely: µ i ri = 255 · cos2 2 gi = 255 · cos2

µ

bi = 255 · cos2

i

2

−

π 3

µi 2π − 2 3

The clustering algorithm for angular values was applied to image ,,house”. The image “house” and its hue component can be seen in Fig. 7. Two hue segmentations can be seen in Fig. 8. The membership function graphs are shown in Fig. 9.

a)

b)

Fig. 7. The image “house” (a) and its hue component visualization (b)

a)

b)

Fig. 8. The hue segmentation for k = 3 (a) and for k = 6 (b)

206

V. P˘ atra¸scu

a)

b)

Fig. 9. The membership function graphs for k = 3 (a) and for k = 6 (b)

6 Conclusions This chapter presented image segmentation using a new fuzzy clustering algorithm. The presented algorithm is based on the triangular function and its n-dimensional extension. Starting from a particular case of an n-dimensional extension, an algorithm for the angular value set was obtained. Also, for the color image clustering, an orthonormal coordinate system was used. A future research object could be using a parameterized formula for the function νi , namely νi = wiβ ∧ τiα with finding optimal values for the parameters α and β.

References 1. Bezdek J.C. (1981) Pattern Recognition with Fuzzy Objective Function Algorithms. Plenum Press, New York 2. Carron I., Lambert P. (1996) Symbolic Fusion of Hue-Chroma-Intensity Features for Region segmentation. ICIP’96, Proceedings of the IEEE Conference on Image Processing, Lausanne, Switzerland, pp. 971-974 3. Castleman K.R. (1996) Digital Image Processing. Prentice Hall, Englewood Cliffs Nj 4. Chen T.B., Chen Y.S., Hsu H.W. (1994) Image processing and understanding based on fuzzy inference approach. Proceedings of FUZZ-IEEE’94, Vol1, pp. 254-259. 5. Filev D., Yager R.R. (1991) A generalized defuzzification method under BADD distributions, Intl. Journal of Intelligent Systems 6, pp. 687-697. 6. Foliquet S.P., Vieira M.B., Sanfourche M. (2002) Fuzzy segmentation of color images and indexing of fuzzy regions. First European Conference on Color in Graphics, Imaging and Vision, CGIV2002, pp. 507-512, Poitiers, France 7. Hanbury A.G., Serra J. (2001) Morphological Operators on the Unit Circle. IEEE Transactions on Image Processing, Vol. 10, No. 12 8. Istratescu V. (1981) Fixed Point Theory, An Introduction. D.Reidel, Holland


207

9. Jain A.K. (1989) Fundamentals of Digital Image Processing. Prentice Hall, Englewood Cliffs Nj 10. Lambert P., Grecu H. (2002) Fast Color Image Segmentation using Fuzzy Clustering. First European Conference on Color in Graphics, Imaging and Vision, CGIV2002, pp. 527-528, Poitiers, France 11. Patrascu V. (1999) A spherical coordinate system in the color space. The 7th Conference on Applied and Industrial Mathematics CAIM’99, Pitesti, Romania 12. Patrascu V. (2003) Color Image Enhancement Using the Support Fuzzification. In Fuzzy Sets and Systems - IFSA 2003, Vol LNAI 2715, (Eds. T. Bilgi¸c, B. De Baets, O. Kaynak), Springer-Verlag Berlin Heidelberg 2003, Proceedings of the 10th International Fuzzy Systems Association World Congress, pp. 412-419, Istanbul, Turkey 13. Patrascu V. (2004) Image Enhancement Method Using a New Measure for Mean Dynamic Range. Proceedings of the conference IPMU’04, Information Processing and Management of Uncertainly in Knowledge-Based Systems, pp. 19211928, Perugia, Italy 14. Peters R.A. (1997) Mathematical morphology for angle-valued images. Proceedings of the SPIE, Nonlinear Image Processing VIII, Vol. 3026, pp. 84-94 15. Reza F. (1971) Linear spaces in engineering. Waltham, Massachusetts 16. Schauder J. (1930) Der Fixpunktsatz in Funktionalr¨ aumen. Studia Mathematica 2, pp. 171-180 17. Tizhoosh H.R. (2000) Fuzzy Image Enhancement: An Overview. Fuzzy Techniques in Image Processing, (Eds. M Nachtegael, E. Kerre), pp. 137-171 Springer Verlag 18. Tizhoosh H.R., Krell G., Michaelis B. (1997) On fuzzy image enhancement of megavoltage images in radiation therapy. In Proceedings of FUZZY-IEEE’97 conference, pp. 1399-1404, Barcelona, Spain 19. Zhang C., Wang P. (2000) A New Method of Color Image Segmentation Based on Intensity and Hue Clustering. In proceedings of the 15th ICPR2000 Conference, Vol. 3, pp. 617-620, Barcelona, Spain 20. www.ecma-international.org/publications/standards/Ecma-262.htm.

Chapter 8 Application of Neuro-Fuzzy Methods for Noise Filtering, Noise Detection and Edge Extraction in Digital Images Corrupted by Impulse Noise Emin Y¨ uksel Department of Electrical and Electronics Engineering Erciyes University Kayseri, 38039, Turkey [email protected]

1 Introduction One of the most important application areas of soft computing in image processing is image restoration. In recent years, various soft computing techniques have successfully been employed for detection and removal of noise from digital images. Although the complexity of the soft computing based image restoration methods are somewhat higher than that of the conventional methods, their restoration performance is significantly better. Moreover, the undesirable distortions and blurring effects introduced into the image during restoration are much less compared to the conventional methods. In this chapter, we will begin by a thorough review of the conventional as well as state-of-the-art methods of image restoration available in the literature. Next, we will present a generalized neuro-fuzzy (NF) based operator suitable for a number of different applications in image restoration. Specifically, we will consider three different uses of the presented NF network: (1) noise filter, (2) noise detector and (3) edge detector. We will pay a particular emphasis to noise filtering since this has been one of the most active research areas in image processing in the last two decades. In the noise filter application, the NF network will be employed as a detailpreserving noise filtering operator to restore digital images corrupted by impulse noise without degrading fine details and texture in the image. In the noise detector application, the NF network will be utilized to detect impulses in images corrupted by impulse noise. Hence, the NF network will be used to guide a noise filter so that the filter will restore only the pixels that are detected by the NF network as impulses, and leave the other pixels (i.e. the uncorrupted pixels) unchanged. Consequently, the NF network will help reduce the undesirable distortion effects of the noise filter. In the edge detector application, the NF network will be used to extract edges from digital images E. Y¨ uksel: Application of Neuro-Fuzzy Methods for Noise Filtering, Noise Detection and Edge Extraction in Digital Images Corrupted by Impulse Noise, StudFuzz 210, 209–235 (2007) c Springer-Verlag Berlin Heidelberg 2007 www.springerlink.com

210

E. Y¨ uksel

corrupted by impulse noise without needing a pre-filtering of the image by an impulse noise filter. In all of these applications, the same NF operator will be used. The fundamental building block of the presented NF operator is a simple 3-input 1output NF system. We will then show that highly efficient noise filtering, noise detection or edge detection operators may easily be constructed by combining a desired number of simple NF blocks within a suitable network structure. Following this, we will present a simple approach for training the NF operator for its particular use. Specifically, we will show that the internal parameters of the NF blocks in the structure of the presented operator may adaptively be optimized by employing simple training images, and the same NF operator may be trained as a noise filter, noise detector or an edge extractor depending on the choice of the training images. We will also show that the NF blocks may be trained by using simple artificial images that can easily be generated in a computer. For each of the three applications of the presented NF network, we will demonstrate the efficiency of the presented approach by appropriately designed simulation experiments and also compare their performance with a number of selected operators from the literature. We will complete the chapter with a brief summary of the advantages and other potential applications of the presented NF operator in image processing. The rest of the chapter is organized as follows: Section-2 provides a detailed literature review of the existing methods of image restoration. Section-3 explains the structure of the NF operator and its building blocks. Section-4 discusses the three applications of the NF operator as a noise filter (Section4.1), noise detector (Section-4.2) and edge detector (Section-4.3). Results of the simulation experiments conducted to evaluate the performance of the presented operator for different applications and comparative discussion of these results are also presented in Section-4. Section-5, which is the final section, summarizes the advantages of the presented NF operator and presents the conclusions and remarks.

2 Literature review Digital images are often corrupted by impulse noise during image acquisition and/or transmission due to a number of non-idealities encountered in image sensors and communication channels. In most image processing applications, it is of vital importance to remove the noise from the image data because the performances of subsequent image processing tasks (such as segmentation, feature extraction, object recognition, etc.) are severely degraded by noise [1]. Many different filtering methods have been proposed for the removal of impulse noise from digital images. A great majority of these methods are based on median filtering techniques, which utilize the rank order information of the pixels contained in the filtering window. The standard median filter [1]

NF Methods for Noise Filtering, Noise Detection and Edge Extraction

211

attempts to remove impulse noise by replacing the center pixel of the filtering window with the median of the pixels in the window. Although this approach provides a reasonable noise removal performance, it removes thin lines, distorts edges and blurs image details even at low noise densities. The weighted median filter [2] and the center-weighted median filter [3] are modified median filters giving more weight to the selected pixels of the filtering window and are proposed to avoid the inherent drawbacks of the standard median filter. These filters demonstrate better edge and detail preservation performance than the median filter at the cost of reduced noise removal performance. A method for optimal design of these filters is also presented in [4]. The standard and the weighted median filters are spatially invariant operators that are incapable of making distinction between the corrupted and the uncorrupted pixels of the noisy input image. Hence, these filters distort the uncorrupted regions of the image while restoring the corrupted regions, causing undesirable blurring effects in the output image. In response to this problem, the switching median filter [5], in which an impulse detector is employed to classify the center pixel of the filtering window, is proposed. If the center pixel is classified as corrupted, the window is filtered by the standard median filter (i.e. the center pixel is replaced with the median of the pixels within the window). Otherwise, the window is not filtered (i.e. the center pixel is left unchanged). Although this approach provides considerable improvement over the standard median filter, its performance naturally depends on the performance of the impulse detector. As a consequence, various impulse detection methods and switching median filter structures have been proposed [5]–[13]. The tri-state median filter [14] and the multi-state median filter [15] are further improved switching median filters that are constructed by including an appropriate number of center-weighted median filters into the basic switching median filter structure. These filters exhibit better performance than the standard and the switching median filters at the expense of increased computational complexity. The progressive switching median filter [16] is also a different type of switching median filter that iteratively accomplishes the detection and removal of impulse noise in two separate stages. Although this filter provides improved filtering performance than many other median-based filters, it has a high computational complexity due to its iterative nature. The partition based median filter [17] is an adaptive median filter that classifies the observed sample vector at each pixel location into one of a number of mutually exclusive partitions. The partitions are constructed by utilizing the differences between the current pixel and the outputs of a number of center-weighted median filters with variable center weights. The estimate at each pixel location is calculated as a linear combination of the outputs of the center-weighted median filters and the current pixel value. The weights are optimized using the constrained LMS algorithm. The partition based median filter has been shown to be capable of suppressing both impulse noise and Gaussian noise.

212

E. Y¨ uksel

The signal-dependent rank-ordered mean filter [18] is a switching mean filter that exploits rank order information for impulse noise detection and removal. The structure of this filter is similar to that of the switching median filter except that the median filter is replaced with a rank-ordered mean filter. This filter has been shown to exhibit better noise suppression and detail preservation performance than some conventional and state-of-the-art impulse noise cancellation filters for both grey scale [18] and color [19] images. The minimum/maximum exclusive mean filter [20] is another type of switching mean filter. The filter demonstrates good filtering performance but it has an increased computational complexity. The peak and valley filter [21] is a highly efficient recursive nonlinear filter. It identifies noisy pixels by inspecting their neighborhood, and then replaces their values with the most conservative ones out of the values of their neighbors. In this way, no new values are introduced into the neighborhood and the histogram distribution range is conserved. The main advantage of this filter is its simplicity and speed, which make it very attractive for real time applications. A modified peak and valley filter [22] has also been proposed. This filter provides very good detail preservation performance but it is slower than the original peak and valley filter. The adaptive two-pass rank order filter [23] has been proposed to remove impulse noise from highly corrupted images. Between the passes of filtering, an adaptive process detects irregularities in the spatial distribution of the estimated noise and selectively replaces some pixels changed by the first pass with their original values. These pixels are kept unchanged during the second filtering. Consequently, the reconstructed image maintains a higher degree of fidelity and has a smaller amount of noise. In [24], a detail-preserving variational method has been proposed to restore impulse noise. It uses a non-smooth data fitting term together with edge-preserving regularization functions. A combination of this variational method [24] with an impulse detector [6] has also been presented in [25] for the removal of random valued impulse noise. The filter offers good filtering performance but its implementation complexity is higher than most of the previously mentioned filters. The two-output nonlinear filter [26] is another rank order filter based on the subsequent activation of two recursive filtering algorithms that operate on different subsets of input data. Two pixel values are updated at each processing step. A nonlinear mechanism for error correction is also provided for avoiding detail blur. The filter provides very good detail preservation performance. The threshold boolean filter [27] employs boolean functions for impulse noise removal. In this approach, the gray level noisy input image is decomposed into a number of binary images by gray level thresholding. Detection and removal of impulse noise are then performed on these binary images by utilizing specially designed boolean functions. Finally, the resulting boolean images are combined back to obtain a restored grey level image.


213

A number of filters utilize the histogram information of the input image. In [28], histograms of homogenous image regions are used to characterize and classify the corrupting noise. Homogeneity level information is also employed for impulse detection in a similar study [8]. In [29] and [30], the histogram information of the input image is used to determine the parameters of the membership functions of an adaptive fuzzy filter. The filter is then used for the restoration of noisy images. An adaptive vector filter exploiting histogram information is also proposed for the restoration of color images [31]. Majority of the above mentioned filtering methods more or less have the drawback of removing thin lines, distorting edges and blurring fine details and texture in the image during noise removal process. Moreover, their performances depend on one or more tuning parameters such as, noise density estimate, pixel weighting factors, impulse detection thresholds, etc. The values of these parameters are heuristically determined and externally supplied by the user for each individual image and/or noise density since there is no analytical method to determine the optimal values of these parameters for a given filtering experiment. In the last few years, there has been a growing research interest in the applications of soft computing techniques, such as neural networks and fuzzy systems, to the problems in digital image processing [32]–[54]. Indeed, neurofuzzy (NF) systems offer the ability of neural networks to learn from examples and the capability of fuzzy systems to model the uncertainty which is inevitably encountered in noisy environments. Therefore, neuro-fuzzy systems may be utilized to design line, edge and detail preserving impulse noise removal operators provided that the appropriate network topologies and processing strategies are employed. Hence, another class of impulse noise removal methods comprises nonlinear operators based on soft computing techniques. One important group of these methods is the fuzzy filters [32, 37]. These filters employ a set of fuzzy rules for the detection of impulse noise and an appropriate inference mechanism for its removal. Although they exhibit considerably better performance than the median-based filters, they are inherently heuristic and the determination of the fuzzy rule base may be quite complicated especially for high noise densities. In order to overcome these difficulties, NF methods allowing the determination of the internal parameters of the fuzzy filter from a set of training data have also been proposed [38, 39, 47, 52]. However, their network structures are usually very complicated, their implementation and training are more complex and the required filtering window size is usually larger than the other methods. Furthermore, the use of two dimensional fuzzy membership functions and multi-output network structures even more increase the complexity of these systems. In addition to the above applications, fuzzy systems have also been adopted to evaluate the components of more traditional filtering methods and to generalize existing families of nonlinear filters such as median operators. Many different variants of these approaches have been proposed for the re-

214

E. Y¨ uksel

moval of different kinds and mixtures of image noise including impulse noise [13, 34, 35, 41, 49].

3 The neuro-fuzzy operator In this section, we will discuss the details of the internal structure and training of the NF operator. Figure 1a shows the general structure of the NF operator. The operator is constructed by combining a desired number of NF blocks with a postprocessor. All NF blocks in the structure operate on the same 3-by-3 pixel filtering window, which is shown in Figure 1b. Each NF block acts as a subblock evaluating a neighborhood relation between the center pixel of the filtering window and two of its neighbors. Some of the possible neighborhood topologies are shown in Figure 1c. Each NF block evaluates a different neighborhood relation. The higher the number of NF blocks, the better the filtering performance, but the higher the computational cost. 3.1 NF blocks Each NF block is a first order Sugeno type fuzzy inference system [55, 56] with 3-inputs and 1-output. The internal structures of the NF blocks are identical to each other. Each input has 3 generalized bell type membership functions and the output has a linear membership function. The input-output relationship of any of the NF blocks is as follows: Let X1 , X2 , X3 denote the inputs of the NF block and Y denote its output. Each possible combination of inputs and their associated membership functions is represented by a rule in the rule base of the NF block. Since the NF block has 3 inputs and each input has 3 membership functions, the rule base contains a total of 27 (33 ) rules, which are as follows: 1. 2. 3. 4. 5.

if if if if if .. . 27. if

(X1 (X1 (X1 (X1 (X1

is is is is is

M11 ) M11 ) M11 ) M11 ) M11 )

and and and and and

(X2 (X2 (X2 (X2 (X2

is is is is is

M21 ) M21 ) M21 ) M22 ) M22 )

and and and and and

(X3 (X3 (X3 (X3 (X3

is is is is is

M31 ), M32 ), M33 ), M31 ), M32 ),

then then then then then

R1 R2 R3 R4 R5

= F1 (X1 , X2 , X3 ) = F2 (X1 , X2 , X3 ) = F3 (X1 , X2 , X3 ) = F4 (X1 , X2 , X3 ) = F5 (X1 , X2 , X3 )

(X1 is M13 ) and (X2 is M23 ) and (X3 is M33 ), then R27 = F27 (X1 , X2 , X3 )

where Mij denotes the jth membership function of the ith input, Rk denotes the output of the kth rule, and Fk denotes the kth output membership function. Unfortunately, there is no analytical way to determine the type of the membership function that gives the best results. In most cases, the type of the membership function is chosen heuristically and verified experimentally. In all simulation results presented in this chapter, it is experimentally determined that the best performances are obtained when the input membership


215

Input Image

Postprocessor

NF Block NF Block

. . .

Output Image

NF Block

a

x(m-1,n-1)

x(m-1,n)

x(m-1,n+1)

x(m,n-1)

x(m,n)

x(m,n+1)

x(m+1,n-1)

x(m+1,n)

x(m+1,n+1)

b

c Fig. 1. (a) The general structure of the neuro-fuzzy operator. The pixels applied to the inputs of each NF block in the structure are chosen so as to utilize the information from a different pixel neighborhood. (b) The filtering window of the operator. (c) Some of the possible pixel neighborhood topologies (Adapted and reprinted from [52] with permission from Elsevier)

functions are chosen to be the generalized bell type. This may be attributed to the fact that the generalized bell type membership function can approximate other types of membership functions to some degree if its parameter values are appropriately optimized. This means that a NF system having generalized bell type input membership functions can adjust the shape of these membership functions to approximate them to other types of membership functions during training to yield the best modeling performance. For this reason, it is observed that NF systems having generalized bell type input membership functions yield in general better modeling performance than the systems having other types of input membership functions.

216

E. Y¨ uksel

Hence, the input membership functions are generalized bell type: Mij (u) =

1 u − aij 2cij 1+ bij

(i, j) = 1, 2, 3.

(1)

and the output membership functions are linear: Fk (u1 , u2 , u3 ) = dk1 u1 + dk2 u2 + dk3 u3 + dk4

k = 1, · · · , 27.

(2)

Here the parameters a, b, c and d are constants that characterize the shape of the membership functions. The optimal values of these parameters are determined by training, which will be discussed in detail later on. The output of the NF block is the weighted average of the individual rule outputs. The weighting factor, wk , of each rule is calculated by evaluating the membership expressions in the antecedent of the rule. This is accomplished by first converting the input values to fuzzy membership values by utilizing the input membership functions and then applying the and operator to these membership values. The and operator corresponds to the multiplication of input membership values. Hence, the weighting factors of the rules are calculated as follows: w1 = M11 (X1 ) . M21 (X2 ) . M31 (X3 ) w2 = M11 (X1 ) . M21 (X2 ) . M32 (X3 )

(3)

w3 = M11 (X1 ) . M21 (X2 ) . M33 (X3 ) w4 = M11 (X1 ) . M22 (X2 ) . M31 (X3 ) w5 = M11 (X1 ) . M22 (X2 ) . M32 (X3 ) .. . w27 = M13 (X1 ) . M23 (X2 ) . M33 (X3 ) Once the weighting factors are obtained, the output of the NF block can be found by calculating the weighted average of the individual rule outputs: 27

Y =

wk Rk

k=1 27

(4) wk

k=1

Readers interested in the details of fuzzy systems may refer to an excellent book on this subject [56]. 3.2 The postprocessor The outputs of the NF blocks are fed to a postprocessor, which generates the final operator output. The postprocessor actually calculates the average value


217

of the NF block outputs and then suitably truncates this value to an 8-bit integer number. The input-output relationship of the postprocessor may be explained as follows: Let Y1 , Y2 , · · · , YK represent the outputs of the NF blocks in the structure of the NF operator, respectively, where K is the number of NF blocks used. The output of the postprocessor is calculated in two steps. In the first step, the average value of the individual NF block outputs are calculated: YAV =

K 1 Yk K

(5)

k=1

In the second step, this value is suitably truncated to an 8-bit integer value so that the luminance value obtained at the output of the postprocessor ranges between 0 and 255:  if YAV < 0 0 if YAV > 255 y(m, n) = 255 (6)  round(YAV ) otherwise where y(m, n) is the output of the postprocessor, which is also the output of the NF operator. 3.3 Training of the NF blocks The internal parameters of the NF operator are optimized by training. Each NF block in the structure is trained individually. Figure 2 represents the setup used for training. Here, the parameters of the NF block under training are iteratively adjusted so that its output converges to the output of the ideal block. The ideal block represents an ideal noise filter, an ideal noise detector, or an ideal edge detector depending on the target application of the NF operator. For example, it represents an ideal noise filter if the NF operator is to be used as a noise filter. It should be noted that the ideal block is conceptual only and does not necessarily exist in reality. It is only the output of the ideal block that is necessary for training. The output of the ideal block is represented by a suitably

Ideal Block Input Training Image

Target Training Image

NF Block

Fig. 2. Training of the neuro-fuzzy blocks (Adapted from [52] with permission from Elsevier)

218

E. Y¨ uksel

chosen target training image, which will again vary depending on the application. The choice of training images will separately be discussed in detail for each of the three applications of the NF operator. 3.4 Adaptation of the parameters of the membership functions The adaptation of the parameters of the input and output membership functions of the NF blocks is achieved by utilizing the Levenberg-Marquardt optimization algorithm [56]–[58]. The Levenberg-Marquardt algorithm is a powerful variant of the gradient-based Gauss-Newton algorithm and is widely used in nonlinear least-squares modeling problems. The algorithm is applied in the training of the NF blocks as follows: A given NF block used in the structure of the NF operator is actually a universal model that establishes a nonlinear relationship between its inputs and the output: Y = f (X; θ)

(7)

Here, Y is the output, X is the vector of the inputs and θ is the vector of the internal parameters of the NF block: X = [X1 X2 X3 ]T

(8)

θ = [aT bT cT dT ]T

(9)

and

where X1 , X2 and X3 are the inputs of the NF block, the vectors a, b and c denote the parameters of the input membership functions, the vector d denotes the parameters of the output membership functions and the superscript “T ” denotes matrix transposition. From the definitions of the input and the output membership functions given in equations (1) and (2), the parameter vectors a, b, c and d are defined as follows: aT = {aij | i = 1, 2, 3; j = 1, 2, 3} bT = {bij | i = 1, 2, 3; j = 1, 2, 3}

(10)

cT = {cij | i = 1, 2, 3; j = 1, 2, 3} dT = {dkl | k = 1, · · · , 27; l = 1, 2, 3, 4} or in full form aT = [a11 a12 a13 a21 a22 a23 a31 a32 a33 ] bT = [b11 b12 b13 b21 b22 b23 b31 b32 b33 ] cT = [c11 c12 c13 c21 c22 c23 c31 c32 c33 ] dT = [d11 d12 d13 d14 d21 d22 d23 d24 d31 · · · d27,4 ]

(11)


219

Given a set of training data pairs, {Xt ; Dt }, with t = 1, 2, · · · , NT , where NT is the number of pairs in the set, the sum of squared errors may be obtained as follows: E(θ) =

NT

(Dt − Yt )2

(12)

t=1

=

NT

(Dt − f (Xt ; θ))2

t=1

=

NT

e2t (θ)

t=1

where Xt is the tth input training data vector, Dt is the desired output of the NF block, Yt is the actual output of the NF block and et (θ) is the output error for the input vector Xt . The function E(θ) can also be expressed in vector form as follows: E(θ) = eT (θ) e(θ)

(13)

where e(θ) is the vector of errors defined as: e(θ) = [e1 (θ) e2 (θ) · · · eNT (θ)]T

(14)

The training of the NF block corresponds to the minimization of the function E(θ) with respect to the parameter vector θ. The Levenberg-Marquardt algorithm accomplishes this goal by performing the following iteration: θnext = θnow − (JT J + λI)−1 JT e(θnow )

(15)

where J is the Jacobian matrix of e(θnow ), I is the identity matrix and λ is a nonnegative value which is usually set to 0.01. The training iteration is usually terminated when the relative change in the parameter vector θ decreases below a predetermined threshold value. Once the training of the NF system is completed, its internal parameters are freezed and it is included in the structure of the NF operator. 3.5 Processing of the input image The overall filtering procedure for processing the input image may be summarized as follows: 1. A 3-by-3 pixel filtering window is slided on the image. The window is started from the upper-left corner of the image and moved sideways and progressively downwards in a raster scanning fashion.

220

E. Y¨ uksel

2. For each window, the appropriate pixels of the filtering window representing different neighborhood topologies are fed to the corresponding NF blocks in the structure. Each NF block individually generates an output for the center pixel of the filtering window by processing the luminance values of its input pixels. 3. The outputs of the NF blocks are fed to the postprocessor, which calculates the average value of the NF block outputs and suitably truncates this value to an 8-bit integer number. The value obtained at the output of the postprocessor is also the output value of the NF operator and represents the calculated value for the center pixel of the filtering window. 4. This procedure is repeated for all pixels of the input image.

4 Applications In this section, we will demonstrate three different applications of the NF operator discussed in the previous section. These are noise filtering, noise detection and edge detection. The same general NF operator shown in Figure 1a will be used in all of these applications. However, the training procedure will make the difference and a different pair of training images will be used for each of the three applications. 4.1 The NF operator as a noise filter In the first application, the NF operator will be used as a noise filter. The setup used for training an individual NF block in the structure of the NF operator as a noise filter is shown in Figure 3. Here, the parameters of the NF block under training are iteratively adjusted so that its output converges to the output of the ideal noise filter which, by definition, can completely remove the noise from the image. It should be remembered that the ideal noise filter is conceptual only and does not necessarily exist in reality. It is only the output of the ideal noise filter that is necessary for training, and this is represented by the target training image. Figure 4 shows the training images used for noise filtering application. The training image shown in Figure 4a is a 64-by-64 pixel artificial image that can easily be generated in a computer. Each square box in this image has a size of 4-by-4 pixels and the 16 pixels contained within each box have the same luminance value, which is an 8-bit integer number uniformly distributed between 0 and 255. The image in Figure 4b is obtained by corrupting the image in Figure 4a by impulse noise of 35% noise density. Although the density of the corrupting noise is not very critical regarding training performance, simulation results show that very low or very high values should be avoided. It is experimentally observed that very good filtering performance is easily obtained for all kinds of images corrupted by an impulse

NF Methods for Noise Filtering, Noise Detection and Edge Extraction Ideal Noise Filter Input Training Image

221


NF Block

Fig. 3. Training of an individual NF block of the NF operator as a noise filter (Reprinted from [52] with permission from Elsevier)

a

b

Fig. 4. Training images: (a) Original (Target training image in Fig. 3), (b) Noisy (Input training image in Fig. 3) (Reprinted from [52] with permission from Elsevier)

noise with a noise probability between 0% and 80% provided that the training image used for the training of the NF filters has a noise probability value between 20% and 50%. The images in Figure 4a and Figure 4b are employed as the target (desired) and the input images during training, respectively. The parameters of the NF block under training are then tuned by using the Levenberg-Marquardt optimization algorithm [56]–[58] so as to minimize the learning error. Once the training of the NF blocks is completed, the internal parameters of the blocks are fixed, and the blocks are combined with the postprocessor to construct the NF operator, as shown in Figure 1a. Figure 5 shows two sample implementations of the presented NF noise filtering operator using two and four NF blocks (filters), respectively. In this figure, x(m, n) and y(m, n) represent the pixels of the noisy input and the restored output images, respectively. The input data to the operator are the luminance values of the appropriate pixels of the 3-by-3 filtering window (Figure 1b) from the noisy input image. In the four block implementation shown in Figure 5b, each of the four NF filters evaluate neighborhood relations between the center pixel of the filtering window and its two neighbors in horizontal, vertical, left diagonal and right diagonal directions, respectively. In the two block implementation, however, only horizontal and vertical neighborhood relations are considered. The performance of the NF operator trained as a noise filter is tested on the popular Baboon image. The image size is 256-by-256 pixels and the pixel values range between 0 and 255. The test image used in the experiments is

E. Y¨ uksel x(m,n-1) x(m,n) x(m,n+1)

Horizontal NF Filter

x(m-1,n) x(m,n) x(m+1,n)

Vertical NF Filter

Postprocessor

222

y(m,n)

x(m,n-1) x(m,n) x(m,n+1)

Horizontal NF Filter

x(m-1,n) x(m,n) x(m+1,n)

Vertical NF Filter

x(m-1,n-1) x(m,n) x(m+1,n+1)

Left Diag. NF Filter

x(m-1,n+1) x(m,n) x(m+1,n-1)

Right Diag. NF Filter

Postprocessor

a

y(m,n)

b Fig. 5. Two sample implementations of the presented NF noise removal operator with two and four NF blocks (filters) respectively. The four filter implementation evaluates pixel neighborhoods in horizontal, vertical, left diagonal and right diagonal directions, while the two filter implementation evaluates horizontal and vertical neighborhoods only (Reprinted from [52] with permission from Elsevier)

obtained by corrupting the original Baboon image by impulse noise with 70% noise density. For comparison, the corrupted test image is also filtered by using several popular noise removal operators including the standard median filter (MF) [1], the signal-dependent rank-ordered mean filter (SDROMF) [18], the adaptive center-weighted median filter (ACWMF) [6] and the fuzzy filter (FF) [32]. All operators including the NF operator operate on the 3-by-3 filtering window shown in Figure 1b. Figure 6 shows the output images of all operators for the Baboon image corrupted by impulse noise of 70% noise density for a visual evaluation of the noise removal and detail preservation performances of the operators. It is observed from this figure that the noise removal performance of the MF operator is very poor compared with the other operators. The outputs of the SDROMF, ACWMF and FF operators are almost the same, but these operators introduce significant blurring effects into the image and destroy the useful information during noise removal process. The two filter implementation of the presented NF operator exhibits much better performance than the others. On the other hand, the four filter implementation of the NF operator offers the best performance of all. It is clearly seen that the NF operator


a

b

c

d

e

f

g

h

223

Fig. 6. Comparison of the output images of the operators for 70% noise density. (a) Original Baboon image (b) Noisy Baboon image, (c) MF, (d) SDROMF, (e) ACWMF, (f) FF, (g) NF operator with two blocks, (h) NF operator with four blocks (Reprinted from [52] with permission from Elsevier)

successfully preserves the details in the image while at the same time efficiently removing the noise. The difference in the detail preservation performance can easily be observed by carefully looking at the appearance of the eyes and the hair around the mouth of the animal in the output images of all operators. Readers interested in additional filtering experiments and comparative discussion of the results of these experiments are referred to reference [52].

224

E. Y¨ uksel

4.2 The NF operator as a noise detector Conventional filters usually distort the uncorrupted regions of the noisy input image during restoration of the corrupted regions, causing undesirable blurring effects in the image. The most straightforward solution to avoid this problem is to combine the noise filter with a noise detector as shown in Figure 7. In this approach, the noise detector attempts to determine whether the center pixel of a given filtering window is a noisy pixel or not. If the center pixel is identified by the noise detector as a noisy pixel, then the output of the system is switched to the output of the noise filter, which generates the restored value for the corrupted pixel at its output. If the center pixel is identified as uncorrupted, which means that there is no need to restore this pixel, the noise filter is bypassed and the output of the system is switched directly to the input. This approach has been used to significantly improve the performance of conventional median filtering, and several median based filters exploiting different impulse detection mechanisms have been proposed. 0

Noisy Image

Noise Filter

1

Restored Image

Noise Detector

Fig. 7. Improvement of the performance of a noise filter by using a noise detector (Reprinted from [12] with permission from the IEEE)

In this section, we will demonstrate the use of the presented NF operator as a noise detector. We will appropriately train the NF operator as an impulse detector and then use it along several impulse noise filters, as iluustrated in Figure 7, to reduce their undesirable blurring effects. Figure 8 shows the arrangement used for training an individual NF block in the structure of the NF operator as a noise detector. As before, the parameters of the NF block under training are iteratively optimized so that its output converges to the output of the ideal noise detector. The ideal noise detector is again a conceptual operator and truly detects the location of corrupted pixels in a noisy image. It is only the output of the ideal noise detector that is needed for training, and this is represented by the noise-free training image. Figure 9 shows the training images used for noise detection application. The training image shown in Figure 9a is the original training image. The image in Figure 9b is the noisy training image and obtained by corrupting the image in Figure 9a by impulse noise. The image in Figure 9c is the noise detection image. It is a difference image obtained from the base training image

NF Methods for Noise Filtering, Noise Detection and Edge Extraction Ideal Noise Detector Input Training Image

225


NF Block

Fig. 8. Training of an individual NF block of the NF operator as a noise detector (Reprinted from [12] with permission from the IEEE)

a

b

c

Fig. 9. Training images for noise detection application: (a) Original, (b) Noisy (Input training image in Fig. 8), (c) Detection image indicating the locations of the noisy pixels in the noisy training image (Target training image in Fig. 8) (Reprinted from [12] with permission from the IEEE)

and the input training image. Its white pixels indicate the locations of noisy pixels in the noisy training image in Figure 9b. The images in Figure 9c and Figure 9b are employed as the target (desired) and the input images during training, respectively. The parameters of the NF block under training are then tuned by using the Levenberg-Marquardt optimization algorithm [56]–[58] so as to minimize the learning error. Once the training of the NF blocks is completed, the internal parameters of the blocks are fixed, and the blocks are combined with the postprocessor to construct the NF operator, as shown in Figure 1a. A two block implementation of the NF noise detection operator is shown in Figure 10. The input data to the operator are the luminance values of the appropriate pixels of the 3-by-3 filtering window (Figure 1b) from the noisy input image. In this way, one of the NF blocks evaluates the horizontal neighborhood and the other vertical. The NF impulse detector is combined with a given impulse noise removal operator as shown in Figure 7 and improves its performance by efficiently reducing its blurring effects. The overall filtering of the noisy input image is completed in three stages. In the first stage, the noisy input image is processed by the NF impulse detector and a noise detection image is generated. The noise detection image is a black and white image, similar to the target training image shown in Figure 9c, and locations of its white pixels correspond to the locations of the corrupted pixels in the noisy input image. In the second stage, the

E. Y¨ uksel x(m,n-1) x(m,n) x(m,n+1)

Horizontal NF Detector

x(m-1,n) x(m,n) x(m+1,n)

Vertical NF Detector

Postprocessor

226

y(m,n)

Fig. 10. A sample implementation of the presented NF noise detection operator with two NF blocks (noise detectors). The two detector blocks evaluate pixel neighborhoods in horizontal and vertical directions, respectively (Reprinted from [12] with permission from the IEEE)

noisy input image is processed by using the noise filter and a restored output image is obtained. In the third stage, the further enhanced, blur-reduced final output image is constructed by appropriately mixing the pixels of the noisy input image and the output image of the noise filter. Those pixels of the final output image that correspond to the white pixels in the noise detection image are copied from the output image of the noise removal operator, and the remaining pixels are copied directly from the noisy input image. In order to demonstrate the performance improvement obtained by using the presented NF noise detection operator, the NF operator is combined with the standard median filter in the setup illustrated in Figure 7 to obtain a switching median filter . This switching median filter is then used to filter the Baboon and the Boats images corrupted by 25% impulse noise. Figure 11 shows the output images obtained in this experiment. The images shown in this figure are obtained by zooming in the same 128-by-128 pixel portion of the output images for better visual evaluation. In each pair, the image on the left is the direct output image of the median filter while the image on the right is the enhanced output image obtained by using the median filter together with the NF impulse detector as indicated in Figure 7. The undesirable blurring effects and the restoration of these distortions by the NF detector can clearly be observed by carefully looking at the appearance of the hair around the mouth of the animal and also observing the appearance of the small details, such as ropes, of the boat image. Readers interested in additional filtering experiments and comparative discussion of the results of these experiments are referred to reference [12]. 4.3 The NF operator as an edge detector Edges are defined to be sudden changes in the local color intensity of an image and provide important information about the objects contained within the image. Edge detection is usually the first operation that is performed before other image processing tasks such as boundary detection, object recognition, image registration, image classification, and so on. Therefore the success of


a

b

c

d

227

Fig. 11. Output images of the standard median filter for the uses without and with the NF noise detector. In each pair, the image on the left is the direct output image of the median filter, and the image on the right is the enhanced output image obtained by using the median filter together with the NF impulse detector as illustrated in Figure 7 (Reprinted from [12] with permission from the IEEE)

these subsequent image processing tasks are strictly dependent on the performance of edge detection. The most important factor decreasing the performance of edge detection is the noise. As discussed before, digital images are usually degraded by noise due to a number of imperfections in the imaging process. Most edge detection methods require the removal of noise by using an appropriate noise filter prior to edge detection. In this case, however, the complexity of the system and the processing time are considerably increased. Furthermore, the edge detection performance strictly depends on the performance of the noise filter. Different methods implementing different approaches to edge detection are available in the literature. The classical methods [1] such as the Sobel, Prewitt and Kirsch detectors use the first directional derivative to determine edges. These detectors are simple to implement but they are usually inaccurate and highly sensitive to noise. The zero-crossing edge detectors [1] use the second derivative along with the Laplacian operator. They have fixed detection characteristics in all directions but they are also very sensitive to noise. The Gaussian edge detectors, such as the Canny detector, reduce the influence of noise by smoothing the image before detecting edges. The Canny detector, which is one of the most popular edge detectors in the literature, has been

228

E. Y¨ uksel

widely used in many applications [59]-[61] because of its superior performance especially in noisy conditions. Although the Gaussian detectors exhibit relatively better performance, they are computationally much more complex. Therefore, a simple edge detector that is capable of extracting edges from digital images corrupted by noise is highly desirable. In this section, we will train the NF operator as an edge detector and show that it successfully extracts edges from noisy digital images without needing a prefiltering of the image. Figure 12 shows the setup used for training an individual NF block in the structure of the NF operator as an edge detector. In this case, the parameters of the NF block under training are iteratively adjusted so that its output converges to the output of the ideal edge detector which, by definition, can truly extract edges from a noisy image independent of the density of the noise. Again, the ideal edge detector is conceptual only and does not exist in reality. Its output is represented by the target training image. Ideal Edge Detector Input Training Image


NF Block

Fig. 12. Training of an individual NF block of the NF operator as an edge detector (Reprinted from [48] with permission from Elsevier)

a

b

c

Fig. 13. Training images: (a) Original, (b) Noisy (Input training image in Fig. 12, (c) True edges (Target training image in Fig. 12) (Reprinted from [48] with permission from Elsevier)

Figure 13 shows the images used for training. The image shown in Figure 13a is the original training image. The image in Figure 13b is the noisy training image and obtained by corrupting the original training image by impulse noise. The image in Figure 13c is the edge image. It is a black and white image and its black pixels indicate the true edges of the original training image. Hence it


229

x(m,n-1) x(m,n) x(m,n+1)

Horizontal NF Subdetector

x(m-1,n) x(m,n) x(m+1,n)

Vertical NF Subdetector

x(m-1,n-1) x(m,n) x(m+1,n+1)

Left Diag. NF Subdetector

x(m-1,n+1) x(m,n) x(m+1,n-1)

Right Diag. NF Subdetector

Postprocessor

represents the output of the ideal edge detector for the noisy training image in Figure 13b. The images in Figure 13c and 13b are employed as the target (desired) and the input images during training, respectively. The parameters of the NF blocks are then tuned so as to minimize the learning error by using the Levenberg-Marquardt optimization algorithm [56]–[58]. Figure 14 shows a sample implementation of the presented NF edge detection operator using four NF blocks. The input data to the operator are the luminance values of the appropriate pixels of the 3-by-3 filtering window (Figure 1b) from the noisy input image. Therefore, each of the four NF subdetectors evaluate neighborhood relations between the center pixel of the filtering window and its two neighbors in horizontal, vertical, left diagonal and right diagonal directions, respectively.

y(m,n)

Fig. 14. A sample implementation of the presented NF edge detection operator with four NF blocks (edge subdetectors). The four subdetector blocks evaluate pixel neighborhoods in horizontal, vertical, left diagonal and right diagonal directions, respectively (Reprinted from [48] with permission from Elsevier)

The presented NF edge detector is implemented and applied to extract the edges of the Boats image corrupted by 20% impulse noise. In order to make comparisons, the same image is also processed by using the Sobel and the Canny edge detectors. Figure 15 shows the output images of all detectors. It is observed from this figure that the performance of the Sobel detector is very poor. Its output image is severely degraded by noise. Most noise pulses are incorrectly detected as edges. Furthermore, only the edges with sharp intensity variation are partially reflected to the output image while the edges with less intensity variation are completely lost in the noise. The Canny detector has a considerably better performance than the Sobel detector. It correctly detects most of the noise pulses and these are not reflected to the output image as edges. However, the effect of noise is still

230

E. Y¨ uksel

a

b

c

d

e

Fig. 15. Comparison of the output images of the edge detection operators for 20% noise density. (a) Original Boats image (b) Noisy Boats image, (c) Sobel edge detector, (d) Canny edge detector, (e) NF edge detector with four blocks (Reprinted from [48] with permission from Elsevier)

clearly visible as real edges are significantly distorted by the noise. Due to the Gaussian smoothing of the detector, the edges in the more detailed regions of the input image are not properly detected. In addition, the edges where the luminance variation is not very sharp are almost lost in the noise. On the other hand, the presented NF edge detector exhibits very good detection performance and successfully detects most of the edges. The effect of noise on the performance of the detector is much less compared to the Sobel and the Canny detectors. Object boundaries and other details in the image are reflected to the output image of the NF edge detector much better than the other detectors. The edges in the more detailed regions of the input image are successfully extracted. In addition, the NF detector shows a significantly better performance at detecting the edges where the luminance variation is not very sharp. Readers interested in additional filtering experiments and comparative discussion of the results of these experiments are referred to reference [48].


231

5 Conclusions and remarks In this chapter, we presented a general NF operator that can be customized for a number of applications in image processing. We demonstrated that this general operator structure can be used for different purposes by suitably choosing the training images. Specifically, we presented three applications of the NF operator: (1) noise filter, (2) noise detector, and (3) edge detector. The fundamental advantages of the presented NF operator may be summarized as follows: 1. It has a flexible structure. The number of NF blocks in the structure can be varied to obtain the desired performance. The higher the number NF blocks included in the structure of the operator, the better the performance, but the higher the computational complexity. 2. It has a very simple structure. Its fundamental building block is a 3-input 1-output NF block. This greatly simplifies implementation. 3. Its performance does not depend on any parameters that are heuristically determined and externally supplied by the user. 4. It is trained by using very simple artificial images, which can easily be generated in a computer. 5. It uses a minimal 3-by-3 filtering window. It should also be pointed out that potential uses of the general purpose NF operator structure discussed here are not limited to the three applications presented in this chapter. It is straightforward to see that the presented NF operator may be used for a number of other applications in image processing provided that appropriate network topologies and training strategies are employed. Two different applications related with image restoration are already available in the literature [53, 54]. Other potential applications are left to the reader.

6 Acknowledgement Part of the material (text, equations, figures, etc.) presented in this chapter has been borrowed from previously published works of the author [12, 48, 52, 53, 54] and reprinted here with permission from the copyright holders of the original material. Specifically, the articles [12, 54] are copyrighted by the IEEE while the articles [48, 52, 53] are copyrighted by Elsevier. The author wishes to thank the IEEE and Elsevier for their kind permission to reproduce their copyrighted material. Part of the results presented in this chapter have been obtained during a research project funded by Erciyes University Scientific and Technological Research Center (Project code: 02-012-4). The author also wishes to thank Erciyes University Scientific and Technological Research Center for its financial support to this work.

232

E. Y¨ uksel

References 1. Umbaugh S.E. (1998) Computer Vision and Image Processing. Prentice-Hall International Inc. 2. Yli-Harja O., Astola J., Neuvo Y. (1991) Analysis of the properties of median and weighted median filters using threshold logic and stack filter representation. IEEE Trans. on Signal Processing 39:395–410 3. Ko S.J., Lee Y.H. (1991) Center weighted median filters and their applications to image enhancement. IEEE Trans. on Circuit and Systems 38:984–993 4. Jeong B., Lee Y.H. (1994) Design of weighted order statistic filters using the perceptron algorithm. IEEE Trans. on Signal Processing 42:3264–3269 5. Sun T., Neuvo Y. (1994) Detail-preserving median based filters in image processing. Pattern Recognition Letters 15:341–347 6. Chen T., Wu H.R. (2001) Adaptive impulse detection using center-weighted median filters. IEEE Signal Proc. Letters 8:1–3 7. Zhang S., Karim M.A. (2002) A new impulse detector for switching median filters. IEEE Signal Proc. Letters 9:360–363 8. Pok G., Liu Y., Nair A.S. (2003) Selective removal of impulse noise based on homogeneity level information. IEEE Trans. on Image Processing 12:85–92 9. Crnojevic V., Senk V., Trpovski Z. (2004) Advanced impulse detection based on pixel-wise MAD. IEEE Signal Proc. Letters 11:589–592 10. Be¸sdok E. (2004) Impulsive noise suppression from images with a modified twostep iterative-median filter. J. Electronic Imaging 13:714–719 11. Be¸sdok E., Y¨ uksel M.E. (2005) Impulsive noise rejection from images with ¨ 59:105– Jarque-Berra test based median filter. Int. J. Electron. Commun. (AEU) 110 12. Y¨ uksel M.E., Be¸sdok E. (2004) A simple neuro-fuzzy impulse detector for efficient blur reduction of impulse noise removal operators for digital images. IEEE Trans. on Fuzzy Systems 12:854–865 13. Y¨ uksel M.E., Ba¸st¨ urk A., Be¸sdok E. (2004) Detail preserving restoration of impulse noise corrupted images by a switching median filter guided by a simple neuro-fuzzy network. EURASIP J. Applied Signal Processing 2004:2451–2461 14. Chen T., Ma K.K., Chen L.H. (1999) Tri-state median filter for image denoising. IEEE Trans. on Image Processing 8:1834–1838 15. Chen T., Wu H.R. (2001) Space variant median filters for the restoration of impulse noise corrupted images. IEEE Trans. on Circuit and Systems-II 48:784– 789 16. Wang Z., Zhang D. (1999) Progressive switching median filter for the removal of impulse noise from highly corrupted images. IEEE Trans. on Circuit and Systems 46:78–80 17. Chen T., Wu H.R. (2001) Application of partition-based median type filters for suppressing noise in images. IEEE Trans. on Image Processing 10:829–836 18. Abreu E., Lightstone M., Mitra S.K., Arakawa K. (1996) A new efficient approach for the removal of impulse noise from highly corrupted images. IEEE Trans. on Image Processing 5:1012–1025 19. Moore M.S., Gabbouj M., Mitra S.K. (1999) Vector SD-ROM filter for removal of impulse noise from color images. In: Proceedings of ECMCS’99 EURASIP Conf. on DSP for Multimedia Communications and Services. Krakow


233

20. Han W.Y., Lin J.C. (1997) Minimum-maximum exclusive mean (MMEM) filter to remove impulse noise from highly corrupted images. Electronics Letters 33:124–125 21. Windyga P.S. (2001) Fast impulse noise removal. IEEE Trans. on Image Processing 10:173–179 22. Alajlan N., Kamel M., Jernigan E. (2004) Detail preserving impulsive noise removal. Signal Processing: Image Communication 19:993–1003 23. Xu X., Miller E.L., Chen D., Sarhadi M. (2004) Adaptive two-pass rank order filter to remove impulse noise in highly corrupted images. IEEE Trans. on Image Processing 13:238–247 24. Nikolova M. (2004) A variational approach to remove outliers and impulse noise. J. Math. Imaging Vis. 20:99-120 25. Chan R.H., Hu C., Nikolova M. (2004) An iterative procedure for removing random-valued impulse noise. IEEE Signal Proc. Letters 11:921–924 26. Russo F. (2004) Impulse noise cancellation in image data using a two-output nonlinear filter. Measurement 36:205–213 27. Aizenberg I., Butakoff C., Paliy D. (2005) Impulsive noise removal using threshold boolean filtering based on the impulse detecting functions. IEEE Signal Proc. Letters 12:63–66 28. Beaurepaire L., Chehdi K., Vozel B. (1997) Identification of the nature of noise and estimation of its statistical parameters by analysis of local histograms. In: Proc. ICASSP-1997 4:2805–2808 29. Wang J.H., Lin L.D. (1998) Image restoration using parametric adaptive fuzzy filter. In: Proc. NAFIPS-1998 1:198–202 30. Wang J.H., Chiu H.C. (1999) HAF: An adaptive fuzzy filter for restoring highly corrupted images by histogram estimation. In: Proc. Natl. Sci. Counc. 23:630– 643 31. Ma Z., Wu H.R. (2003) A histogram based adaptive vector filter for color image restoration. In: Proc. 4th Pacific Rim Conf. on Multimedia 1:81–85 32. Russo F., Ramponi G. (1996) A fuzzy filter for images corrupted by impulse noise. IEEE Signal Proc. Letters 3:168–170 33. Choi Y.S., Krishnapuram R. (1997) A robust approach to image enhancement based on fuzzy logic. IEEE Trans. on Image Processing 6:808–825 34. Lee C.S., Kuo Y.H., Yu P.T. (1997) Weighted fuzzy mean filter for image processing. Fuzzy Sets and Systems 89:157–180 35. Lee C.S., Kuo Y.H. (1999) The important properties and applications of the adaptive weighted fuzzy mean filter. Int. J. Intelligent Systems 14:253–274 36. Pham D.L., Prince J.L. (1999) Adaptive fuzzy segmentation of magnetic resonance images. IEEE Trans. on Medical Imaging 18:737–752 37. Russo F. (1999) FIRE operators for image processing. Fuzzy Sets and Systems 103:265–275 38. Russo F. (1999) Hybrid neuro-fuzzy filter for impulse noise removal. Pattern Recognition 49:307–314 39. Russo F. (2000) Noise removal from image data using recursive neurofuzzy filters. IEEE Trans. on Instrumentation and Measurement 49:307–314 40. Lin C.T., Lee Y.C., Pu H.C. (2000) Satellite sensor image classification using cascaded architecture of neural fuzzy network. IEEE Trans. on Geoscience and Remote Sensing 38:1033–1043 41. Eng H.L., Ma K.K. (2001) Noise adaptive soft-switching median filter. IEEE Trans. on Image Processing 10:242–251

234

E. Y¨ uksel

42. Park D.C., Woo Y.J. (2001) Weighted centroid neural network for edge preserving image compression. IEEE Trans. on Neural Networks 12:1134–1146 43. Boskowitz V., Guterman H. (2002) An adaptive neuro-fuzzy system for automatic image segmentation and edge detection. IEEE Trans. on Fuzzy Systems 10:247–262 44. Senel H.G., Peters R.A., Dawant B. (2002) Topological median filters. IEEE Trans. on Image Processing 11:89–104 45. Lee S.J., Ouyang C.S., Du S.H. (2003) A neuro-fuzzy approach for segmentation of human objects in image sequences. IEEE Trans. on Systems, Man and Cybernetics (Part B) 33:420–437 46. Van De Ville D., Nachtegael M., Van der Weken D., Kerre E.E., Philips W, Lemahieu I (2003) Noise reduction by fuzzy image filtering. IEEE Trans. on Fuzzy Systems 11:429–436 47. Y¨ uksel M.E., Ba¸st¨ urk A. (2003) Efficient removal of impulse noise from highly corrupted digital images by a simple neuro-fuzzy operator. Int. J. Electron. ¨ 57:214–219 Commun. (AEU) 48. Y¨ uksel M.E., Yıldırım M.T. (2003) A simple neuro-fuzzy edge detector for digi¨ 58:72– tal images corrupted by impulse noise. Int. J. Electron. Commun. (AEU) 75 49. Xu H., Zhu G., Peng H., Wang D. (2004) Adaptive fuzzy switching filter for images corrupted by impulse noise. Pattern Recognition Letters 25:1657–1663 50. Be¸sdok E., C ¸ ivicio˘ glu P., Al¸cı M. (2004) Impulsive noise suppression from highly corrupted images by using resilient neural networks. Lecture Notes in Artificial Intelligence 3070:670–675 51. Be¸sdok E., C ¸ ivicio˘ glu P., Al¸cı M. (2005) Using an adaptive neuro-fuzzy inference system-based interpolant for impulsive noise suppression from highly distorted images. Fuzzy Sets and Systems 150:525–543 52. Y¨ uksel M.E., Ba¸st¨ urk A. (2005) A simple generalized neuro-fuzzy operator for efficient removal of impulse noise from highly corrupted digital images. Int. J. ¨ 59:1–7 Electron. Commun. (AEU) 53. Y¨ uksel M.E. (2005) A simple neuro-fuzzy method for improving the performances of impulse noise filters for digital images. Int. J. Electron. Commun. ¨ 59:463–472 (AEU) 54. Y¨ uksel M.E. (2006) A hybrid neuro-fuzzy filter for edge preserving restoration of images corrupted by impulse noise. IEEE Trans. on Image Processing 15:928– 936 55. Jang J.S.R. (1993) ANFIS: Adaptive network based fuzzy inference systems. IEEE Trans. on Systems, Man, and Cybernetics. 23:665–685 56. Jang J.S.R., Sun C.T., Mizutani E. (1997) Neuro-fuzzy and soft computing. Prentice-Hall International Inc. 57. Levenberg K. (1944) A method for the solution of certain problems in least squares. Quan. Appl. Math. 2:164–168 58. Marquardt D.W. (1963) An algorithm for least squares estimation of nonlinear parameters. J. Soc. Industrial and Applied Mathematics 31:431–441 59. Linares I., Maersereau R.M., Smith M.J.T. (1996) JPEG estimated spectrum adaptive post filter using image-adaptive Q-tables and Canny edge detectors. In: Proc. ISCAS-96 IEEE International Symposium on Circuits and Systems 722–725. Atlanta, Georgia, USA


235

60. Ali M., Clausi D. (2001) Using the Canny edge detector for feature extraction and enhancement of remote sensing images. In: Proc. IGARSS-01 IEEE International Symposium on Geoscience and Remote Sensing 2298-2300. Sydney, Australia 61. Hongjian S., Ward R. (2002) Canny edge based image expansion. In: Proc. ISCAS-2002 IEEE International Symposium on Circuits and Systems, 785–788. Scottsdale, Arizona, USA

Part IV

Other Applications

Chapter 9 Creating Photorealistic Models by Data Fusion with Genetic Algorithms Dmitry Chetverikov, Zsolt Jank´ o, Evgeny Lomonosov, and Aniko Ek´ art Computer and Automation Research Institute Budapest, Kende u.13-17, H-1111 Hungary [email protected]

1 Introduction Building photorealistic 3D models of real-world objects is a fundamental problem in computer vision and computer graphics. Such models require precise geometry as well as detailed texture on the surface. Textures allow one to obtain visual effects that are essential for high-quality rendering. Photorealism is further enhanced by adding surface roughness in form of the so-called 3D texture represented by a bump map. Typical applications of precise photorealistic 3D models are: • Digitalising cultural heritage objects (the Pieta project [8], the Great Buddha project [9], the Digital Michelangelo project [23]). • Surgical simulations in medical imaging. • E-commerce. • Architecture. • Entertainment (movie, computer games). Different techniques exist to reconstruct the object surface and to build photorealistic 3D models. Although the geometry can be measured by various methods of computer vision, for precise measurements laser scanners are usually used. However, most of laser scanners do not provide texture and colour information, or if they do, the data is not accurate enough. (See [30] for a detailed discussion.) We are currently working on several projects related to the automatic fusion and high-level interpretation of 2D and 3D sensor data for building models of real-world objects and scenes. One of our major goals is to create rich and geometrically correct, scalable photorealistic 3D models based on multimodal data obtained by different sensors, such as camera and laser scanner. In this chapter, we present a sophisticated software system that processes and fuses geometric and image data using genetic algorithms and efficient methods of computer vision.

D. Chetverikov et al.: Creating Photorealistic Models by Data Fusion with Genetic Algorithms, StudFuzz 210, 239–263 (2007) c Springer-Verlag Berlin Heidelberg 2007 www.springerlink.com

240

D. Chetverikov et al.

Our primary goal is to create a system that only uses a PC, an affordable laser scanner and a commercial (although high-quality) uncalibrated digital camera. The camera should be used freely and independently from the scanner. No other equipment (special illumination, calibrated setup, etc.) should be used. No specially trained personnel should be needed to operate the system: After training, a computer user with minimal engineering skills should be able to use it. The above mentioned ambitious projects [8, 9, 23] have developed sophisticated technologies for digitising statues and even buildings, but these technologies are extremely expensive and time-consuming due to the size of the objects to be measured. They require specially designed equipment and trained personnel. Creation of a model takes weeks [8] or even months. What we aim at is a much simpler system that would, for instance, allow an arts expert in a museum to create, in 1–2 hours, a scalable photorealistic model of relatively small cultural object (say, up to 50 cm in size) and transmit this model to his/her colleague in another part of the world. Our modelling system receives as input two datasets of diverse origin: a number of partial measurements (3D point sets) of the object surface made by a hand-held laser scanner, and a collection of high quality images of the object acquired independently by a digital camera. The partial surface measurements overlap and cover the entire surface of the object; however, their relative orientations are unknown since they are obtained in different, unregistered coordinate systems. A specially designed genetic algorithm (GA) automatically pre-aligns the surfaces and estimates their overlap. Then a precise and robust iterative algorithm (Trimmed Iterative Closest Point, TrICP [4]) developed in our lab is applied to the roughly aligned surfaces to obtain a precise registration. Finally, a complete geometric model is created by triangulating the integrated point set. The geometric model is precise, but it lacks texture and colour information. The latter is provided by the other dataset, the collection of digital images. The task of precise fusion of the geometric and the visual data is not trivial, since the pictures are taken freely from different viewpoints and with varying zoom. The data fusion problem is formulated as photo-consistency optimisation, which amounts to minimising a cost function with numerous variables which are the internal and the external parameters of the camera. Another dedicated genetic algorithm is used to minimise this cost function. When the image-to-surface registration problem is solved, we still face the problem of seamless blending of multiple textures, that is, images of a surface patch appearing in different views. This problem is solved by a surface flattening algorithm that gives a 2D parametrisation of the model. Using a measure of visibility as weight, we blend the textures providing a seamless and detail-preserving solution. All major components of the described system are original, developed in our laboratory. Below, we present the main algorithms and give examples of photorealistic model building using GA-based registration and fusion of spatial and pictorial data. Since GA-based data fusion plays a key role in our

Creating Photorealistic Models

241

project, we start with a discussion of the previous use of GAs for this purpose. The discussion is based on our paper [22]. 1.1 Previous work on genetic algorithms in data registration Genetic algorithms have already been used for registration of 3D data. In their recent survey on genetic algorithms in computer-aided design [25], Renner and Ek´ art mention a few related studies. In particular, Jacq and Roux [15] applied GAs to registration of medical 3D images. They considered elastic transformation of 3D volumetric bitmaps rather than rigid transformation of 3D point clouds. Cordon and co-authors [6] also addressed the problem of 3D image registration. They used a CHC evolutionary algorithm and compared binary coding to real value coding. Brunnstr¨ om and Stoddart [2] used GAs to match 3D surfaces represented by measured points. Their approach significantly differs from ours as they perform optimisation in the correspondence space rather than the transformation space. A genetic algorithm finds the best matches, then the optimal transformation is calculated from these matches. The cost function used in [2] is computationally demanding: the number of operations needed to evaluate it grows quadratically with the number of points. For this reason, the method is only applicable to a relatively small number of points (up to a few hundreds). Yamany et al. [29] used a genetic algorithm for registration of partially overlapping 2D and 3D data by minimising the mean square error cost function. The method is made suitable for registration of partially overlapping data sets by only considering the points such that pi ∈ G1 ∪ G2 , where G1 and G2 are space bounding sets for the two datasets. Unfortunately, the authors give very few details about their genetic algorithm, focusing on the Grid Closest Point transformation they use to find the nearest neighbour. Salomon et al. [27] apply a so-called differential evolution algorithm to medical 3D image registration. Differential evolution uses real-valued representation and operates directly on the parameter vector to be optimised. Otherwise, only the reproduction step is different from GA. On the other hand, this method requires much more computation than the simpler algorithm we use. In [27], differential evolution is used to register two roughly pre-aligned volumetric images of small size. Robertson and Fisher [26] applied GAs to the registration of range data. They use the mean square error objective function and perform optimisation in the six-parameter space not considering the overlap. Their genetic algorithm uses four different mutation operators and two crossover operators. Each of the operators has a probability of selection associated with it. The algorithm adjusts these probabilities adaptively. The study [26] includes experiments with partially overlapping measured data; however, no method to automatically deal with arbitrary overlap is presented. Silva et al. [28] use GAs for alignment of multiview range images. A novel Surface Interpenetration Measure is introduced that allows one to evaluate

242


registration results more precisely, especially in the case of small registration errors. A predefined residual threshold is used to handle partial overlap. However, no method is given to adjust the threshold automatically. Since the fitness function depends implicitly on the range scanner measurements error and the interpoint distance (quantisation error), it might be necessary to adjust the threshold manually for other datasets. Hill-climbing techniques are used to refine the GA result. A recent study on the use of genetic algorithms for range data registration appeared in [5]. Euclidean parameter space optimisation is considered. To handle partially overlapping data, the median of the residuals is used as error metric. This improves the robustness but renders the method inapplicable to data overlaps below 50%. An advanced operator of dynamic mutation is introduced, which reportedly improves registration error and helps avoid premature convergence. An attempt is made to improve the precision by using dynamic boundaries. After the GA has converged, the search space is reduced and the GA is applied again. However, using genetic algorithms for precise registration does not seem reasonable since faster and more precise iterative methods exist for registration of pre-aligned datasets.

2 Pre-registration of surfaces using a genetic algorithm This section deals with genetic pre-alignment of two arbitrarily oriented datasets, which are partial surface measurements of the object whose model we wish to build. (See figure 1 for an illustration of such measurements.) The task is to quickly obtain a rough pre-alignment suitable for subsequent application of the robust Trimmed Iterative Closest Point algorithm [4] developed in our lab earlier. Our experience with TrICP shows that, depending on the data processed, the algorithm can cope with initial angular misalignments up to 20◦ ; 5◦ is certainly sufficient. This means that the genetic pre-registration should provide an angular accuracy of 5◦ , or better.

Frog data

GA

GA+TrICP

Fig. 1. The Frog dataset, GA alignment and final alignment


243 N

Consider two partially overlapping 3D point sets, the data set P = {pi }1 p m and the model set M = {mi }N 1 . Denote the overlap by ξ. Then the number of points in P that have a corresponding point in M is Npo = ξNp . The standard ICP [1] assumes that P is a subset of M. ICP iteratively moves P onto M while pairing each point of P with the closest point of M. The cost function of ICP is the mean square error (MSE), that is, the mean of all residuals (distances between paired points). In contrast to ICP, our TrICP [4] only assumes a partial overlap of the two sets, which is more realistic. TrICP finds the Euclidean motion that brings an Npo -point subset of P into the best possible alignment with M. The algorithm uses another cost function. At each iteration, Npo points with the least residuals are selected, and the optimal motion is calculated for this subset so as to minimise the trimmed MSE Npo 1 2 e= d , Npo i=1 i:Np

(1)

N

where {d2i:Np }1 p are the sorted residuals. The subset of the Npo paired points is iteratively updated after each motion. In practice, the overlap ξ is usually unknown. It can be set automatically [4] by running TrICP for different values of ξ and finding the minimum of the objective function Ψ (ξ, R, t) =

e(ξ, R, t) , ξ2

(2)

which minimises the trimmed MSE while trying to use as many points as possible. When an object is scanned by a 3D scanner, P and M are often obtained in different coordinate systems. As a result, their orientations may be very different. TrICP provides an efficient and robust solution when the two sets are roughly pre-registered. This is typical for all iterative algorithms, for which the pre-alignment is usually done manually. Our genetic pre-registration [22] procedure complements TrICP yielding a robust and completely automatic solution. The genetic pre-registration algorithm minimises the same objective function Ψ (ξ, R, t) as TrICP, but this time as a function of all the seven parameters, namely, the overlap ξ, the three components of the translation vector t, and the three Euler angles of the rotation matrix R. The difference between the genetic solution and the overlap selection procedure [4] is essential. The former means evaluating Ψ (ξ, R, t) for different values of ξ, R, and t, while the latter means running TrICP for different values of ξ. Our genetic solution provides an elegant way to estimate the overlap and the optimal motion simultaneously, by treating all parameters in a uniform way. The solution [4] only works for pre-registered sets. If desired, it can be used to refine the overlap estimate obtained by the GA; we did not do this in our tests.

244


To minimise the objective function Ψ (ξ, R, t), we applied a genetic algorithm tuned to the problem. The objective function was evaluated by mapping each integer parameter onto a real-valued range using normalisation. Simple one-point crossover was employed. Different population sizes were tested and an optimal value was selected for the final experiments. Two mutation operators were introduced. Shift mutation shifts one parameter randomly by a value not exceeding 10% of the parameter range, while replacement mutation replaces a parameter with a random value. The corresponding probabilities were also set after preliminary experimentation. Tournament selection was applied, as it is easy to implement and helps avoid premature convergence. An elitist genetic algorithm was employed, where one copy of the best individual was transferred without change from each generation to the next one. The method is presented in detail in our paper [22]. The main steps of the genetic algorithm are as follows: 1. Generate initial population. 2. Calculate objective function values. 3. Apply genetic operators (crossover, mutation) to selected individuals. Next generation will contain the offspring and, additionally, the best fit individual from current generation. 4. Calculate objective function values for the new population. 5. If best fitness has not changed for Ng generations, stop and designate the best fit individual as solution, otherwise go to step 3. 6. If the solution obtained at step 5 is not of acceptable precision, restart from step 1. We have tested the genetic pre-alignment and the combined method (GA followed by TrICP) on different data. Three examples, the Frog, the Angel and the Bird data, are shown in figures 1, 2 and 3, respectively. To test the method under arbitrary initial orientations, set P was randomly rotated prior to alignment in each of the 100 tests. Results of all tests were visually checked. No erroneous registration was observed. Typical results of alignment are displayed in figures 1, 2 and 3. In each figure, the first two pictures show the two datasets to be registered. The datasets result from two separate measurements of the same object obtained from different angles. The third picture of each figure (GA) displays the result of our genetic preregistration algorithm. Here, the two datasets are shown in different colours. One can see that the datasets are roughly registered, but the registration quality is not high: the surfaces are displaced, and they occlude each other in large continuous areas instead of ‘interweaving’. Finally, the rightmost picture is the result of the fine registration obtained by TrICP using the result of the genetic pre-registration. Here, the surfaces match much better, and they are interwoven, which is an indication of the good quality of the final registration. We have analysed the precision and the computational performance of the genetic alignment algorithm. Some of the numerical experimental results obtained in the 100 tests are presented in tables 1 and 2. (For more numerical


Angel data

GA

245

GA+TrICP

Fig. 2. The Angel dataset, GA alignment and final alignment

Bird data

GA

GA+TrICP

Fig. 3. The Bird dataset, GA alignment and final alignment

results, see [22].) To assess the precision, reference values were obtained by the TrICP algorithm[4] applied to the result of genetic registration. (Since TrICP is much more precise than GA, its output can be used as ground truth.) Table 1 shows the angular error of genetic alignment. The error is defined as follows. Rotation by any matrix R can be viewed as rotation by an angle α around some axis. This angle can be calculated as α = arccos

R11 + R22 + R33 − 1 , 2

(3)

where Rii are the diagonal elements of R. To compute the angular error, we obtain the matrix that rotates the result of the GA to the result of TrICP. The angular registration error shown in table 1 is the angle α calculated by (3) for this rotation matrix. Table 1. GA angular registration errors α (◦ ) Dataset Frog Angel Bird

Mean 0.89 1.17 2.15

Max 1.72 2.11 3.34

Min 0.23 0.28 0.41

Median 0.87 1.15 2.24

Mean deviation 0.22 0.38 0.50

246


Finally, table 2 summarises the performance of genetic alignment in terms of the execution time, the number of elapsed generations, and the number of evaluations of the objective function. The reported experiments were performed on a 2GHz P4 PC running Linux. Table 2. GA computational performance dataset Exec. time (s) Generations Mean Max Mean Max Frog 58 202 203 654 Angel 29 62 122 252 Bird 53 120 176 377

Evaluations Mean Max 101,291 327,004 61,151 126,002 88,046 188,502

3 Fusion of surface and image data In this section, we address the problem of combining geometric and textural information of the object. As already mentioned, the two sources are independent in our system: the 3D geometric model is obtained by 3D scanner, then covered by high quality optical images. After a brief survey of relevant previous work, we discuss our photo-consistency based registration method with genetic algorithm based optimisation. Then we deal with the task of blending multiple texture mappings and present a novel method which combines the techniques of surface flattening and texture merging. Finally, results for real data are shown. 3.1 Registering images to a surface model All registration methods discussed in section 1.1 register unimodal data structured in the same way (measured point sets, range datasets). Fewer methods are available for fusion of essentially different types of data, such as registration of images to surfaces. (Note that this is different from the multimodal image registration, where the data structure is the same.) Several 2D-to-3D (image-to-surface) registration methods were proposed in computer vision and its medical applications. Most of them are based on feature correspondence: Feature points are extracted both on the 3D surface and in the images, and correspondences are searched for. (See, for example, [11, 12, 21].) However, the features are often difficult to localise precisely in 3D models. In addition, defining similarity between 2D and 3D features is not easy. Intensity based registration is another approach to the problem. The algorithm of Clarkson et al. [10] applies the photo-consistency to find the precise registration of 2D optical images of a human face to a 3D surface model. They use calibrated images, thus the problem is reduced to estimating the pose of


Images

3D model

247

Textured model

Fig. 4. The Bear Dataset and result of registration of images to surface

the cameras. None of the methods [10, 11, 12, 21] uses genetic algorithms, since the optimisation problems they consider are easier and solved faster by conventional non-linear iterative strategies. We do not use calibrated camera, so the number of parameters is much higher. The size of the parameter space and the behaviour of the cost function motivated the use of genetic algorithm-based optimisation. Based on our paper [16], let us discuss the registration of images to a 3D model. The input data consists of two colour images, I1 and I2 , and a 3D surface model. They represent the same object. (See figure 4 for an example.) The images are acquired under fixed lighting conditions and with the same camera sensitivity. (The latter can be easily achieved if the images are taken by the same camera.) All other camera parameters may differ and are unknown. The raw data is acquired by a hand-held 3D scanner, then processed by the efficient and robust triangulator [18] developed in our lab. The 3D model obtained consists of a triangulated 3D point set (mesh) P with normal vectors assigned. The finite projective camera model is used to project the object surface to the image plane: u P X, where u is an image point, P the 3 × 4 projection matrix and X a surface point. ( means that the projection is defined up to an unknown scale.) The task of registration is to determine the precise projection matrices, P1 and P2 , for both images. Since the projection matrix is up to a scale factor, it has only 11 degrees of freedom in spite of having 12 elements. The collection of the 11 unknown parameters is denoted by p, which represents the projection matrix P as an 11-dimensional parameter vector. Values of p1 and p2 are sought such that the images are consistent in the sense that the corresponding points – different projections of the same

248


3D point – have the same colour value. Note that the precise mathematical definition is valid only when the surface is Lambertian, that is, the incoming light is reflected equally to every direction on the surface. This is usually true for diffuse surfaces. The formal definition is the following: We say that images I1 and I2 are consistent by P1 and P2 (or p1 and p2 ) if for each X ∈ P: u1 = P1 X, u2 = P2 X and I1 (u1 ) = I2 (u2 ). (Here Ii (ui ) is the colour value in point ui of image Ii .) This type of consistency is called photo-consistency [10, 20]. The photo-consistency holds for accurate estimates for p1 and p2 . Inversely, misregistered projection matrices mean much less photo-consistent images. The cost function is the following: Cφ (p1 , p2 ) =

1 2 I1 (P1 X) − I2 (P2 X) . |P|

(4)

X∈P

Here φ stands for photo-inconsistency while |P| is the number of points in P. Difference of the colour values I1 − I2 can be defined by a number of different colour models. (For details see [16].) Finding the minimum of the cost function (4) over p1 and p2 yields estimates for the projection matrices. The cost function (4) is robustified against occlusion and wrong measurements. Occluded points are eliminated by using the surface normals, and the outliers by rejecting a certain amount of the smallest and largest squares (α-trimmed mean technique). In spite of the simplicity of the cost function Cφ (p1 , p2 ), finding the minimum is a difficult task. Due to the 22-dimensional parameter space and the unpredictable shape of Cφ (p1 , p2 ), the standard local nonlinear minimisation techniques failed to provide reliable results. We have tested a number of widely-known optimisation methods: Newton-like methods, the LevenbergMarquardt algorithm and the Broyden-Fletcher-Goldfarb-Shanno (BFGS) variable metric method. Experiments have shown that local search techniques terminate every time in local minima quite far from the expected global optimum. A global nonlinear optimisation technique has also been tested. However, the stochastic optimisation method by Csendes [7] did not yield acceptable results either. The randomness of a stochastic method is excessive, and it does not save nearly good solutions. In contrast, genetic algorithms preserve the most promising results and try to improve them. (Running a GA without elitism yields unstable and imprecise results, similarly to the stochastic optimisation.) The methods mentioned above and other popular techniques, such as simulated annealing and tabu search process one single solution. In addition to performing a local search, simulated annealing and tabu search have specific built-in mechanisms to escape local optima. In contrast, genetic algorithms work on a population of potential solutions, which compete for survival. The competition is what makes GAs essentially different from single


249

solution processing methods [24]. Consequently we decided to apply a genetic algorithm, as a time-honoured global search strategy. We pre-register the images and the 3D model manually. This yields a good initial state for the search, which narrows the search domain and accelerates the method. Manual pre-registration is reasonable since this operation is simple and fast compared to the 3D scanning, which is also done manually. The photo-consistency based registration makes the result more accurate. The genetic algorithm starts by creating the initial population. The individuals of the population are chosen from the neighbourhood of the parameter vector obtained by the manual pre-registration. The values of the genes are from the intervals defined by the pre-registered values plus a margin of ±. In our experiments was set to values between 1% and 3%, depending on the meaning and the importance of the corresponding parameter. The individual that encodes the pre-registered parameter vector is also inserted in the initial population to avoid losing it. To avoid premature convergence we decided to run the algorithm three times: The algorithm starts three times from the beginning, preserving only the best individual from the previous step and re-initialising the whole population. An iteration is finished if a certain number of generations has been created, or if the best of the population has changed 10 times. Our genetic algorithm for image-to-surface registration works as follows: BEST ←− manual pre-registration. For three times Generate initial population around BEST. BEST ←− best of the population. Loop Calculate cost function values. Apply genetic operators, create new population. BEST ←− best of the population. Until (Ng generations created) or (BEST changed 10 times). End for. We have tested the method with a number of different genetic settings. With proper settings, the projection error of registration can be decreased from 18–20 pixels (the average error of the manual pre-registration) to 5– 6 pixels. After preliminary testing with semi-synthetic data, the following genetic setting has been selected: Steady state algorithm with Tournament selector, Swap mutator and Arithmetic crossover, with 250 individuals in the population, with mutation probability of 0.1 and crossover probability of 0.7. The typical running time with a 3D model containing 1000 points was 5–6 minutes on a 2.40 GHz PC with 1 GB memory. We applied the method to different real data. One of them, the Bear Dataset, is shown in figure 4. The precision of the registration can be best judged at the mouth, the eyes, the hand and the feet of the Bear. Figure 5 visualises the difference between the manual pre-registration and the photo-

250


Manual

Genetic

Fig. 5. Difference between manual pre-registration and genetic registration

consistency based registration. The areas of the mouth, the eyes and the ears show the improvement of the quality. To quantitatively asses the method, we created semi-synthetic groundtruthed data by covering the triangular mesh of a dataset with a synthetic texture. (In this case, the Shell mesh shown in figure 14 was used.) Two views of the object produced by a visualisation program provide the two input images for which the projection matrices are completely known. The projection error is measured as follows. The 3D point set P is projected onto the image planes by both the ground truth and the estimated projection matrices, then the average distance between the corresponding image points is calculated. Evaluating the result of the manual pre-registration by this metric we obtained the above mentioned average error of 18–20 pixels. The results of the genetic method are shown in table 3. The algorithm was tested 10 times with 500 individuals in the population and 10 times with 1000. The average error of 5–6 pixels is acceptable for the images of 512 × 512 pixels. The typical running time with a 3D model containing 1000 points was 15 minutes for 500 individuals and 40 minutes for 1000. (Note that a laser scanner usually provides much more points, but for the image-to-surface registration 1000 points are sufficient.) The test was performed on a 2.40 GHz PC with 1 GB memory. More numerical results related to various parameters of the genetic algorithm are presented in [17]. 3.2 Merging multiple textures When the image-to-surface registration problem is solved, we still face the problem of seamless merging (blending) of multiple textures, that is, images of a surface patch appearing in different views. There are several ways to paste


251

Table 3. Projection error for 10 runs (pixels) 500 individuals 1000 individuals 3.36 3.44 8.77 6.42 5.42 7.79 7.35 4.02 5.97 5.61 12.2 5.38 5.15 6.64 7.52 3.12 5.16 5.35 4.51 8.55 average: 6.54 average: 5.63

texture to the surface of an object. Graphics systems usually have the requirement of two pieces of information: a texture map and the texture coordinates. The texture map is the image we paste, while the texture coordinates specify where it is mapped to. Texture maps are usually two-dimensional, although during the last years the application of 3D textures has also become general. Currently, we only deal with 2D textures. The geometric relation between the 3D surface and the 2D texture map can be described by a transformation. The texture mapping function M assigns a 2D texture point u to a 3D model point X as u = M (X). The relation between the surface and the texture map is illustrated in figure 6. 3D Model

2D Texture Map

M

Fig. 6. Relation between 3D model and texture map

It is straightforward to choose a photo of an object to be a texture map. An optical image of an object contains the colour information we need to paste to the surface. Projection of a 3D surface point X can be described in where P is the 3 × 4 projection matrix and means P X, matrix form: v homogenous coordinates [14]. In this way the texture mapping function is a simple projective transformation.

252


The difficulty of image-based texturing originates from the problem of occlusion, which yields uncovered areas on the surface. An optical image shows the object from only a single view. Therefore, it contains textural information only about the visible parts of the surface; the occluded areas are hidden from that view. (See figure 7b.) This insufficiency can be reduced – in optimal cases eliminated – by combining several images.

(a) Input images

(b) Partially textured models

Fig. 7. Textures cover only parts of the model

Figure 7a shows two images of the globe. These images can be considered as texture maps. Visualising multiple textures on the same surface is not trivial. Visualisation toolkits usually apply interpolation techniques to colour the surface continuously. However, interpolation cannot be applied between different texture maps, hence handling multiple textures yields a gap appearing between the borders. The result is the same when textures are merged by appending the second image to the first one and modifying the second mapping function with a translation. We designed a flattening-based method to create a texture map based on optical images. Flattening the surface mesh of an object provides an alternative two-dimensional parametrisation. The advantage is that this parametrisation preserves the topology of the three-dimensional mesh. A texture that covers entirely the flattened 2D surface covers also the original 3D surface. Figure 8 illustrates the relation between the 3D surface, the optical image and the flattened surface. Transforming optical images to flattened surfaces provides partial texture maps. (See figure 9.) But since flattening preserves the structure of the 3D mesh, these texture maps can be merged, in contrast to the original optical images. Flattening algorithm published in the literature use different principles and require different techniques. (For a recent survey, see [13].) Many methods optimise various energy functions – based on physical models – to minimise the geometric distortion in some sense; the price is that the optimisation is highly non-linear. Other methods ignore some geometric attributes in order to have a simpler optimisation problem. The so-called convex combination method is frequently used by most of these techniques, including the algorithm of K´ os and V´ arady [19] applied in our system.


253

3D Model Flattened Surface

Image

T P F

Ti

Tf

A

Fig. 8. Relation between 3D model, optical image and flattened surface. T is a triangle, F flattening, P projective mapping, A affine mapping

Partial textures

Merged texture

Fig. 9. Partial and merged texture maps

Usually, complex meshes cannot be flattened at once, they need to be cut before flattening. We have chosen to cut by plane, since the cutting plane can be easily determined manually: three points selected on the surface define a plane. The 3D mesh is cut in half by this plane, then the halves are flattened and textured separately. The problem of re-merging the textured halves will be discussed later. Figure 10 shows an example of using the algorithm in our experiments.

(a)

(b) Fig. 10. Mesh of Frog and its parametrisation

254


After flattening the 3D surface, we need to convert our optical images to flattened texture maps. In contrast to the projection matrix, this map is complicated, since neither the transformation of flattening nor the relation between the optical image and the texture map can be represented by a matrix. We use the mesh representation of the 3D surface for conversion: Given a triangle of the mesh, the vertices of the corresponding triangles are known both in the optical image and on the flattened surface. Let us denote these triangles by Ti in the optical image and by Tf on the flattened surface (figure 8). One can easily determine the affine transformation between Ti and Tf , which gives the correspondence between the points of the triangles. Note that the affine transformation is unique for each triangle pair. The algorithm of the conversion is the following: For each triangle T of 3D mesh If T completely visible Projection: Ti ← P · T . Flattening: Tf ← FLAT(T ). Affine transformation: A ← AFFINE(Tf , Ti ). For each point uf ∈ Tf : Colourf (uf ) ← Colouri (A · uf ). End for. End if. End for. Conversion of optical images provides partial texture maps. To obtain the entire textured surface, one needs to merge these texture maps. Since flattening preserves the topology of the mesh, the texture coordinates of the partial texture maps are consistent. The only problem is that of the overlapping areas, where texture maps must be smoothly interpolated. We have tested three methods for handling overlapping areas. The essence of the first method is to decide for each triangle which view it is mostly visible from. The measure of visibility of a 3D point can be the scalar product of the normal vector and the unit vector pointing towards the camera. The measure of visibility of a triangle is the average visibility of its three vertices. For each triangle the texture is given by the texture map that provides the greatest visibility. The main problem of this method is the seams appearing at the borders of the texture maps. (See figure 11.) The second method tries to eliminate this drawback by blending the views. The method collects all the views the given triangle is entirely visible from. All of these views are used for setting the colours of the triangle points. The measure of visibility can be used to set a weight for each view. If the point is better visible from the view, the weight is greater; if less, the weight is smaller. Note that visibility is calculated for each point separately. For this, the 3D surface point and the normal vector are also estimated from the 3D coordinates and the normal vectors of the vertices, respectively.


Method 1

Method 2

255

Method 3

Fig. 11. Differences between the three merging methods

The third method applies the multiresolution spline technique [3] for blending the images. It uses the image pyramid data structure for merging the images in different resolution levels and eliminating the seams between the borders of the texture maps. The same weighting function was applied as in the second method. Using these blending methods the border between the neighbouring texture maps becomes nice and smooth, as one can see in figure 11. The difference between the results of the second and the third method is insignificant. We mentioned that complex meshes need to be cut into pieces before flattening. The pieces are textured separately; however, re-merging them yields seams between the borders. These artefacts can be eliminated by the technique called alpha blending as follows. Colours are usually represented by triples: RGB, CMY, etc. A fourth channel can be used to represent the transparency of the pixel. This is termed as alpha channel and it has a value between 0 and 1. 0 means that the pixel is fully transparent, while 1 means that it is fully opaque. Varying alphas of pixels close to the borders makes discontinuities disappear. Due to the blending methods discussed above, using only values of 0 and 1 is completely sufficient. Cutting the mesh should be performed so as to have overlapping areas, that is, points and triangles close to the borders should belong to both halves. These overlapping areas are covered by the same texture on both halves. The alpha values of the pixels in the overlapping areas are set to 0 in one half and to 1 in the other. This technique guarantees the continuity of the texture in the re-merged model, as illustrated in figure 12.

Images

Textureless 3D model

Texture maps

Views of textured 3D model Fig. 12. Building photorealistic model of Bear


257

4 Tests Our photorealistic modelling system has been tested both on synthetic and real data. The synthetic data provides the ground truth necessary for assessing the performance of the system in terms of precision and computational efficiency. In this section, we give further examples of processing real measured data and creating high-quality models using the algorithms described above. The already mentioned Bear dataset, as well as the Frog, the Shell, and the Cat datasets were acquired by a 3D laser scanner and a high-resolution digital camera. In each case, the complete textureless 3D model (triangular mesh) was obtained by the surface registration algorithm presented in section 2 and the triangulator [18]. Then, some 5–6 images of each object were taken. The images were registered to the 3D model and blended by the methods presented in section 3. For the blending, the 3D models were interactively cut in half. The halves were handled separately and merged only at the end of the process. The results can be seen in figures 12, 13, 14, and 15. To better demonstrate the quality of the model, the Cat model is also shown enlarged in figures 16 and 17.

Images


Texture maps

Views of textured 3D model

Fig. 13. Building photorealistic model of Frog

258


Images


Texture maps

Views of textured 3D model Fig. 14. Building photorealistic model of Shell

5 Conclusion We have presented a software system for building photorealistic 3D models. It operates with accurate 3D model measured by laser scanner and high quality images of the object acquired separately by a digital camera. The complete 3D model is obtained from partial surface measurements using a genetic based pre-registration algorithm followed by a precise iterative registration procedure. The images are registered to the 3D model by minimising a photo-consistency based cost function using a genetic algorithm. Since textures extracted from images can only cover parts of the 3D model, they should be merged to a complete texture map. A novel method is used to combine partial texture mappings using surface flattening. Test results with real data demonstrate the efficiency of the proposed methods. A high-quality model of a relatively small object can be obtained within two hours, including the processes of 3D scanning and photography. Our next step will be adding the surface roughness by measuring the bump maps using a photometric stereo technique.


Images


Texture maps

Views of textured 3D model Fig. 15. Building photorealistic model of Cat

259

260


Fig. 16. Enlarged view of the Cat model


Fig. 17. Another enlarged view of the Cat model

261

262


Acknowledgement This work is supported by the EU Network of Excellence MUSCLE (FP6507752).

References 1. P.J. Besl and N.D. McKay. A Method for Registration of 3-D Shapes. IEEE Trans. Pattern Analysis and Machine Intelligence, 14:239–256, 1992. 2. K. Brunnstr¨ om and A.J. Stoddart. Genetic algorithms for free-form surface matching. In Proc. International Conference on Pattern Recognition, volume 4, pages 689–693. IEEE Comp. Soc., 1996. 3. P.J. Burt and E. H. Adelson. A multiresolution spline with application to image mosaics. ACM Trans. Graph., 2(4):217–236, 1983. 4. D. Chetverikov, D. Stepanov, and P. Krsek. Robust Euclidean Alignment of 3D point sets: the Trimmed Iterative Closest Point algorithm. Image and Vision Computing, 23:299–309, 2005. 5. C.K. Chow, H.T. Tsui, and T. Lee. Surface registration using a dynamic genetic algorithm. Pattern Recognition, 37:105–117, 2004. 6. O. Cordon, S. Damas, and J. Santamaria. A CHC Evolutionary Algorithm for 3D Image Registration. In LNAI, volume 2715, pages 404–411. Springer, 2003. 7. T. Csendes. Nonlinear parameter estimation by global optimization – Efficiency and Reliability. Acta Cybernetica, 8:361–370, 1988. 8. F. Bernardini et al. Building a digital model of Michelangelo’s Florentine Piet` a. IEEE Comp. Graphics & Applications, 22(1):59–67, 2002. 9. K. Ikeuchi et al. The great Buddha project: Modeling cultural heritage for VR systems through observation. In Proc. IEEE ISMAR03, 2003. 10. M.J. Clarkson et al. Using photo-consistency to register 2D optical images of the human face to a 3D surface model. IEEE Tr. on PAMI, 23:1266–1280, 2001. 11. P. David et al. SoftPOSIT: Simultaneous pose and correspondence determination. In Proc. 7th European Conf. on Computer Vision, pages 698–714, 2002. 12. R.B. Haralick et al. Pose estimation from corresponding point data. IEEE Tr. on SMC, 19:1426–1445, 1989. 13. M.S. Floater and K. Hormann. Surface parameterization: a tutorial and survey. In N. A. Dodgson, M. S. Floater, and M. A. Sabin, editors, Advances in Multiresolution for Geometric Modelling, Mathematics and Visualization, pages 157–186. Springer-Verlag, Berlin, Heidelberg, 2005. 14. R. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision. Cambridge Univ. Press, 2000. 15. J.J. Jacq and C. Roux. Registration of 3-D images by genetic optimization. Pattern Recognition Letters, 16:823–841, 1995. 16. Z. Jank´ o and D. Chetverikov. Photo-consistency based registration of an uncalibrated image pair to a 3D surface model using genetic algorithm. In Proc. 2nd Int. Symp. on 3D Data Processing, Visualization & Transmission, pages 616–622, 2004. 17. Z. Jank´ o, D. Chetverikov, and A. Ek´ art. Using a genetic algorithm to register an uncalibrated image pair to a 3D surface model. Int. Sci. Journal of Engineering Applications of Artificial Intelligence, 2005. Accepted for publication.


263

18. G. K´ os. An algorithm to triangulate surfaces in 3D using unorganised point clouds. Computing Suppl., 14:219–232, 2001. 19. G. K´ os and T. V´ arady. Parameterizing complex triangular meshes. In Proc. 5th International Conf. on Curves and Surfaces, pages 265–274, 2003. 20. K.N. Kutulakos and S.M. Seitz. A Theory of Shape by Space Carving. Prentice Hall, 1993. 21. M.E. Leventon, W.M. Wells III, and W.E.L. Grimson. Multiple view 2D-3D mutual information registration. In Proc. Image Understanding Workshop, 1997. 22. E. Lomonosov, D. Chetverikov, and A. Ek´ art. Pre-registration of arbitrarily oriented 3D surfaces using a genetic algorithm. Pattern Recognition Letters, Special Issue on Evolutionary Computer Vision and Image Understanding, 2005. Accepted for publication. 23. M. Levoy et al. The digital Michelangelo project. ACM Computer Graphics Proceedings, pages 131–144, 2000. 24. Z. Michalewicz and D. B. Fogel. How to Solve It: Modern Heuristics. Springer, 2000. 25. G. Renner and A. Ek´ art. Genetic algorithms in computer aided design. Computer-Aided Design, pages 709–726, 2003. 26. C. Robertson and R. Fisher. Parallel evolutionary registration of range data. Computer Vision and Image Understanding, 87:39–55, 2002. 27. M. Salomon, G.R. Perrin, and F. Heitz. Differential Evolution for Medical Image Registration. In International Conference on Artificial Intelligence, pages 201– 207, 2001. 28. L. Silva, O.R.P. Bellon, and K.L. Boyer. Enhanced, robust genetic algorithms for multiview range image registration. In Fourth International Conference on 3-D Digital Imaging and Modeling, pages 268–275, 2003. 29. S.M. Yamany, M.N. Ahmed, and A.A. Farag. A New Genetic-Based Technique for Matching 3D Curves and Surfaces. Pattern Recognition, 32:1817–1820, 1999. 30. Y. Yemez and F. Schmitt. 3D reconstruction of real objects with high resolution shape and texture. Image and Vision Computing, 22:1137–1153, 2004.

Chapter 10 The Use of Fuzzy Information Granular for Content Representation of Video Sequences Congyan Lang, De Xu, Jiehua Yu Institute of Computer Science School of Computer Science & Information Technology Beijing Jiaotong University, Beijing, 100044, China [email protected],[email protected]

Summary In order to provide more efficient content-based functionalities for video applications, it is necessary to extract meaningful video regions from scenes as perceptual-oriented representation of video content. We present a generalpurpose content representation framework for video sequences that employs fuzzy information granulation to capture human perception subjectivity. In particular, the main purpose is to extract spatial-temporal salient grain that is fundamental element for content representation of video sequences. Since perceptual saliency for visual information is a subjective concept, a class-related fuzzy information granulation is constructed for each feature of homogenous regions, mapping original feature space to concepts space. To detect spatial salient regions, segmented homogenous regions are classified according to their prominent importance. After salient region detection, a region tracking mechanism is proposed based on region temporal consistency analysis. The tracking results are sequences of coherent salient regions, called spatial-temporal salient grain. Salient grain can be used to obtain meaningful perceptual-oriented unit in a high-level content description scheme. The experimental results using different types of video sequences are presented to demonstrate the efficiency and accuracy of our proposed algorithm. Keywords: Fuzzy information granular, Content representation, Visual attention, Salient region, Spatio-temporal salient grain, Human visual system

1 Introduction Over the last decade, there has been growing interest in content representation of video sequences with rapid developments in multimedia and internet applications, that is, new representation of video sequences needs to be constructed as not only compact forms of data, but also semantic entities for C. Lang et al.: The Use of Fuzzy Information Granular for Content Representation of Video Sequences, StudFuzz 210, 265–292 (2007) c Springer-Verlag Berlin Heidelberg 2007 www.springerlink.com

266

C. Lang et al.

content-based functionalities such as retrieval and manipulation of semantic content of scenes or specific event in video sequences. A shot can be considered as a basic unit of video sequences, representing a continuous action in time and space. Content representation of video shot is important to content-based video retrieval and analysis, and there has been much work [1, 2] done on matching video shots based on global similarities of low or median visual features such as color, texture and motion. However, current approaches to video processing suffer from one or more shortcomings that stem from the semantic gap between low-level features and high-level semantic descriptions. In order to further exploit the video semantic content, a video shot needs to be decomposed into meaningful content unit. Now, there are three main classes of content representation approaches. (1) One possible approach on content representation is to provide textural descriptions of the shot contents. Manual textural annotation is time-consuming task for large collections of video data. And automatic content annotation is still impossible for the semantic gap between low-level features and highlevel semantic concepts; (2) Key-frames based content representation [3] is another approach. However, key frames are only compact content representation in temporal domain, not provide a semantic meaning of shot. Content representation for key frame still needs to be constructed for content-based applications; (3) Video semantic object is a perfect content representation approach. To bridge the semantic gap, most previous works [4] select video object plane supported by the MPEG-4 standard as the underlying video patterns for video content representation and feature extraction. However, the major problem is that semantic video object extraction in general do not performed automatically, a good performance still needs human’s interaction at the current stage. Recently, many researches on human visual system show that moving objects can be easily distinguished and have more attraction of visual attention. Motivated by the fact, many existing approaches [5, 6] extract moving object as semantic objects for content representation. Moving object can provide a good pattern for motion-related high-level semantic analysis, but moving object based representation approach has poor performance in complexity motion background and is not effect in some static scenes, such as dialog scenes in movie and out-of-field scene in sports games. Studies on visual attention and eye movements [7, 8] have shown that humans generally only attend to a few areas in an image. Even with unlimited viewing time, attention will continue to focus on these few areas rather than scan the whole image. According to the fact, an increasing number of researchers are now exploring the intermediate-level processing, shifting the focus of attention away from the purely local and pixel-based indicators to more global measures that seem to provide a stepping-stone towards a more robust high-level processing. Many image and video processing applications could be made both more efficient and effective if a number of salient regions were first segmented. Generally, most existing approaches [9, 10] aim at detecting the salient regions for images, which are mainly based on the construction of a

Content Representation of Video Sequences

267

saliency map model as an integration of different measurable, low-level image feature such as color, orientation, depth information etc. The purpose of the saliency map is to represent the conspicuity of each locations of the visual field, that is, extracted salient region have higher prominent important than the other regions. In [11], authors use motion information to construct salient map for video sequence, which give superior performance for moving region analysis in video sequence. Salient region extraction based on saliency map [28] provides a good starting point for semantic-sensitive content representation. However, perceived salient region extraction for image or video is still an unsolved problem. One reason is that video sequence has more context information than single image, hence, low-level features are often not enough to classify some regions unambiguously without the incorporation of high-level and human perceptual information into the classification process. Another reason for the problems is perception subjectivity. Different people can differ in their perception of high-level concepts, thus a closely related problem is that the uncertainty or ambiguity of classification in some regions cannot be resolved completely based on measurement methods. A basic difference between perceptions and measurements is that, in general, measurements are crisp whereas perceptions are fuzzy [12]. In this chapter we present a general-content representation framework based on fuzzy information granular. Information granulation is defined by Zadeh [29, 30] as clumps of objects (data points, in particular) drawn together by the criteria of indistinguishability, similarity or functionality, which is an important endeavor of information processing. And fuzziness of granules, their attributes and their values is characteristic of the ways in which human concepts are formed, organized and manipulated [13]. The use of fuzzy granular to partition universes has significant advantages over traditional approaches, and fuzzy granular based classifier tends to be more robust and less sensitive to small changes in attribute values near partition boundaries, therefore, the good generalization capabilities can be achieved. Since human utilize high-level concepts rather than low-level features in perceptual processing, we extract region-level semantic entities from spatial and temporal domains to represent content of video shot. That is, salient grain is extracted as the result of spatio-temporal segmentation for video shot to address the fact that, from the idea observer’s perception, the visual content of a spatio-temporal unit of shots satisfies his/her information needs. Since saliency of visual information is a subjective concept, saliency analysis based on fuzzy information granules can shows better performance than traditional methods based on measurements methods, which provides the possibility to represent content of video sequences in semantic and perceptual manner. Accurate detection of video shot boundaries is beyond the scope of this paper. It is assumed that accurately detected boundaries of video shots have been obtained through shot detection algorithms.

268

C. Lang et al.

After this introduction, this chapter is organized as follows: A brief overview of the proposed framework is described in Section 2. After presegmentation stage in Section 3, we present algorithm for class-based feature granular and temporal coherence analysis by tracking salient region, then spatiotemporal salient grain is constructed in section 4. The effectiveness of the proposed approach is verified by two applications of video processing in the section 5. Concluding remarks are given in Section 6.

2 System overview Figure 1 shows the flowchart of our framework. Due to its importance and foundational to many content-based applications, a content representation framework is required to generate a content pattern that is semantically meaningful based on human perception and to be compact form by efficient reduction for unrelated information. Toward satisfying these requirements, the proposed system extracts spatio-temporal salient grain from video shot. The extracted spatio-temporal salient grain could be used as a good pattern to bridge semantic gap between low-level features and semantic understanding.

Fig. 1. Flowchart of Spatio-temporal Salient Grain Extraction

Since saliency of information is a matter of degree, traditional methods based on measurements methods cannot efficiently analyze saliency. Henceour first attempt will be to use fuzzy information granular for mapping original feature space to concepts space in order to capture the subjectivity of perception, which lead to a good performance for saliency analysis. Then, to detect spatial salient regions, homogenous regions modeled by granular features are classified according their prominent importance. Finally, salient regions are


269

tracked and the results of tracking are sequences of temporal coherent regions, called spatio-temporal salient grain. Spatio-temporal salient grain extracted has distinct characters in spatial and temporal domains. On the one hand, spatial segmentation for the frames can extract perceptually important regions(salient regions) by visual salient features. On the other hand, temporal boundaries of spatio-temporal salient grain can reflect discontinues of temporal content change. Therefore, spatio-temporal salient grain can be exploited as the fundamental elements in semantic level content representation in perceptual manner. Definition: Spatial-Temporal Salient Grain (STG), which is denoted as ST Gk , ST Gk = {SRktb , . . . , SRkte } , is a set of temporally adjacent spatial salient regions within the interval {tb , te }. And salient region, denoted as SRkt , is the region which has high prominent importance in the frame ft , t = tb , . . . , te ,tb < te .

3 Pre-processing In this section, each frame is initially segmented into homogeneous regions based on mean shift algorithm[14] . To segment an image, we first extract a 3-D feature vector for each block. In particular, color components are computed in CIE LUV color space due to its perceptually uniform derivation of the standard CIE XYZ space. We apply the mean shift clustering [14] to segment the image into homogenous regions. Figure 2 shows the results of partitioned homogenous regions for two frames. The level of accuracy in this initial step is important for the overall performance of the proposed prominent region detection, as these pre-segmented homogenous regions will constitute the basic contours of the extracted perceptual prominent regions. For that reason, the parameters of this initial step are selected so that the frames are over-segmented rather than under-segmented.

Fig. 2. Homogenous region segmentation examples: (a,c) original images; (b,d) respective homogenous region segmentation

270

C. Lang et al.

4 Salient region extraction based on fuzzy information granular 4.1 Region feature extraction Human selective attention makes us distinguish important features of input stimulus from others and focus on a critical feature of an input stimulus. Most basically, our visual system is attuned to perceiving change both in time and space, since one of the main goals of human visual system is to minimize uncertainty [8]. This is also in agreement with Gestalt Organization Theories. By taking advantage of these facts, the filmmaker uses the arrangement of the mise-en-scene to attract our attention by means of changes in light, shape, movement and other aspects of the image [15]. Thus, in order to get suitable content pattern of region in video sequences, we extract a number of perceptual features described below. In our previous work [20], we have demonstrated efficacy of these perceptual features for salient analysis. 1) Contrast of region with surroundings (CSr) Regions, which have a high contrast with their surroundings, are likely to be greater visual importance and attract more attention. The contrast importance CSr (Ri ) of a region Ri is calculated as: CSr (Ri ) =

n

|I ∗ (Ri ) − I ∗ (Ri−neighboursm )|

(1)

m=1

2) Orientation Conspicuity of Region (CCr) Gabor filtering allows getting information about local orientation in the image, thus orientation map computed by Gabor filtering is an important recognition cue. Here, this technique is also employed to descript region orientional information importance. Local Orientations Oθ are obtained by applying Gabor filters to the original images from particular orientations θ = {0◦ , 45◦ , 90◦ , 135◦ }. The four oriented images are added up to generate the desired orientation map, and then normalized orientation map I is achieved by using a traditional normalization operator. p Ip , p ∈ Ri CCr (Ri ) = (2) Npixel (Ri ) where denotes the number of pixels in the region. 3) Shape Indicator of Region (SIr) The shape indicator of region can be calculated as Sr (Ri ) =

Nedge (Ri ) , p ∈ Ri Npixel (Ri )

(3)

where Nedge (Ri ) is the number of pixels in the region R( i) which border with other regions.


271

4) Compositional Balance Indicator of Region (CIr) Compositional Balance is an importance factor of the Mise-en-scene to guide our attention and shape our viewing of the image, it can be interpreted as the extent to which the areas of screen space have equally distributed masses and points of interest. Generally the filmmakers often center the frame on the character’s body which is a simplest way to achieve compositional balance [15]. However, no existing works pay attention to this information. Based on the above observations, the compositional balance indicator is determined according to the following measure:  CSr(Ri )   , gc(R) ∈ Ri ,  gc(Ri ) − gc(R) CIr(Ri ) = CSr(Ri )   , gc(R) ∈ Ri  CSr(Ri ) − CSr(Ri ) + gc(Ri ) − gc(R) where gc(Ri ) is the gravitational center of the region Ri and the mean of gravitational center for all regions within a frame is denoted as gc(R). And the region Ri is selected whose gravitational center is the nearest neighbor of the symmetrical point of gc(Ri ) with respect to the midline of the frame. If the character’s body is located in the frame center, we know that the larger CSr and the nearer distance between its gravitational center and gc(R) the region in image is, the larger CIr the region is, meaning that the higher possibility that it will be a character portion of the frame. For the second case, as the same sense, the higher CIr shows that the frame may balance two or more elements encouraging our eye move between these regions. 5) Motion Prominent Indicator (MIr) of region Motion information is an important cue for human to perceive video content. In our work, block motion vectors are used for motion analysis.Since of a motion vector reflects motion velocity of the block, we define the IM (Ri ) as the mean of the magnitude of the motion vectors in the region. An approach similar to [11] is adopted to descript our motion consistency of region, denoted as M C(Ri ). Specially, the motion consistency is calculated based on the entropy, which should be consulted for the details in [11], estimation of the probability is obtained by summation over rows and columns of eight-bin phase histogram of the region. Then we define the motion prominent importance: M Ir(Ri ) = IM (Ri )gM C(Ri )

(4)

4.2 Feature granulation based on fuzzy information granular The goal of this section is to map raw numeric feature space to fuzzy granule space in order to simulate human perception process. Fuzzy Information granulation can be modeled in several formal environments including set theory, rough sets, random sets or fuzzy sets. The use of the fuzz sets is based on

272

C. Lang et al.

the facts that fuzzy sets are distinctive with two important advantages [16]: (1)Fuzzy sets support modeling of concepts that exhibit continuous boundaries. The overlap between fuzzy sets helps avoid brittleness effect when moving from one concept to another. (2) Fuzzy sets exhibit a well-defined semantics and emerge as fully meaningful conceptual entities-building modules defined in problem solving. Therefore, fuzzy set is ideal for our feature space, which is exploited as form of information granular. Mapping the original data to the fuzzy granular domain is preformed by labeling each data with linguistic variables. For each feature fi , a set of fuzzy granules is constructed, the results can be represented by a 3-D structure, denoted as (fi , Gi , M f ), where Gi = gi1 , . . . , gmi i = 1, . . . , n,mi is the number of granules for feature fi . Gi is a set of pre-defined linguistic variable and M f is mapping function. The purpose of feature granulation is to determinate membership function. Building or choosing a proper membership function is an application dependent problem. There have much work on development of membership functions, three main classes is concluded in [17]. One passable approach is that designer of the system completely identifies the form of the information granules, which is not efficient when priori knowledge of system can not be acquired exactly. In order to determine proper membership function by analysis feature distribution, data-driven approaches are exploited usually [18]. In this case, information granules come as a result of optimization of a certain performance index, such as using FCM or other clustering methods. The third class approach is a hybrid of two above methods. For instance, some parameters of the information granulation process can be set up by the user while the detailed parameters of the information granules can be determined through some optimization mechanism. Though data-driven or hybrid approach can convey more experimental evidence than the first mentioned method, they could not be able to reflect the semantics of the problem. In order to overcome this drawback, we propose a new class-related feature granulation algorithm, which is different from previous approaches in two main respects. Firstly, unlike other methods that determine membership function only in a data-drive manner, we combine classification task and data distribution into granular processing. In addition, two-stage granular processing performs construction of fine grain membership function and class-related granules, respectively. In the first stage, fine grain membership function can be obtained in data-driven manner; then at the second stage, optimizing procedure is driven directly by requirements of the classification task. Hence, final granules are constructed which leads to achieve best performance using simpler classifier. In the following, we describe the feature granulation algorithm in more details. By observation of our experiments, we find that the type of membership function is insensitive to classification of salient region, so we select gauss membership function due to its generality and continuity. Therefore, the key point is to how determinate parameters of the membership function. Firstly,


273

similar to the method [19], initial granules are constructed as a result of optimization of a performance index Q defined as: Q = E(A)/Sp( A) N

= E(A)

2 |A(x(k))|

(5)

(6)

k=1

= card(supp(A)) Sp(A)

(7)

is the energy measure of fuzzy set A, and Sp(A) denoted semanwhere E(A) Obviously, if A tends to maximize energy measure, tic measure fuzzy set A. its size could be prohibitively large, then its semantics will be lost. The optimization of performance index needs two requirements. In this sense the granulation process at the first stage is attempting to achieve the highest possible generalization while maintaining the specificity of data sets. After initial granules are determined, the second stage of class-related granules construction is driven directly by requirements of classification. Since each feature has different importance to classification performance, classrelated granules are formed using a criterion of maximizing the objective function. In our work, the classifier aims at finding regions to which a human observer is mostly attracted, salient region extraction belongs to two-class problems, the output class label denoted as {CSR , CSR }. Relative entropy, denoted as DpA(a) ||pB(b) , indicates how distinguishable pA is from pB by maximum likelihood hypothesis testing when the actual data obeys pA . Small values indicate the densities are ’close’. Since relative entropy is not symmetric, we define symmetric divergence W based on relative entropy as objective function to construct class-related granules: = D(pA , pB ) + D(pB , pA ) W (pA , pB ) A } DpA ||pB = x∈k pA log { ppB

(8)

where, pA = p(CSR |Gi ), pB = p(CSR |Gi ) are posterior probabilities. For each feature, final fuzzy granules are determined by maximizing the divergence W. This is a reasonable measure to employ, because it measures how much diversity between two posterior probabilities and distance between of two posterior probabilities can be used as measure for classification performance. Therefore, final feature granules can reflect classification task efficiently and simplify design of classifier. Divergence W can be evaluated by estimated values of p(CSR ), p(Gi |CSR ), p(Gi |CSR ). Assuming a particular p(CSR ), conditional density can be estimated as fuzzy histogram obtained for sum all the data of training set: His(Gi )j = µij (x), j = 1, . . . , mi (9) x∈k

274

C. Lang et al.

Given an original feature F = (f1 , . . . , fn ) in n-dimensional feature space, for each feature fi , i = 1, . . . , n, fuzzy granules Gi is determined by proposed algorithm of class-related feature granulation, the algorithm is summarized as follows: Step 1: Run the mean shift procedure[14] for each feature to find all the modes which are local maxima points, the detected modes are achieved as centers of initial membership functions. The number of initial Gi is equal with total number of modes; Step 2: Maximize performance index Q defined as formula(6) by changing parameter σ within the intervals between two adjacent center points; Step 3: The overlap of two fuzzy sets is calculated as ˜ B) ˜ = sup µ ˜ ˜ (x) L(A, A∩B (x) = min[µA˜ (x), µB˜ (x)] µA∩ ˜ B ˜ Step 4: Calculate symmetric divergence W according to feature granlues Gi by formula (9); Step 5: If the number of granules ≤ Gmin (Gmin = 3), go back to step 7; Step 6: Merge two fuzzy sets which have maximum of overlap, the merging processing of two fuzzy sets is defined as : µA∪ ˜ (x) = max[µA ˜ B ˜ (x), µB ˜ (x)] Step 7: The final form of information granules is determined corresponding to maximum of divergence W. Figure 3 shows energy and performance index curve for the contrast feature, where X-axis represent parameter σ of membership functions, Y-axis is fuzzy set energy in Fig. 3(a) and performance index Q in Fig. 3(b). By optimizing performance index Q initial granules can be determined. At the first stage the center point set of initial membership functions is (0.081, 0.152, 0.213, 0.233, 0.564, 0.816, 0.985). As shown in Fig. 3, there is large overlap among initial granules. After the second stage, four class-related membership functions are created with center points (0.081,0.211,0.564,0.860). By experiment for different video sequences, the results of proposed feature granulation algorithm make classifier tend to be more robust and less sensitive to small changes in attribute values near partition boundaries, the good generalization capabilities can be achieved. The use of fuzzy granulation for mapping original feature has significant advantages, which can capture subjectivity to a certain extent and provide good feature space for saliency analysis. Experiment result for salient region extraction based on fuzzy granules is described in 4.5. 4.3 Classification for salient region The theory of mass assignments developed by Baldwin [21] provides a formal mapping between probability distributions and linguistic descriptions. Due to its capability of manipulating both probabilistic and fuzzy uncertainty,


275

Fig. 3. (a) σ-Energy Curve (b) σ -Performance Index Q Curve

fuzzy granular classifier based on mass assignments [21] is ideal for our salient region classification problem. The aim of the classifier is to find the regions of each frame to which a human observer is mostly attracted. Therefore a prominent importance, related to its perceptual meaning, is assigned to each region of frame. Here, we briefly present the concepts for mass assignments and procedures of classification, the interested reader may consult literature [21] for more details. The bi-directional transformation proposed by Baldwin provides possibility to construct a fuzzy granular classifier combining fuzzy granular and Bayes theorem. A fuzzy set can be transformation, via mass assignment theory, into its corresponding unique probability distribution and vice versa. A Ω mass assignment on a finite set Ω is a function m : 2 → [0, 1] such that s⊆Ω (m(S)) = 1. Given a normal fuzzy set S, its corresponding mass assignment is defined as: ms =< f1 , . . . , fi : πi − πi+1 >

(10)

where πi = µs (fi ), πn+1 = 0 and ∀i ∈ i, . . . , n. And mass assignment can then be transformed into a probability distribution, by distributing the mass associated with each focal element amongst the constituent propositions using prior distribution. A point probability Pr’ is calculated as follows: P rA (fi )ms (A) (11) P r(fi |ms ) = A∈P (F ),fi ∈A

where P(X) denotes the power set of a finite set ΩF , ms (A) denotes the mass associated with focal element A in the mass assignment ms , P rA (fi )is a local probability distribution on a focal element A , which is defined as follows: P r(fi ) P rA (fi ) = P r(z)

(12)

z∈A

For feature vector fi , the fuzzy granular is denoted as G = {g11 , . . . , g1m1 ; . . . ; gn1 , . . . , gnmn }(n = 4). In order to predict the class to which the region

276

C. Lang et al.

belongs we need to calculate the probability P r(Ck |f1 , . . . , fn ), k = SR, SR, . To simplify the model, we assume region features extracted are all conditionally independent, which is intuitively reasonable. According to Bayes theorem, the posterior probability is given by: n -

P r(Ck )

p(fi |Ck )

i=1

P r(Ck |F ) =

p(F )

, k = SR, SR

(13)

p(fi |Ck ) can be estimated from the linguistic prototype for Ck , that is, given the prototype < mg1k , mg2k , . . . , mgmk >, we take p(fi |Ck ) = p(fi |mgik ). From this we can obtain an estimate for P r(Ck |F ). A frame fk is decomposed into two parts by salient region classifier, its salient region is denoted as SRk . 4.4 Spatial-temporal salient grain extraction The final module of the proposed content representation framework aims at detecting spatio-temporal salient grain by temporal coherence analysis for salient region extracted. The temporal boundaries can reflect well discontinuous of content change in video shot. Assuming that there is typically little difference among consecutive frames within short clip of video shot, then rather than processing every frame of the video shot, salient region extraction is performed for the first frame of every temporal window (let length of temporal window l = 5 frames). And for each temporal window, salient regions of the first frame are used as tracking the salient regions in the remainder frames. Salient region tracking is based on region matching, and the similarity of two salient region is defined as: t

t t Sim(SRip , SRjq ) t

t

Dis(SRip , SRjq ) = 1 + d2ij

(14)

t

where Dis(SRip , SRjq ) is Euclidian distances between color features of two region, and dij represents the distance between the centroids of the salient regions. In order to reduce affections of noise region, we defined salient energy measure of salient region as following: E(SRr ) =

l

Sim(SRrt1 , SRrjj ).e−αj t

2

(15)

j=1

Since salient energy reflects temporal coherence within a temporal window, the salient region whose energy value is lower than the threshold Te could be discarded. Therefore, for each temporal widow, a set of salient regions is


277

obtained as the results of tracking salient region and analysis for temporal window salient energy. That is, the salient region SRr that has high temporal coherence, E(SRr ) > Te , is selected as elements of temporal-window level salient grain, denoted as SGtw . After SGtw was detected, the next step aims to group adjacent SGtw with similar content into different spatio-temporal salient grain. The result of grouping is to form temporal boundary of spatio-temporal salient grain. Under the assumption that salient region belonging to the same SGtw have similar content, the criterion used for grouping SGtw is based on the similarity between the salient regions in first frames of two SGtw . For the salient region matching task at hand, this poses a serious problem that is the same salient region at different temporal windows Wi may split in several parts due to noise, occlusion, and low contrast, which will inevitably lead to the fact that a salient region at temporal window Wi+1 may have multiple matches in adjacent temporal window Wi+1 and, therefore, the purpose of the salient region matching is to identify optimal match in with the salient region in Wi . In order to overcome this problem, we defined the similarity of salient based on hypothesis test. Let SRjt+1 and SRkt+1 denote two salient regions in the first frame ft+1 of the temporal window Wi+1 which are similar to the salient region SRit in the first frame ft of the temporal window Wi according to the formula (14). Then the optimal match for the salient region SRit , denoted as SRst+1 , SRst+1 ∈ {SRjt+1 , SRkt+1 , SRjt+1 ∪ SRkt+1 }, is determined by hypothesis test. Assuming that salient region SRit and SRst+1 belong to the same spatio-temporal salient grain, the null hypothesis H0 ; Clearly, the alternative hypothesis H1 states that SRit and SRst+1 do not belong to the same spatio-temporal salient grain. Similar to the method in[22], we adopt the Kolmogorov-Smirnov(K-S) statistics[23] in our work due to its nonparametric nature. The test statistic, denoted as D(SRit , SRst+1 ), is a measure of the maximum difference between the cumulative distribution functions, the distribution of intensity is used in this paper. The test statistic D(SRit , SRst+1 ) can be computed as follows: D(SRit , SRst+l ) = max |F1 (x) − F2 (x)| x

(16)

where x F1 (x) =

x p1 (x)dx, F2 (x) =

−∞

p2 (x)dx

(17)

−∞

p1 (x) = N (x, µ1 , σ1 ) and p2 (x) = N (x, µ2 , σ2 ) is estimated as distribution of intensity. Then, the hypothesis test can be expressed as H0 : D(SRit , SRst+1 ) = 0 (18) H1 : D(SRit , SRst+1 ) > 0

278

C. Lang et al.

The significance level of an observed value of D is calculated as λ = (N + 0.12 + 0.11/N )D, N = min(SRit , SRst+l ) ∞ 2 2 Qks (λ) = 2 (−1)j−1 e−2j λ

(19)

j=1

By calculation for the significance level, merge two temporal windows is performed if the null hypothesis is accepted. Moreover, the optimal match SRst+1 is determined according to maximum of Qks (λ) computed respectively for three salient regions {SRjt+1 , SRkt+1 , SRjt+1 ∪ SRkt+1 }. For instance, as shown in Fig. 7(c), a spatio-temporal salient grain ST G10 is formed by the above methods. Specially, beginning at 17-th temporal window, matching and merging are performed for temporal-window level salient grains W17 W21 respectively. The first frame of temporal window 17 is the frame f86 , and the last frame of temporal window 21 accords to frame f110 . Then the length of ST G10 denotes as l10 , l10 = 25. 4.5 Experimental results Salient region extraction The proposed algorithm was integrated into a system that was tested using several video sequences. In pro-processing stage, clustering parameters is selected on hs = 7, ht = 6.5. Figure 4 shows the results for two sampled frames taken from hall-monitor video Sequences and movie Season II-3 of Friends. Specifically, from left to right, it shows the original frames, corresponding results of homogenous region segmentation, and salient region detection results, respectively. Salient regions are located and used to obtain a binary image that contains white and black pixels to represent prominent region and non-prominent regions, respectively. The hall monitor video sequence is a surveillance video, which is typically used for video object plane generation in MPEG-4 encoding. As we can see, salient region extracted can accord to the fact of human visual perception and is appropriate for use as semantic pattern of content representation. As shown in the Fig. 4 (b), one of the background regions not only has high color contrast but also locates at near the center of image, so both of this region and the character Chandler draw more attention and have high perceptual prominence, which are correctly extracted. Although a direct precision comparison with other system is not possible due to the lack of standard system setups and video sequences, we compared our results with the results of the SVM-based algorithm using low-level information (luminance, color, texture) described in [24]. Figure 5 shows the comparison results. And the superior performance of our approach is illustrated by the comparison results. As we can see, some noisy areas in Fig. 5(a) are removed correctly. Our results also accord to the fact of human selective


279

(a) the results for 31# and 141# frames in hall-monitor

(b) the results for two successive frames in Friends Fig. 4. Salient region extraction results for two frames taken form hall-monitor and movie Friends

attention, that is, when viewing the video clip, human will put little attention on these noisy areas, even though they have a high contrast. That is different from viewing single image. In addition, the method proposed in [24] mistakenly classifier some noise regions that are adjacent salient region, our results has higher spatial accuracy and are closer to actual contour of objects.

(a)

(b)

(c)

(d)

Fig. 5. Four group salient region detection results for two frames taken from sports video clip and hall-monitor sequences: For each group, original frame, the salient region results produced by the method in [24] and corresponding results of our proposed approach are displayed from left to right

280

C. Lang et al.

In our experiments, one major limitation is caused by the noise areas that have the same contrast and similar motion consistency as the salient regions. As demonstrated in Fig. 6, one of background region is assigned mistakenly a high perceptual prominence. However, we expect this drawback can be improved by using spatial relations analysis, which is one of our future works. The noise region located around the salient regions is the other limitations of our system, which is difficult to handle because, in many cases, noise region located around the salient regions has similar high saliency, mean that it has higher value of compositional balance and color contrast. In Fig. 6, the noise region in the middle of two characters is regarded as a salient region mistakenly.

Fig. 6. Salient Region Extraction Results for two frames taken from the movieForrest Gump

Temporal segmentation evaluation on spatio-temporal salient grain extraction Five video shots are taken from different type of video sequences, in particular, which include two movies (Forrest Gump,Friends), two sports video sequences (Basketball games and Soccer games ) and a surveillance video sequence hallmonitor. Figure 7 shows spatio-temporal salient grain extraction results for hall-monitor video, where the result is represented using the first and the last frame of each STG. As we can see, the boundaries of STG can reflect discontinuity of content change within video shot. Although there is not a standard objective


281

(a)

(b)

(c)

(d) Fig. 7. Four spatio-temporal salient grains in hall-monitor video sequences: salient regions at first and last frame of each salient grain

evaluation due to subjectivity of perception and the lack of standard system setups, we employ the quantitative evaluation method proposed by Zacks, et.al. [25] as a more objective evaluation of our results. In the literature [25], they asked human observers to independently mark the boundaries of the finegrained meaningful unit of input video sequences, our expectation is that this fine-grained unit by human observers is likely to be correlated with spatiotemporal salient grain detected by our methods. In our experiment, we ask 10 observers to mark the boundaries of fine-grained meaningful unit, then a histogram of boundary detections is constructed as a function of the frame index. The peaks of these histograms correspond to ground truth of boundaries. Let Nd be the total number of boundaries marked by human observers, Nr be the number of STG boundaries detected by our methods. Therefore, precision and recall can be calculated to evaluate temporal segmentation performance. Table.1 shows the experimental results for five video sequences.

282

C. Lang et al. Table 1. Temporal segmentation results for five video shots Video sequences Number of frames Nd Nr Recall Precision Hall monitor Friends Forrest Gump BG SG

217 331 349 196 236

12 18 11 9 11

15 22 9 12 12

0.92 0.94 0.82 1.00 0.91

0.73 0.77 1.00 0.75 0.83

As expected, the temporal boundaries of STG detected by our method can be correlated well with the discontinuity of content change within video shot. In particular, the result for soccer video sequences is very encouraging due to background similarity. Since STG extraction approach is automatic and nothing prior knowledge is used, temporal over-segmentation occurs in the results. Some salient regions in a time-window is mismatched with regions in successive time-windows due to the noise region, so some short STGs only including one time-window were produced, such as the results shown in Fig. 7(b).

5 Applications In this Section we will demonstrate two practical applications, where the detected spatio-temporal salient grain are used: video shot classification and video sequence matching. These applications essentially involve computing the similarities and dissimilarities between various types of sequences using a certain content pattern for video shot. To represent content of video shot, key frames can be extracted based on spatio-temporal salient grain. Due to the temporal boundaries of spatio-temporal salient grain reflect well discontinuity of content, for each salient grain the first frame is selected as key frame in video shot. 5.1 Sports shot classification Global feature extraction Most existing approaches [26, 27] for soccer shot classification use the ratio of green grass area as main feature. Due to variations of lighting and field conditions, medium shots with high pass colored ratio will be mislabeled as long shots, therefore, the accuracy of this approach is not sufficient for the employment of higher level algorithms. To overcome these limitations, for each shot we defined a number of global features which both computationally low and robust to variations of lighting and field conditions within the sequences.


283

Specially, grass color is computed using information provided by salient region in key frames to handle possible variations of lighting and field conditions. Then for each shot, global features are obtained by integrating grass area ratio and information of salient region, which are fed to input of shot type classifier. 1) Grass Area Ratio (GR): By studying the soccer videos and results of salient region map, we find that for the strong illuminated frames, there is a very distinct salient region map with other frames.

Fig. 8. Salient region detection for two illuminated frames

Form the Fig. 8 we can see that rare salient region can be detected for the frames corresponding to the variations of lighting. Hence, we improve the measure of grass color computation by discarding the frames where the number of salient region NSRi is low (NSRi < Tn ). After detection of prominent region, the histogram of the hue component of non-salient region is added up over these frames. The grass color is represented as the peak of this cumulative hue histogram. 2) Average Relative Size of Salient Region(APsize ): For a salient region of key frame, relative Salient size PS is defined as follows: P Si = Npixel (SRi ) ∗ e−αD(SRi ) D(SRi ) = ||gi−center − gcenter ||

(20)

where Npixel (SRi ) denotes the number of pixels in the salient region SRi , is the gravitational center of the frame and gi−center is the gravitational center of the region SRi .D(SRi ) presents the distance between the location of the region and the center of corresponding frame. The parameter balances the two concerns in the prominent size computation and is chosen so as to control an extent to which the location of region impacts the relative prominent size, which is reasonable because salient region near to the center of the frame is more important than others. Finally, average size of salient region APsize is obtained by computing average value of all the salient regions whose size is smaller than pre-defined threshold Ts . 3) Average Shape of salient Region(APshape ): The shape of salient region is defined as: P Di =

Nedge (SRi ) Npixel (SRi )

(21)

284

C. Lang et al.

where Nedge (SRi ) is number of edge pixels in the region. Then, we compute average value of all the salient region shape as the third global feature APshape . For long shot and Out of field shot, APsize is small and APshape is big, the GR is high in long shot but low in the out of field shot. For medium shot and close up shot, APsize is big and APshape is small, the GR is relative high in medium shot but lower in close-up shot. Therefore, the three global features correlate very well to shot type. Then for classification view types of shot, the global features are used as input of shot classifier to classify shot based on fuzzy granular classifier proposed in section 4.3. After the Global features are granulated, feature granular is fed to the system inputs, four class labels (LS, MS, CS and OS) are output (l=4) of shot type classifier. Shot type classification experimental results Experiments use several soccer video sequences to evaluating the proposed approach. The parameters are determined by experimental tests, more reasonable results of our method could be achieved by setting threshold Tn =0.15, Ts =0.3 and balance parameter α=0.5 in global feature extraction phase. In order to perform training, three video sequences including 1769 frames are randomly selected. And then 181 key frames are extracted by selecting the first frame of spatio-temporal salient grain. For each key frame, we label the ground truth manually. Figure 9 shows the salient region detection results of four key frames taken from different type of shot. Specifically, the original frames, corresponding results of homogenous region segmentation, and salient region detection results are shown from top to bottom row. To evaluate the efficiency for shot type classification with the use of content representation pattern extracted, we compared our results with the results of the crisp classification algorithm (we refer to this methods as “CCA method” throughout the rest of this section) which employed naive bayes classifier using grass-color-ratio described in [27] considering the assumption of feature independence. The soccer video sequence taken from the soccer game: Italian Serie A “Juventus Vs Ascoli”, which has more strong variations of lighting, is selected to perform comparison. Table.2 shows performance of the classification for the test video sequences using two methods. The classification accuracy is calculated as the total number of correctly labelled video clips over total number of detected shot types. The average classification accuracy 90.1% can be achieved using our method. More Specially, from the experiment results, we can see that our approach achieves very high accuracy in classifying long shot (LS) and Out-of fields shot (OS).By comparison with the other type views, accuracy for Medium shot (CS) is relative lower due to the errors created in salient grain detection. The reason for this limitation is that some salient regions are mismatched with salient regions in successive time-windows due to the noise regions, and thus lead to classifier mistakenly medium shots as close-up shots. However, comparison results for classification accuracy still


285

Fig. 9. Salient region detection result Table 2. Statistic Results of Shot Type Classification for the proposed classification method and the CCA method View Types

LS MS CS OS

Ground Truth 382 The proposed Method Miss 10 False 11 Classification Accuracy (%) 97.1 The CCA Method Miss 6 False 15 Classification Accuracy (%) 96.0

112 15 14 85.6 26 18 79.1

107 14 12 87.1 17 12 86.7

56 3 5 90.6 3 7 86.8

show a significant increase in classification especially for medium shot. It is worth pointing that global features based on salient grain extracted has a little sensitivity to the illumination, which provides a good semantic feature to character different types of camera view. 5.2 Video sequence matching Another application is presented on video matching in our work. The key point of video sequence matching is how to construct content representation pattern for video sequences. Hence to show the effectiveness for the proposed framework for content representation, we selected several types of video sequences as testing data to perform video shot matching. Given a video sequences database, we want to match video shot from the database. The result of matching is that similarity can be calculated between video shots in video sequences database, which will lead to efficiently retrieval for video shot. The matching decision depends on the matching feature pattern of content representation based on spatio-temporal salient grain STG. After extraction for STG, a video shot is decomposed into two parts, represented as follows:

286

C. Lang et al.

Shoti = (ST Gk , ST Gk , [tkb , tke ]), k = 1, . . . , Si where, Si denotes the total number of spatio-temporal salient grain in Given two shots Shoti and Shotj , the similarity is measured based on salient information extracted from STG and context information extracted from STG. Since spatio-temporal salient grain can efficient reflect discontinuity of content change within video shot, we select the first frame of every salient grain as key-frame to construct feature vector. Salient region feature of k-th key frame in Shoti , denoted as SFi k , is extracted, which includes color feature (8*3*3 HSV histogram), texture and motion information described in section 4.1. Then 24-dimensions (8-bins for H, 3-bins for S) HSV histogram is extracted from STG, set color threshold TDC =0.2, and context feature, denoted as CFi k , can be constructed as domain color, which is calculated according to the color value that is larger than TDC . CFi k = {(Cj , Hj ), j = 1, 2, ..., NDk }

(22)

where Cj is color value, and Hj is value of histogram corresponding color value Cj, NDk is number of domain color. Once two feature vectors are constructed, a weighted sum of their respective similarity gives the desired similarity measure as shown as follows: SimS(i, j) = WSF ∗ DisShotSF + WCF ∗ DisShotCF DisShotSF (i, j) =

DisShotCF (i, j) =

(23)

max

(DisFSF (k, l))

(24)

max

(DisFCF (k, l))

(25)

k∈Shoti ,l∈Shotj

k∈Shoti ,l∈Shotj

where DisFSF (k, l) and DisFCF (k, l) denote the similarity distance between two key frame k and l according to their salient feature SF and context feature CF. WSF and WCF are the weight for salient feature and context feature respectively, WSF + WCF = 1. To show the performance of video sequence matching methods, we test video sequences matching for the video sequence taken from a movie Forrest Gump, which consists of 428 shots. The result is shown in Fig. 10, where Fig. 10 (a) and (b) showed dissimilarity DisShotSF and DisShotCF (i, j) curves according to salient feature and context feature, respectively. Figure 10(c) descript two similarity curves of weight sum for two features, where red curve is the result for WSF =0.9,WCF =0.1, blue curve is for WSF =0.7,WCF =0.3. As we can see, experiment demonstrates that the descriptions based on salient region are discriminative enough for video shot matching. Video sequences matching can be performed by evaluating the similarity between data sets extracted based on spatio-temporal salient unit, data sets are constructed


(a) Dissimilarity curve of context feature

287

(b) Dissimilarity curve of salient feature

(c) Dissimilarity curve for weight sum of two features Fig. 10. the result of matching for video sequences taken form movie Forrest Gump

by feature extraction for two part of key frames, salient region and non salient region, therefore, salient information and context information can represent well content of video shots. In Fig. 10, context feature has strong discrepancy from about 265-th shot to last shot, which is different with salient feature, that is, these shots have high similarity in salient region. And in our paper, we set larger value for weight of salient feature WSF so that the result of matching depends largely on content similarity of salient region. If requirement of matching application will pay more attention to context information, we can adjust two weights to assign larger value for WCF . Therefore, content representation based on spatio-temporal salient grain provides efficient and flexible matching mechanism for divers applications.

288

C. Lang et al.

Video sequences matching can be performed by evaluating the similarity between data sets extracted based on spatial-temporal salient unit, data sets are constructed by feature extraction for two part of key frames, salient region and non salient region, therefore, salient information and context information can represent well content of video shots. To show the performance of video sequence matching method based on spatio-temporal salient grain, we compare our matching method (denoted as SGM) with the conventional method based on global frame-level feature extraction (denoted as GFM). By the observations from the experiments, more reasonable results of our method could be achieved by setting WSF =0.7,WCF =0.3. Specially, to perform reasonable comparison, 24-bins HSV histogram and 8-bin motion intensity histogram are computed as global features to represent content of key frames conducted on the GFM method. And the similarity is determined by the Kullback-Leibler divergence in both of two methods, the more small the similarity value is, the more two video shots have similarity. Figure 11 shows an example query output that is shown to perform even better than the matching algorithm based on frame-level feature extraction (GFM). For each video shot, first key frame of the shot is displayed. In figure 11, for each replies corresponding to the video query shot, the similarity values, denoted as Sim − SGM and Sim − GF M , are computed by our proposed method SGM and the GFM method, respectively. As can be seen in figure 11, more reasonable results can be achieved and similarity measure using our method can meet the human perception according to more semantic similarity. In particular, for the GFM method, the mismatching lies in the third and fifth replies, due to the scene illumination and limitation for global feature extraction.

Fig. 11. Results of video sequences matching involving five replies corresponding to the video query shot. For each reply, we give the similarity values, denoted as Sim − SGM and Sim − GF M , computed by our proposed method SGM and the GFM method, respectively

Figure 12 shows another comparison results using two matching methods, where the five closest replies from the matching results are presented respectively. As can be seen in figure 12, experimental results show that the proposed method yields better performance than conventional algorithm based on frame-level global feature extraction (GFM). Due to the salient grain extracted, visual information can be organized semantically and feature can


289

(a)

(b) Fig. 12. Results of video sequences matching involving five closest replies corresponding to the video query shot, by the GFM method (a) and our proposed method SGM (b), respectively. For each video shot, the first key frame of the shot is displayed

be constructed to represent content of video sequences. Compared with the method based on frame-level global feature extraction (GFM), the proposed matching method can afford with more semantic content matching and is more robust to the sensitive variety of context information. And thus experiment results verified the benefits more from the good content represent pattern extracted. To carry out a more quantitative evaluation for our matching algorithm, we measure the precision and recall for five video sequences. Five sets of video shots are taken including four video sequences described in Section 4.4 and another video sequence taken from the movie Drunken master 2. For each video sequence, 10 observers determine the similarity of video shots in these sequences, average values of similarities are used as ground truth. Table 3 shows the precision and recall results by proposed method. Table 3. Precision and Recall statistic results of video matching Video sequences

Time

# frames # shots Precision Recall

Forrest Gump Season II-3 of Friends Drunken master2 .mpg Sports Games Basketball games

00:36:19 00:29:08 00:51:56 00:50:21 00:20:00

59372 45582 77830 75686 42000

622 316 684 568 215

0.92 0.92 0.91 0.78 0.81

0.88 0.84 0.87 0.84 0.83

290

C. Lang et al.

The representation increases the flexibility and semantics of content-based video processing since it provides an interpretation closer to the human perception. A more robust description of visual content can be extracted, since possible imprecision and subjectivity of visual information segmentation in spatial and temporal domains are captured to a certain extent. In particular, fuzzy information granules are efficiently exploited in feature analysis, which provide a fundamental scheme for perceptual information processing.

6 Conclusions In this chapter, we present a content representation framework based on spatio-temporal salient grain extraction. Spatio-temporal salient grain using fuzzy information granulation can be exploited as fundamental elements of content representation. Especially, considering the subjectivity and imprecise of human perception, a class-based feature granulation algorithm is performed to map original feature space to fuzzy granules space. Therefore, spatial salient region can be extracted by using simple classifier accurately. Through soft decisions on salient region detection, good performance can be achieved in this stage. Finally, temporal coherence analysis can group temporal similar salient regions to spatio-temporal salient grain whose boundaries reflect content discontinuity of video shot. Hence, salient grain provides good content pattern to represent video shot and can be automatically extracted in perceptionoriented manner. There are a number of potential applications of content representation pattern described here. Since the semantic concept in visual information perception, such as saliency, is usually a matter of degree, different people can differ in their perception of prominent importance, thus a closely related problem is that the uncertainty or ambiguity of classification in some video shot cannot be resolved completely based on measurements methods. Information granulation is an important endeavor of information processing. The use of fuzzy granular to partition universes has significant advantages over traditional approaches. In the area of visual information processing, this endeavor is expected as good solution to capture information uncertainty and subjectivity. Although the experimental results show the encouraging performance, the conducted research also shows that there is plenty of room for improvement. Future work will be focused on the learning rules generation for complicated semantic relations, which is important to high-level semantic content analysis of video sequences, such as goal detection, semantic event classification, etc. 5). In addition, this work can be flexibly extended to the low-level feature extracted directly compressed domain. In our work, however, due to the importance of the low-level feature extraction, how to improve the feature extraction and selection in both compressed and uncompressed domain is the other future work, which needs to perform a good trade-off between computation speed and performance of spatio-temporal content representation.


291

References 1. S. Antani, R. Kasturi and R. Jain, “A survey on the use of pattern recognition methods for abstraction, indexing and retrieval of images and video”, Pattern Recognition Vol.35 pp. 945-965, 2002. 2. Y. Li, S. Narayanan and C.-C. Jay Kuo, “Content-based Movie Anaysis and Indexing Based on AudioVisual Cues”, IEEE Transactions on Circuits and Systems for Video Technology, Vol.14(8), pp. 1073-1085, 2004. 3. H.J.Zhang, J.Wu, D.Zhong and S.Smoliar, “An integrated system for contentbased video retrieval and browsing”, Pattern Recognition, Vol.30(4) pp.643-658, 1997. 4. D. Gatica-Perez, C.Gu, and M.-T. Sun, “Semantic video object extraction using four-band watershed and partition lattice operators”, IEEE Transactions on Circuits and Systems for Video Technology, Vol.11, pp. 603-618, 2001. 5. H. Xu, A. A. Younis and M. R. Kabuka,” Automatic Moving Object Extraction for Content-based Applications”, IEEE Transactions on Circuits and Systems for Video Technology, Vol.14(6), pp. 796-812, 2004. 6. A.S. Ogale, C. Fermuller and Y.Aloimonosh, “Motion segmentation using occlusions”, IEEE Transactions On Pattern Analysis and Machine Intelligence, Vol.27(6), pp. 988-992, 2005. 7. J. Senders, “Distribution of Attention in Static and Dynamic Scenes”, Proceedings SPIE 3026, pp. 186-194, 1997. 8. A. Yarbus, Eye Movements and Vision. Plenum Press, NewYork NY, (1967). 9. L. Itti, C. Koch and E. Niebur, “A Model of Saliency-Based Visual Attention for Rapid Scene Analysis”, IEEE Transactions On Pattern Analysis and Machine Intelligence, Vol.20(11), pp. 1254-1259, 1998. 10. T. Lindeberg, “Detecting salient blob-like image structures and their scales with a scale-space primal sketch: a method for focus-of-attention”, International Journal of Computer Vision, Vol.11, pp.283-318, 1993. 11. Y.F. Ma and H.J. Zhang, “A Model of Motion Attention for Video Skimming”, Proceedings of ICIP (2002) pp. 22-25, 2002. 12. L. Zadeh, “A Note on Web Intelligence, World Knowledge and Fuzzy Logic”, Data & Knowledge Engineering, Vol.50, pp. 291-304, 2004. 13. J. G. Shanahan, “Soft Computing for Knowledge Discovery: Introducing Cartesian Granule Features”, Kluwer Academic Publishers. (2000) 14. D. Comaniciu and P. Meer, “Mean Shift: A Robust Approach toward Feature Space Analysis”, IEEE Trans. Pattern Analysis Machine Intelligence, Vol.24(5), pp. 603-619, 2002. 15. D. Bordwell and K. Thompson, “Film Art: An Introduction”, McGraw-Hill Higher Education, 2001. 16. W. Pedrycz and A.V. Vasilakos, “Linguistic Models and Linguistic Modeling”, IEEE Trans. Systems, Man, and Cybernetics, Vol.29(6), pp. 745-757, 1999. 17. W. Pedrycz and S. Gacek, “Temporal granulation and its application to signal analysis”, Information Sciences, Vol.143, pp. 47-71, 2002. 18. W. Pedrycz and F. Gomide, “An Introduction to Fuzzy Sets. Analysis and Design”, Cambridge, MA:MIT Press, 1998. 19. W. Pedrycz and G. Vukovich, “On Elicitation of Membership Functions”, IEEE Transactions On Systems, Man, and Cybernetics - Part A: Systems and Humans, Vol.32(6), pp. 761-767, 2002.

292

C. Lang et al.

20. L. Congyan and X. De, “Perception-Oriented Prominent Region Detection in Video Sequences”, Informatica: International Journal of Computing and Informatics (to appear in Vol.29(3) 2005). 21. J.F. Baldwin, J. Lawry and T.P. Martin, “A Mass Assignment Method for Prototype Induction”, International Journal of Intelligent Systems, Vol. 14(10), pp. 1041-1070, 1999. 22. S. Hongeng, R. Nevatia and F. Bremond, “Video-based event recognition: activity representation and probabilistic recognition methods”, Computer Vision and Image Understanding Vol.96, pp. 129-162, 2004. 23. R. von Mises, “Mathematical Theory of Probability and Statistics”, Academic Press, New York, 1964. 24. B.C. Ko, S.Y. Kwak and H. Byun, “SVM-based Salient Regions Extraction Method for Image Retrieval”, Proceedings of the 17th International Conference on Pattern Recognition (ICPR’04), 2004. 25. J. Zacks, B. Tversky and G. Iyer, “Perceiving, Remembering, and Communicating Structure in Events”, Journal of Experimental Psychology: General 130(1), pp. 29-58, 2001. 26. Y.-L. Kang, J.-H. Lim, Q. Tian, M.S. Kankanhalli and C.-S. Xu, “Visual Keywords Labeling in Soccer Video”, Proceedings of IEEE ICPR, pp. 850-853, 2004. 27. P. Xu, et al, “Algorithms and System for Segmentation and Structure Analysis in Soccer Video”, Proceedings of IEEE ICME, pp. 928-931, 2001. 28. V. Navalpakkam and L. Itti, “Modeling the influence of task on attention”, Vision Research, Vol.45(2), pp. 205-231, 2005. 29. L. A. Zadeh, “From Computing with Numbers to Computing with Words-From Manipulation of Measurements to Manipulation of Perceptions”, IEEE Trans. On Circuits and Systems-I: Fundamental Theory and Applications, Vol.45(1), pp. 105-119, 1999. 30. L. A. Zadeh, “Information granulation lies at the center of human reasoning and concept formation”, Abstract of BISC Seminar, 1996.

Chapter 11 Wavelets and Fuzzy Contours in 3D-CAD for Digital Breast Tomosynthesis Gero Peters1,2 , Serge Muller1 , Sylvain Bernard1 , and Isabelle Bloch2 1

2

GE Healthcare Europe 283, rue de la Minière, 78533 Buc, France [email protected], [email protected], [email protected] GET-ENST Dép. TSI - CNRS UMR 5141 LTCI - 46 rue Barrault, 75013 Paris, France [email protected]

Summary In this chapter we present a novel approach for the detection of different kinds of lesions in Digital Breast Tomosynthesis datasets. It consists in working directly on the projected views, providing the advantage of a reduced data volume to process, while staying independent of any given reconstruction algorithm, not yet fully optimized for this emerging modality. Our aim was to develop a single processing framework for the detection of different kinds of breast lesions. Introducing fuzzy processing enables us to maintain the evidence, and the strength of the evidence, gathered from each DBT projection image for each potential finding without making hard decisions in isolation. First, the projected views are filtered using different banks of multiscale wavelet filters allowing to better fit the wavelet to the pattern that may vary in a defined range of sizes. The different filter responses are then thresholded and the result is combined to obtain microcalcification and mass candidates. For each candidate, we create a fuzzy contour through a multi-level thresholding process. We extract attributes for each candidate contour that are characteristic for the different kinds of lesions to be detected. Fuzzy set definitions for the classes of the respective lesions are introduced that allow for the computation of fuzzy membership values for each candidate contour. Then, an aggregation operator is presented that combines information over the complete set of projected views, resulting in 3D fuzzy particles. A final decision is made taking into account information acquired over different successive processing steps. Visual examples of detection results are presented, along with a preliminary quantitative evaluation of the algorithm.

G. Peters et al.: Wavelets and Fuzzy Contours in 3D-CAD for Digital Breast Tomosynthesis, StudFuzz 210, 293–323 (2007) c Springer-Verlag Berlin Heidelberg 2007 www.springerlink.com

294

Gero Peters et al.

1 Introduction Breast cancer continues to be one of the leading causes of cancer mortality among women. Since the underlying causes for this disease remain unknown, early screening is the only means to reduce mortality among the female population. X-ray mammography is currently the primary method for detecting early breast cancers, reducing the mortality rate by about 30% for women 50 years and older [12]. However, about 30% of breast cancers are still missed by conventional screening mammography. One of the main reasons is the superimposition of tissue that obscures lesions in dense breasts [8], while digital mammography recently demonstrated superior performance in dense breast imaging compared to film screen mammography [18]. Digital Breast Tomosynthesis (DBT) [9, 27] is a new three-dimensional (3D) limited-angle tomography breast imaging technique that will substantially overcome the superimposition problem for lesion detection. It then remains important to accurately detect and localize microcalcification clusters and masses , which are radiological findings potentially related to breast cancer. The introduction of DBT brings a variety of new challenges and benefits. Several projected views from different acquisition angles will potentially reduce the number of false positives (FP) caused by summation artifacts as well as the number of false negatives (FN) caused by the masking effect of overlying tissue. At the same time, the dose per acquired image is significantly reduced in comparison to standard 2D mammograms, to maintain a comparable total patient dose per scan. This has a major impact on any processing in the projections, as the characteristics of these images change dramatically, and algorithms developed for 2D mammograms cannot be generally applied without modification. As DBT systems become available for clinical testing, different strategies for Computer-aided Detection (CAD) on DBT data are emerging. Chan et al. have presented an approach applying CAD processing on reconstructed slices [5]. A method applying mass detection algorithms directly on the projected views was presented in [19]. Candidates are detected in each projected view separately and afterwards combined in 3D using the acquisition geometry. CAD processing for calcification detection in 3D DBT data has not been made public so far and therefore represents one of the original contributions of our work. Since DBT is a relatively new modality, 3D reconstruction algorithms for its particular geometry are still not fully optimized. Hence, it is desirable to devise a CAD approach that is independent of the reconstruction algorithm used to generate tomosynthesis slices. Fuzzy processing has been widely accepted for use in microcalcification detection tasks [4, 7, 17]. In the present work, we propose an original method using fuzzy particles to account for ambiguities in shape and appearance of microcalcifications and masses for the purpose of modeling and identification. The use of a fuzzy set description enables us to maintain the evidence, and the strength of the evidence, gathered from each DBT projection image for each

Wavelets and Fuzzy Contours in 3D-CAD for DBT

295

potential finding without making hard decisions in isolation. The final decision as to the presence or absence of findings is then made in 3D through aggregation of all available information from all projections. Working directly on DBT projected views offers several advantages. The processing time is reduced compared to the processing of reconstructed slices since they are generally much more numerous than the projected views. The processing is performed on a data space independent of the reconstruction algorithm used to generate 3D images. There are however some issues that need to be addressed. The DBT projected views have a lower Contrast to Noise Ratio (CNR) rendering the detection task in a single image more difficult when using approaches designed for conventional 2D mammograms. They contain ambiguities about the objects in the image, including uncertainty about a candidate particle being a microcalcification or a mass, imprecision of its position and extent, as well as the incomplete nature of data in the individual projections. It is therefore crucial to delay the detection decision for each candidate particle until information from each view can be jointly considered. With this as our motivation, we developed a fuzzy logic approach to take these ambiguities into account and preserve them up to a point in the processing where we have gathered sufficient information to make a decision that simultaneously utilizes all available information. The context of digital mammography and CAD is summarized in Section 2. The novel approach presented here consists of the following processing steps. In Section 3 we start by detecting candidate particles that potentially are microcalcifications or masses using a wavelet filter bank type of detector. In Section 4 we build a fuzzy contour for each candidate particle, based on several extracted attributes and multi-level segmentation. In Section 5 a partial defuzzification is applied, resulting in fuzzy particles better suited for the aggregation operation described in Section 6. Once information from the entire set of projected views has been aggregated resulting in 3D fuzzy particles, their properties are extracted in Section 7 before the final step deciding whether those particles correspond to microcalcifications, masses or other structures. Finally some results are presented in Section 8 and are briefly discussed in Section 9.

2 Context 2.1 Digital mammography The breast is composed of different types of tissue: skin, fat and fibro-glandular tissues. Mammography provides a projection of the breast volume onto the image plane. In order to obtain a good contrast, images are acquired using relatively low-energy photons. Filters and anodes made of Molybdenum (Mo) or Rhodium (Rh) can be used to generate an x-ray spectrum of adequate energies [15]. The detector captures those photons that are transmitted through

296

Gero Peters et al.

the breast tissue. As a result, the pixel intensity in the acquired images is proportional to the number of photons arriving at the detector. These are socalled intensity images, where the signal (assuming monoenergetic photons) is given by: I = I0 e−

L 0

µ(x,y,z)dl

(1)

where µ is the attenuation coefficient of the penetrated tissue and L its thickness (Figure 1).

I0 L

µ

I=I0e-µL Fig. 1. Attenuation of x-rays when passing through an object of thickness L and attenuation coefficient µ (here, as a simplification, for a constant µ and monochromatic x-rays)

Mammography today aims at finding different kinds of lesions indicating potential breast cancers. The most important radiological findings are clustered microcalcifications and masses (dense kernel and spiculated) as shown in Figure 2. 2.2 Digital breast tomosynthesis Digital Breast Tomosynthesis (DBT) is a new three-dimensional (3D) limitedangle tomography breast imaging technique that has the potential to substantially overcome the superimposition problem for lesion detection. Several X-ray projection images of the breast under compression are acquired from different X-ray source positions as shown in Figure 3. A tomographic reconstruction is then performed to produce a 3D image of the breast tissue. Typically 11 to 21 images are acquired over an angular range of ±15 to ±45 degrees. Total patient dose in DBT is comparable to current two-view (CC, MLO) mammography, but is distributed over many projections. The dose per acquired image is thus significantly reduced in comparison to standard 2D mammograms, to maintain a comparable total patient dose per scan. This has a major impact on any processing in the projections, as the characteristics


(a) clustered microcalcifications

(b) dense kernel mass

297

(c) spiculated mass

Fig. 2. Different types of breast lesions

Fig. 3. Principle of Digital Breast Tomosynthesis: Several X-ray projection images are acquired from different X-ray source positions

of these images change dramatically, and algorithms developed for 2D mammograms cannot be generally applied without modification. Furthermore, the increase of data volume when passing from 4 views in 2D mammography to over 100 reconstructed slices in DBT, calls for a means to guide radiologists in their decision process. 2.3 3D CAD approaches Three fundamental processing steps are involved in the architecture of a Digital Breast Tomosynthesis CAD System: data analysis, decision, and backprojection . The analysis of the data leads to particles that are potential findings of interest. The decision step allows to eliminate particles that do not likely correspond to breast structures of interest. Backprojection enables to pass from 2D projected views to 3D space. Depending on the manner in which we assemble these basic modules to form a working processing chain, we can derive three main groups of CAD approaches on 3D-data. Figure 4 shows a schematic of these groups of approaches. In the first group (A) we start by performing a backprojection of the acquired data, therefore

298

Gero Peters et al.

Backprojection

Analysis

Decision

A

Analysis

Backprojection

Decision

B

Analysis

Decision

Backprojection

C

Fig. 4. CAD Strategies: Working directly on 3D data (A), performing part of the processing in 2D, backprojecting the intermediate results and taking the decision in 3D (B), or performing the entire processing in 2D and only backprojecting the result into 3D (C).

Raw Image

tio ca ifi ss la C

C D lus et te ec r tio n

F Su uzz rfa y ce s

Ag

gr

eg

at

io

n

n

n tio P D art ef ia uz l zi fic a

F C uzz on y to ur s

C D an et di ec da tio te n

applying all further processing directly in 3D space. An example is given in [5]. The second group (B) involves a data analysis in the 2D domain, and back projecting the results of this analysis for the decision to be taken in 3D. Finally, the third group (C) consists of approaches where the entire processing is performed on the projected views, and only after the decision has been taken markers of detected objects will be back projected to 3D space [19]. The approach presented in this chapter is a member of the second group. It involves processing in both, the 2D and the 3D domains. This approach benefits from the advantages provided by the tomographic nature of the data while at the same time staying independent of the DBT reconstruction algorithm. A schematic representation of the proposed CAD approach is given in Figure 5. The individual processing modules are described in detail in Sections 3-7.

Acquisition Geometry

Fig. 5. Algorithm scheme: order of the different processing modules in the proposed CAD concept


299

3 Candidate particle detection using wavelets In the initial processing performed on the projected views we extract a map of candidate particles. For each projection, the multiscale wavelet filter responses are thresholded according to a model that includes information about noise. The results are then combined to obtain microcalcification and mass candidates. An important advantage of using wavelets compared to top-hat operator [20, 21] for lesion detection on X-ray mammograms is that we can quite easily predict the impact of linear filters on noise level. Different banks of multiscale wavelets are used that are adapted to the pattern of different types of lesions. For microcalcification detection, we use Mexican hat wavelets that are second derivatives of Gaussians. For detection of masses with dense kernels, we apply modified Mexican hat wavelets on sub sampled images, benefiting from the large size of these patterns. For detection of stellate masses, directional filters are used. A multiscale approach allows to better fit the wavelet to the pattern that may vary in a defined range of sizes. This initial step is crucial to all further processing. Any particle missed by the initial detection cannot be recovered later on. A high sensitivity in this step is therefore of utmost importance. To achieve the desired sensitivity we accept an elevated number of false positives (FP) that will be reduced at a later stage introducing fuzzy particles and the aggregation of information from different projected views. 3.1 Candidate calcification detection Microcalcifications are small deposits of calcium that are expected to be more radio-opaque than their local environment. Their diameter ranges typically from 100 µm to 1 mm with a predominantly compact shape. In the method described below, we suppose that the gray level observed at each pixel (i, j) incorporates the attenuation of a spherical shaped microcalcification of known radius s (the source to pixel X-ray path is assumed to cross the center of the sphere). Then we check whether the contrast measured in the image at (i, j) is sufficient for this point to be considered a calcium pixel given the sphere radius and noise level. Since the projection of the sphere is a disk of radius s, we compute contrast using a Mexican hat wavelet with scale parameters equal to s. Therefore, the negative coefficients of the wavelet corresponds to the area of this disk. A 2D representation of a Mexican hat wavelet is given on Figure 6a. By convolving an image with this filter, we get at each pixel (i, j) the difference between the weighted mean gray level computed in the disk around (i, j) and the weighted mean value in its neighborhood (Figure 6b). This wavelet provides a linear computation of local contrast for a given shape and size of objects of interest [16]. The wavelet’s equation is given in polar coordinates by: r 2 r2 (2) fs (r) = − 1 − 2 e− s2 s

300

Gero Peters et al.

s

Background Calcium

+

Fig. 6. Graph (left) and front view (right) of a Mexican hat wavelet

If the measured contrast is not sufficient to originate from the attenuation of a microcalcification of size s or if the probability for this contrast to originate from quantum noise is too high, then pixel (i, j) is not considered a calcium pixel. Now, we compute the expected attenuation as (r) of a sphere of calcium of radius s in a uniform breast considering a mono-energetic X-ray spectrum characterized by the mean energy of the real spectrum: √ 2 2 Ib e−∆µ2 s −r if r ∈] − s, +s[ (3) as (r) = otherwise Ib Ib is the background intensity in the microcalcification neighborhood and ∆µ is the difference in the attenuation coefficients between breast tissue and microcalcifications. Using Taylor’s series we can demonstrate that, in low contrast situations (where 2s∆µ 1), the contrast ∆I of the microcalcification measured with the wavelet is given by: ∆I = s∆µKIb

(4)

where K is a constant. ∆µ depends on breast composition, microcalcification composition and photon energy. It is difficult to predict ∆µ since the exact composition of breast and microcalcifications are unknown. We can nevertheless fix a lower bound ∆µmin for calcifications that will be tuned according to performance assessment of the algorithm. Thus, we will consider to be calcium pixels only those respecting the following condition. ∆I > αsIb

with

α = ∆µmin K.

(5)

The background intensity Ib is measured in the image with the positive part of the wavelet coefficients (Figure 6) and s is the radius of the microcalcification. The condition expressed in Equation 5 is required but not sufficiently selective especially in low Contrast to Noise Ratio (CNR) areas that often result from


301

low dose acquisitions. In these regions, the probability for the measured contrast to originate from quantum noise is high. Thus, we need to add a second condition on the CNR. To compute the local noise standard deviation σ, we calculate at each pixel (i, j) the 50% fat/50% glandular equivalent breast thickness required to obtain the background gray level Ib using a simulation of the acquisition chain. From an input X-ray spectrum controlled by the acquisition parameters (kVp, mAs, anode, filter), we simulate the attenuated spectrum taking into account the different elements composing the image chain. The gray level is derived from the transformation of the attenuated spectrum into gray level performed by the detector [20]. Changing breast thickness with a dichotomy method, the operation is repeated until the simulated gray level matches the measured gray level Ib . Once we found the equivalent breast thickness, the noise standard deviation is derived from the attenuated spectrum adding electronic noise, the effect of Modulation Transfer Function and wavelet filtrations. Finally a pixel (i, j) is selected as a candidate calcium pixel if for one of a set of scales covering the microcalcification size range, the condition expressed in Equation 5 and a condition on contrast to noise ratio are respected, that is if : ∆I > max(αsIb , βσ)

(6)

α and β are algorithm parameters that are tuned according to performance assessment on a biopsy truthed database. 3.2 Candidate mass detection Masses constitute the second type of findings associated to breast cancer. Mass detection is often difficult because their contours are not always clearly defined and their contrast is often weak. Their diameter varies typically from 0.5 cm to 3 cm. Most suspicious masses are composed of a kernel (of variable density) possibly followed by spicules converging towards the kernel. Dense kernel detection Assuming that mass kernels are of spherical shape, we can apply the same process as for microcalcification detection (Equation 5), with adapted scale parameters. For mass detection, the second condition on contrast to noise ratio may not be required, since we integrate the signal on large areas reducing noise level considerably. Figure 7b shows an example of difference in attenuation coefficients computed using Equation 4, where ∆I and Ib are measured using a single scale Mexican hat wavelet. Strong responses are not only obtained in the mass area but also at the border between pectoral muscle and breast, at

302

Gero Peters et al.

the transition between glandular and fatty areas, as well as close to the breast border.

(a) Original Image

(b) Computed with Mexican wavelet

∆µ hat

(c) Response of our detector

Fig. 7. Dense Kernel Detector

These strong responses outside the mass region come from a biased estimation of the background intensity. With Mexican hat wavelets, the background intensity Ib is computed as a weighted mean value around the mass, in a neighborhood proportional to its size. This makes sense for microcalcification detection because their size range varies from 100 µm to 1 mm. As a consequence, neighborhood background variations are small even close to the breast border. Masses have a diameter that may vary from 0.5 cm to 3 cm. In this case, breast thickness variations in the mass neighborhood may be significant. Thus, we have to face the problem of breast thickness decrease that leads to a high response of Laplacian of Gaussian close to the breast border. In [11, 24] the authors propose to artificially compensate the image for the breast fall-off before applying the filter. To be efficient, this requires a high precision of the compensation level and does not avoid the strong responses at the glandular/fat and pectoral muscle/breast transitions. Since estimating background intensity as the weighted average value in the entire neighborhood of the supposed mass leads to an overestimation of the mass contrast, we prefer computing a lower bound of the contrast by: • estimating mass intensity Im as the weighted mean value in the supposed mass area, where weights are decreasing with the distance from the center of the mass. • considering the most attenuating part in the mass neighborhood as background intensity Ib .


303

Ib is obtained using a sliding window on a circular path around the mass kernel and computing the weighted mean value in the window (Figure 8), where weights are decreasing with the distance to the center of the window. Then, the smallest mean value among the obtained set is selected as background intensity. This results in a more selective response of the detector (Figure 7c) than with conventional filtering (Figure 7b).

BackBackground ground Mass

Fig. 8. Sliding window process illustration

Once we have estimated background intensity Ib , we compute : ∆µ = µb − µm

∆I sKIb

with

∆I = Ib − Im

(7)

where K is a constant. Only pixels for which ∆µ > ∆µmin for at least one scale s are considered to be dense kernel pixels. ∆µmin is a threshold tuned according to a truthed database. This operation is repeated for a set of different scale values s representing the range of mass sizes. Stellate mass detection Stellate lesions are composed of spicules converging towards a kernel of variable density. To detect stellate lesions, we detect spicule pixels, estimate their direction and compute an indicator of convergence. We detect spicules by convolving the raw image with a bank of directional filters having different directions and scales. In [11, 24] the authors convolve the image with only three filter kernels that are second-order directional derivatives of a Gaussian kernel. Their directions are given by θ = π3 Nd , with Nd = 0, 1, 2. The response for an arbitrary direction is derived from the responses of the three filters. Estimating local direction from the response of only three directional filters may be imprecise. Using more filters can be computationally demanding since a convolution requires O(N W ) operations, N being the image size and W the filter size. Computing convolutions in the Fourier domain leads to a reduced computation time since it requires only O(N log(N )) operations, taking advantage of the FFT algorithm. Nevertheless, algorithm complexity

304

Gero Peters et al.

remains considerable for large images. Convolving the image with a β-spline based wavelet [25, 26] requires only O(N ) operations using a recursive implementation method. These are approximations of Gabor wavelets that are compositions of a bandpass filter in the wave front and of a lowpass filter in the orthogonal direction (see Figure 9):   u = xcos(θ) + ysin(θ) 1 − u2 +v2 2 −iωu v = −xsin(θ) + ycos(θ) (8) with gω,θ,σ (x, y) = √ e 2σ e  πσ π w = 2σ

(a) Real Part

(b) Imaginary Part Fig. 9. Gabor Wavelet

Thus β-spline based wavelets are direction and frequency selective as well. Since computation time has been reduced using β-spline based wavelet convolutions, we can increase the number of directional filters to obtain a better sampling of the direction domain. After convolving the image with the filter bank, we select the filter of maximal response in each pixel providing local direction. This selection leads to a directionally filtered image that is thresholded against background noise level in order to select only spicule pixels providing reliable directional information. The background noise level is estimated using the acquisition chain simulation as described earlier (Section 3.1). Now, we suppose that a pixel (i, j) is the center of a stellate mass and we examine the configuration of the spicule pixel directions in its neighborhood. The neighborhood size is provided by the dense kernel detection process by considering the size of the filter of maximal response. We used the method of [11, 24] to compute an indicator of spicule convergence towards (i, j). Then, we decide on the presence of a stellate mass at (i, j) from the magnitude of the convergence indicator. For the detection of stellate masses containing also a dense kernel, the indicator of convergence might be combined with the response of the dense kernel detector using, for example, a learning algorithm or a rule-based system to provide a better decision.


305

4 Fuzzy contours and fuzzy measurements 4.1 Introduction Fuzzy logic [28] can be used as a tool to manage ambiguity in images. We can first deal with ambiguity at the segmentation level, then during the classification of the segmented particles using attributes measured from the contours. We usually distinguish between two types of ambiguity: imprecision and uncertainty. At the segmentation level, imprecision characterizes particles we can identify with a high level of confidence but having contours difficult to localize with a high level of accuracy (Figure 10a). On the other hand, uncertainty characterizes particles having several potential contours for which it is difficult to determine the most relevant one (Figure 10b). This is the case when we have superimposition of objects, as in digital mammography. Contour ambiguity propagates to the attributes measured from segmented particles and therefore to classification using those attributes. Based on previous developments [4, 22], our approach presents the advantage to take into account ambiguity in the image from the particle contours, introducing fuzzy contours, and to transfer this ambiguity up to the classification step through fuzzy attributes.

(a) Imprecision

(b) Uncertainty

Fig. 10. (a) A particle identified with a high level of confidence, but with contours difficult to localize with a high level of accuracy, presents imprecision. (b) A particle having several potential contours, for which it is difficult to determine the most relevant one, presents uncertainty

306

Gero Peters et al.

4.2 Segmentation of particles and fuzzy contours We considered images with particles that can be extracted using a thresholding technique. Therefore, we started from the regular thresholding pyramids approach [13] to describe thresholded images. This transformation consists in applying a set of thresholds corresponding to the gray levels available in the image. Each threshold determines connected subsets that can be linked one to the other using the usual inclusion operator. A fuzzy segmentation method was proposed by Kanzaki [10] that can be considered to be the application of a regular thresholding pyramid to the result of the top-hat transform applied to an image. Particles are therefore considered to be membership functions to the class contour, each alpha-cut applied to these functions being a potential contour for a particle. Smits et al [23] proposed a similar approach improving computation of membership degrees and using a more sophisticated method for propagation of contours. In this approach, the image is processed at structure level and no more at pixel level. It is therefore possible to calculate membership degrees of pixels from attributes based on gray levels of the current and surrounding pixels but also based on geometrical properties of the particles themselves. Nevertheless, these two approaches cannot handle superimposed objects, as present in radiological images, since they are both based on the assumption that membership degrees increase with gray levels. Our fuzzy segmentation approach is composed of three steps: • provide a marker for each particle to be segmented; • determine an ordered set of candidate contours for each particle; • build a fuzzy contour for each particle through the valuation of the candidate contours using their membership degrees to the class contour. Handling imprecision and uncertainty of the particles at contour level, as opposed to pixel level, allows the reduction of the fuzzy segmentation dimensionality. Let the regional maxima of the intensity image I be defined as M ax(I). We then consider as markers the regional maxima MI ∈ M ax(I) of the intensity image I determined in the support of the candidate particles detected using wavelet filtering. Ordered set of contours For each particle marked by a marker MI a set of candidate contours is obtained by region growing. The threshold of I at level ti is defined as: Iti = {(x, y) | I(x, y) ≥ ti }

(9)

Using this definition, we determine a set of contours CMI using a multithresholding operator such as:


CMI

. = ci = Cont(CMI (Iti )), t1 = max(I) > t2 > · · · > tnI MI

307

(10)

with Cont(CMI (Iti )) being the contour of the connected component CMI for the set CMI (Iti )i=1,··· ,nI that includes the marker MI of the image I, and t1 = maxMI (I) being the maximum gray level of the image I in the marker MI . This process is depicted in Figure 11a and 11b.

I(x,y)

I(x,y)

M

I(x,y)

M

M2 M1

ci

CM

ti

ci

ci

(x,y)

(a) gray level profile of a candidate particle and applied threshold at ti .

(x,y)

(b) corresponding extracted contours for different ti .

(x,y)

(c) contours belonging to several particles marked with different markers.

Fig. 11. Set of Candidate Contours

Each maximum of the image is processed independently. Therefore, some contours ci can be duplicated and belong to several sets of contours built for particles marked with different markers (Figure 11c). This property allows handling superimposition in radiological images. For a given particle marked with a local maximum MI , the set of contours CMI is considered the universe of discourse X for the fuzzy logic approach. Using a multi- threshold segmentation approach, candidate contours of a given particle are ordered with a partial order relation ≤ defined by: ∀ci , ∀cj , 1 ≤ i, j ≤ nI ,

ci ≤ cj ⇔ CMI (Itj ) ⊆ CMI (Iti )

(11)

Therefore, the universe of discourse X = {ci }i=1,... ,nI is ordered3 . Fuzzy contour computing To each candidate contour ci , we assign a numerical value corresponding to its membership degree to the class contour using the characteristics of the 3

Since I is a function, its successive thresholds are nested, i.e. ti ≤ tj ⇒ Itj ⊆ Iti , therefore ≤ is a partial order. Eliminating identical contours, extracted for different threshold levels ti , leads to a total order relation.

308

Gero Peters et al.

particle to be segmented (contrast, shape, size, texture index, localization, etc.). Therefore, a particle in the image I with a marker MI is characterized by a fuzzy contour, fuzzy subset of X , defined by a membership function µC/A1 such as: µC/A1 : {c1 , · · · , cnI } → [0, 1]

ci → µC/A1 (ci )

(12)

with µC/A1 (ci ) being the membership value for the candidate contour ci and A1 a fuzzy subset characterizing a criterion that allows the calculation of the membership value. The membership function to the class contour µC/A1 is built using a criterion characterized by a fuzzy subset A1 that represents a priori knowledge about the pertinence of the candidate contours (Figure 12). Let us assume that the most pertinent contours of a given particle have the highest gradient mean values under the contour. Based on the knowledge we have about the images to process, we build the membership function µA1 that characterizes the fuzzy subset A1 of the high gradients defined on the universe Y1 of the gradient values a1 . For a given particle, we measure the gradient value under each candidate contour ci , determining an application ψ from the universe X of the candidate contours ci to the universe Y1 of the gradient values a1 . Using the fuzzy subset A1 on the universe Y1 allows defining the fuzzy subset CA1 on the universe X : µC/A1

µA1

fuzzy subset contour for the criterion high gradient

fuzzy subset high gradient

1

1

0

a1 0 a1

ci gradient values for each candidate contour

ci

Fig. 12. A priori knowledge about the images to process allows building the fuzzy subset high gradient (top left). For a given particle, the value of the gradient under each candidate contour is measured (bottom right). Therefore, the fuzzy subset contour relative to the criterion high gradient for the particle is derived (top right)


µC/A1 (ci ) = µA1 (ψ(ci ))

309

(13)

When a combination of several criteria is considered (e.g., high gradient and small area for particles being potential microcalcifications), the valuation of the candidate contours ci can be extended. Let us consider the fuzzy subsets A1 , A2 , . . . , Ap defined on the universes Y1 , Y2 , . . . , Yp of the values of characteristics a1 , a2 , . . . , ap respectively. The fuzzy subset C on the universe X relative to the criteria characterized by the fuzzy subsets A1 , A2 , . . . , Ap is then the intersection of the fuzzy subsets CA1 , CA2 , . . . , CAp (Figure 13). The membership function to the class contour µC/A1 ∧A2 ∧...∧Ap based on the criteria a1 , a2 , . . . , ap is defined by: % µC/A1 (ci ), µC/A2 (ci ), · · · , µC/Ap (ci ) ∀ci ∈ X , µC/A1 ∧A2 ∧...∧Ap (ci ) = (14) / where denotes a conjuctive operation (we used the minimum).

µC/A1

µC/A2

fuzzy subset contour for the criterion high gradient

1

µC/(A1Â2)

fuzzy subset contour for the criterion small area

fuzzy subset contour for the criterion high gradient and small area

1

ci

0

(a)

1

ci

0

(b)

ci

0

(c)

Fig. 13. The fuzzy subset contour relative to the criteria high gradient and small area is obtained from the minimum between the fuzzy subset contour relative to the criteria high gradient and the fuzzy subset contour relative to the criteria small area

4.3 Measurements of fuzzy attributes Once the fuzzy contour of each particle marked with a marker MI has been built, we are interested in differentiating particles that are potential radiological findings from particles that may correspond to normal breast structures. We therefore suggest using fuzzy attributes measured on the fuzzy contours as input of a classifier (e.g., a fuzzy decision tree) to achieve this classification. For each segmented particle in the image I, we have a fuzzy contour characterized by its membership function µC defined on the universe XC of the

310

Gero Peters et al. µAj,C

µC/(A1Â2) fuzzy subset contour for the criterion high gradient and small area

1

0

1

ci

fuzzy attribute compacity for the criterion high gradient and small area

Aj

0

Aj

1

0

ci

Fig. 14. For a given particle, the value of the attribute (e.g., compacity) is measured for each candidate contour (bottom left). From the particle segmentation step, we know already the fuzzy subset contour relative to the criterion high gradient and small area (top left). Once the value of the attribute has been measured for each candidate contour (bottom left), the extension principle is applied to derive the fuzzy attribute compacity relative to the criterion high gradient and small area for the current particle (top right)

candidate contours ci . For each candidate contour ci , we can therefore calculate the value aj (ci ) of the attribute aj (e.g., contour compacity), defining an application ϕ from the universe X of the candidate contours ci to the universe Yj of the attribute values aj . Finally, we build the fuzzy attribute Aj,C of Yj characterized by the membership function µAj ,C using the extension principle (Figure 14): ∀aj ∈ Aj µAj ,C (aj ) = supci ∈X |aj =ϕ(ci ) µC (ci ) µAj ,C (aj ) = 0

if if

ϕ−1 (aj ) = ∅ ϕ−1 (aj ) = ∅

(15)

With fuzzy attributes, we keep the uncertainty and the imprecision of the particles in the values of the fuzzy attributes. The larger the support of the membership function, the larger the imprecision. The uncertainty leads to multiple lobes in the membership function.


311

5 Partial defuzzification We now have a description of each candidate particle based on a set of contours and a number of attributes extracted for this set. Ultimately, the goal of the applied processing will be to derive a 3D representation of the candidates. In order to achieve this goal, information from all the 2D projected views has to be aggregated in the 3D space. The aggregation operator will be discussed in more detail in Section 6. Here, we address an intermediate step introduced to simplify the implementation of the aggregation. This step can be seen as partial defuzzification . At this stage of the implementation of our CAD algorithm, we propose to convert the actual contour description into a pixel description (see Figure 15). This provides a great flexibility in the choice of a fusion operator. In order to reduce complexity and therefore gain in terms computational cost, we build a particle representation, where each pixel holds exactly one value. This value should be a summary of all information extracted so far for this position.

I(x,y)

I(x,y)

M

I(x,y)

M

M

ck ci

max

ci (x,y)

(a) candidate contour ci .

(x,y)

(b) associated connected component c˙i .

cnI

(x,y)

(c) set of n = (nI − k + 1) membership values for a given pixel.

Fig. 15. Conversion of the contour description into a pixel description

The transformation we introduce here consists in defuzzifying some but not all aspects of the fuzzy contours, thus reducing the degree of fuzziness of the candidate particles. The set of contours of a particle marked by the marker MI ∈ M ax(I) is given by Equation 10. The set of points belonging to the particle is then simply defined as: . C˙MI = c˙i = CMI (Iti ), t1 = max(I) > t2 > · · · > tnI (16) MI

This set is crisp, and the next step consists in assigning attribute values to each point, based on the information on the contours. Here, we choose to rep-

312

Gero Peters et al.

resent this information by using the membership value of the given candidate contour to the class contour. This membership value is given by Equation 12. This value will now be attributed to all pixels of the corresponding connected component (Figure 15(b)). Each c˙i becomes a fuzzy set with the following membership function: ∀(x, y) ∈ c˙i ,

µc˙i (x, y) = µC (ci ).

(17)

As shown in Equation 11, the set of candidate contours {ci } is an ordered set. Thus, succeeding contours of this ordered set are nested. We have attributed a membership value to all pixels of the connected component that corresponds to a given candidate contour. Therefore, we have for each pixel (x, y) a set of possible membership values MMI (x, y) as shown in Figure 15(c). The number of elements of MMI (x, y) is equal to the number of connected components n = (nI − k + 1) that include the pixel (x, y): ∃i | (x, y) ∈ c˙i

with

c˙i ∈ C˙MI ,

MMI (x, y) = {µC (ck ), · · · , µC (cnI )} (18)

where n is the number of connected components c˙i satisfying the condition: (x, y) ∈ c˙i .

(19)

In order to obtain a single value for each pixel position, we need to define an operator that combines the set of membership values MMI (x, y) into a single membership value per pixel µC . Since we have no prior knowledge about the best contour, we choose to proceed in a cautious manner, which leads us to a disjunctive operator . This type of operator, i.e. a t-conorm , performs a union of all pieces of information. It appears obvious that the resulting membership value should not be higher than the highest membership value in the set. We have therefore chosen the max operator, being the smallest t-conorm (see Figure 16). 0 Using the max operator denoted as , we derive a single membership value for each pixel of a given candidate particle using: 1 µc˙i (x, y) (20) ∀(x, y) ∈ I, I (x, y) = i

where I is the set of pixels contained in a tomosynthesis projection image, I (x, y) is the resulting 2D fuzzy particle and ci (x, y) is the ith connected component containing the pixel (x, y). However, since we have allowed contours to belong to different markers at the same time (see Section 4.2), we cannot simply combine the fuzzy particles into a resulting image. The same pixel may receive different resulting membership values with respect to different candidate particles (Figure 16). In fact, we have to apply the max operator not only to decide between the membership values of competing candidate contours of a given particle, but

Wavelets and Fuzzy Contours in 3D-CAD for DBT I(x,y)

I(x,y)

MI(2) MI(1)

I(x,y)

313

MI(2)

MI(1) max max (x,y)

(a) contours marked by two different (1) and markers MI (2) MI .

(x,y)

(x,y)

(b) resulting membership value relative to (1) marker MI .

(c) resulting membership value relative to (2) marker MI .

Fig. 16. Problem of Overlapping

at the same time to decide between the membership values of a given candidate contour to different candidate particles (16). Taking this into account, Equation 20 becomes: 1 11 Im (x, y) = µc˙m,i (x, y) (21) ∀(x, y) ∈ I, I (x, y) = m

m

i

where c˙m,i (x, y) is the ith connected component of the candidate particle marked by the mth marker that contains the pixel (x, y) and I (x, y) is the resulting 2D fuzzy particle map, which is a representation that is well suited for the coming aggregation step, in order to combine information acquired in different tomosynthesis views.

6 Aggregation After performing separate fuzzy detections in each of the Np projection images of the DBT acquisition, we have generated a so-called 2D fuzzy particle map corresponding to each projected view. The next step now consists in aggregating the information given by this set of 2D fuzzy particle maps in order to obtain what we call a 3D fuzzy particle map. The spatial correlation between the set of 2D fuzzy particle maps and the resulting 3D fuzzy particle map is established through the use of a-priori knowledge about the acquisition geometry. Our goal is to find for each 3D voxel the corresponding information in all of the Np fuzzy particle maps that were created. The aggregation of information gathered in the fuzzy particle maps for a given voxel is expressed as:

314

Gero Peters et al. p V (xv , yv , zv ) = Ψk=1 [Ik (xk , yk )]

N

(22)

where V (xv , yv , zv ) is the voxel intensity at position (xv , yv , zv ), Ik (xk , yk ) is the pixel intensity at position (xk , yk ) of the kth fuzzy particle map (see Equation 21), corresponding to the projection of position (xv , yv , zv ), and Ψ is the aggregation operator. Figure 17 illustrates this aggregation operation.

(xV,yV,zV)

(x1,y1)

(x2,y2)

...

(xN-1,yN-1)

(xN,yN)

Fig. 17. Information aggregation strategy: for a given voxel (xv , yv , zv ) the information from all corresponding pixels (xk , yk ) is aggregated using the operator Ψ . The position of the pixel (xk , yk ) corresponding to the projection of a given voxel (xv , yv , zv ) in the kth fuzzy particle map is computed using a priori knowledge about the acquisition geometry

We now have to carefully choose the aggregation operator Ψ . A classification of data fusion operators with respect to their behavior was proposed in [2]. Three classes of operators are distinguished: Context Independent Constant Behavior (CICB) operators, Context Independent Variable Behavior (CIVB) operators, and Context Dependent (CD) operators. CD operators are not only computed from the information issued by the sensors, but also depend on global knowledge about the sources to be fused. The behavior of CIVB operators is context independent, but their behavior depends on the information issued from the sensors. Since in our case all sources (the different projected views) are equitable, and because we expect a behavior that is the same whatever the values to combine, we have chosen an operator of the CICB class. Three types of operators used in fuzzy sets theory are CICB: triangular norms (t-norms), triangular conorms (t-conorms) and mean operators. Within these types of operators, the mean operators are the only ones that always behave like a compromise. For reasons of simplicity, we have chosen the arithmetical mean to be used as aggregation operator in combining the set of 2D fuzzy particle maps. Using this aggregation operator, Equation 22 can be rewritten as:


V (xv , yv , zv ) =

315

Np 1 [Ik (xk , yk )] Np k=1

=

Np 1 Ik (sk (zv ) · xv + ξk (zv ), sk (zv ) · yv + ηk (zv )) (23) Np k=1

where ξk (zv ) and ηk (zv ) are the shift factors in xv and yv direction and sk (zv ) is the scaling factor. These factors are known since they are directly related to the acquisition geometry.

7 3D processing Once we have established a 3D fuzzy particle map, some particles may still represent false alarms. A simple threshold applied to the 3D fuzzy particle map can eliminate candidate particles that potentially correspond to noise structures. Alternatively, particle markers can be obtained by detection of regional maxima of the fuzzy particle map: MV ∈ M ax(V ).

(24)

Using the markers, we can extend 2D fuzzy processing to 3D particles in order to differentiate radiological findings from other structures. 7.1 Extension of 2D fuzzy processing After 3D fuzzy particles have been created, and each candidate has been given a unique marker, a fuzzy surface is computed for each marked candidate particle. This operation is analog to the creation of fuzzy contours in 2D described in Section 4.2. Let the threshold of V at level ti be defined as: Vti = {(x, y) | V (x, y) ≥ ti }

(25)

Applied to fuzzy surfaces of 3D fuzzy particles, Equation 10 becomes: SMV = {si = Surf (CMV (Vti )), t1 = max(V ) > t2 > · · · > tnV } MV

(26)

where SMV is the set of candidate surfaces for the particle marked by the marker MV , si is the ith candidate surface and Surf (CMV (Vti )) is the surface of the 3D connected component of the set Vti containing the marker MV . To extract this set of candidate surfaces, we use a multilevel thresholding technique. Now, a series of decreasing thresholds is applied to the 3D fuzzy particle map, and the level-sets are extracted. This process is repeated until a given set of conditions is met. Those conditions depend directly on the type of object that is to be detected. Typically measures of size and membership value

316

Gero Peters et al.

may be included. For the example of microcalcification detection, possible conditions are: Vol(si ) ≤ Volmax

(27)

µC (si ) > 0

(28)

and

where Vol(si ) is the volume enclosed by the current candidate surface, and µC (si ) is the 3D aggregation of the 2D membership values to the class contour for the candidate surface si . In the same manner it can be shown that, using the segmentation algorithm described above, the candidate surfaces of a given candidate particle form a partially ordered set: ∀si , ∀sj , 1 ≤ i, j ≤ nV ,

si ≤ sj ⇔ CM (Vtj ) ⊆ CM (Vti ) V V

(29)

where si and sj are the ith and jth candidate surface respectively. Prior knowledge about the objects to be detected in the images is translated into a fuzzy set description using specific characteristics of the particles: B1 , B2 , . . . , Bq . Functions expressing the degree of membership to the class surface of a particle as a function of the different characteristics are defined. Using these functions, the membership values for each candidate surface si are calculated as the conjunction of membership values for the different particle characteristics used to establish the class definition: µS/B1 ∧B2 ∧...∧Bq (si ) : {s1 , · · · , snV } → [0, 1]

si → µS/B1 ∧B2 ∧...∧Bq (si ) (30)

Here, µS/B1 ∧B2 ∧...∧Bq (si ) is the membership value to the class surface of the ith candidate surface si . When computing the membership values of the candidate surfaces to the class surface, we will have to be very careful not to confuse the newly computed membership value to the class surface µS/B1 ∧B2 ∧...∧Bq (si ) with the membership value in each voxel of the candidate surface µC (si ), that stems from the aggregation of membership values to the class contour µC (ci ). Once the fuzzy surface has been created, we can extract fuzzy attributes describing this fuzzy surface using again the extension principle. Those fuzzy attributes can then be used as input of a classifier to differentiate particles that are potential radiological findings from particles that may correspond to normal breast tissue. 7.2 Cluster detection For some detection tasks, like the detection of microcalcifications, it is of great interest to extract more information than the one based on individual


317

particles only. Hence, we proceed in a manner analog to the processing of the individual particles. The processing can be divided into two steps: a first step for identifying the objects and a second one to analyze them. The type of objects that we are looking for is a group or specific configuration of particles. In order to detect this kind of objects, we need to analyze the particles and their (spatial) relations, enabling us to compare them to a given prototype. This type of processing is achieved by construction of a graph , where a node represents each particle. The relations between particles are represented as arcs of the graph. Once the object has been identified, a multi-thresholding technique is applied to derive a fuzzy representation of a cluster of particles. This fuzzy cluster is then analyzed by extracting a set of fuzzy attributes (density, homogeneity, morphology, size, etc.). 7.3 Classification We have now gathered information over the full range of tomographic projection images on particle level as well as on cluster level. Using fuzzy logic based processing we were able to propagate the ambiguities present in the data up to the final decision step. This decision is now achieved by use of a classifier suitable to process the fuzzy data provided by the preceding processing steps. Many classification techniques today are trying to take advantage of the benefits of fuzzy processing. Incorporation of fuzzy rules and fuzzy processing exists for most popular classifiers. However, there are only very few approaches where classifiers actually work on fuzzy data. Such implementations have notably been made public for decision trees [3] and neural networks [6, 14]. When comparing these two types of classifiers, fuzzy decision trees seem to respond well to our needs, since they offer the possibility to easily include expert knowledge, process numerical as well as symbolic data, sort the attributes depending on their discriminating power and allow for an easy interpretation of the obtained system. Implementation of a fuzzy decision tree working on fuzzy data enables us to derive a final segmentation decision from the data gathered over the whole range of previous processing steps.

8 Results In this section we show the result of applying these methods to clinical DBT data. Figure 18 shows a projected view and the corresponding fuzzy particle map. In Figure 19 we see the results of aggregating in 3D (middle and right) alongside a reconstruction slice (left) that was reconstructed for comparison using Simultaneous Algebraic Reconstruction Technique (SART) [1]. In Figure 20 the same microcalcification cluster is visualized from two different angles using a surface rendering technique. The validity of the proposed approach is illustrated in this example for a cluster of microcalcifications. Microcalcifications of different sizes, shapes and

318

Gero Peters et al.

Fig. 18. Selected region of a DBT projected view (left) and the corresponding 2D fuzzy particle map for microcalcification detection (right). Tomographic projection data are provided by courtesy of Dr. D. Kopans, Massachusetts General Hospital, Boston, MA, USA

Fig. 19. Selected region of a reconstructed slice (left), the corresponding 3D fuzzy particles for microcalcification detection in the same slice (middle), and corresponding 3D particles resulting from defuzzification of the 3D fuzzy particles by applying a threshold (right)


319

local contrast are detected. Since a clinical database providing ground truth at particle level is hard to come by, a visual evaluation is today the only means to verify our results. As 3D DBT datasets become increasingly numerous, a validation for detection of clusters of microcalcifications on a clinical database should be envisioned.

Fig. 20. The same selected region as in Figure 19 is here visualized from two different angles using a surface rendering technique

Fig. 21. Selected region of a DBT projected view (left) and the corresponding 2D fuzzy particle map for mass detection (right). Tomographic projection data are provided by courtesy of Dr. D. Kopans, Massachusetts General Hospital, Boston, MA, USA

The performance of the presented algorithm was evaluated in 3D space for mass detection (Figures 21 and 22). The aggregation of the information over the complete set of projected views results in 3D fuzzy particles that are evaluated to obtain the final decision. The data used to validate the algorithm consists of 11 DBT cases provided by Massachusetts General Hospital (MGH), where each case contains a medio-lateral oblique (MLO) DBT acquisition of a single breast. The database used to evaluate the sensitivity of the algorithm consists of 7 cases containing malignant mass lesions: 3 invasive ductal carcinoma, 3 invasive carcinoma and 1 metastatic lymph node. The False Positive Rate (FPR) was established by use of a database comprised of 4 cases with

320

Gero Peters et al.

Fig. 22. Selected region of a reconstructed slice (left), the corresponding 3D fuzzy particles for mass detection in the same slice (middle), and corresponding 3D particles resulting from defuzzification of the 3D fuzzy particles by applying a threshold (right)

no malignant masses. These malignant cases do however contain microcalcifications. Applying the algorithm to this database resulted in a sensitivity of 0.86 (6 out of 7). Tests performed using the data free of masses indicated a FPR of 3.5 false positives per case (14 FPs in 4 cases). Here, a false positive is considered to be a connected 3D object that is not corresponding to an actual mass. The only missed malignant mass lesion was the metastatic lymph node, whose properties vary significantly from those of the other malignant masses.

9 Conclusion We have proposed a novel approach to detect microcalcifications and masses in DBT datasets. Our approach exhibits numerous advantages. Working directly


321

on the DBT projected views enables us to work independently of the reconstruction algorithm used to generate the 3D images. In addition, the processing time is expected to be significantly reduced compared to the application of similar operators on reconstructed slices, since they are generally much more numerous than the projected views, while the required 3D processing in the presented approach is sparse. We have introduced a fuzzy description of the candidate particles to account for the ambiguities in the image data. Another key advantage of combining fuzzy techniques with a detection applied directly on the projected views is that information about each potential finding can be preserved until the aggregation step. A final defuzzification of the aggregated particles allows the reduction of false positives that were accepted at a high level during the initial detection step in the projected views. The experiments presented here for microcalcification detection are quite promising as far as a visual verification is concerned. Nevertheless, an investigation on a clinical database is needed for comparing detection results to state-of-the-art 2D detection algorithms. For the evaluation of mass detection performance, the limited number of available cases obliged us to accept using the same database for training as well as for testing the algorithm. Furthermore, the FPR assessment was not conducted on normal cases. These shortcomings should be easily overcome in the future, when clinical data becomes more readily available and will hopefully confirm our preliminary results.

References 1. A. H. Andersen and A. C. Kak. Simultaneous Algebraic Reconstruction Technique (SART): A Superior Implementation of the ART Algorithm. Ultrasonic Imaging, 6(1):81–94, 1984. 2. I. Bloch. Information Combination Operators for Data Fusion: A Comparative Review with Classification. IEEE Transactions on Systems, Man, and Cybernetics, 26(1):52–67, 1996. 3. S. Bothorel. Analyse d’image par arbre de décision floue - Application a ` la classification s´ eminologique des amas de microcalcifications en mammographie numérique. PhD thesis, Université Paris 6, 1996. 4. S. Bothorel, B. Bouchon-Meunier, and S. Muller. A Fuzzy Logic Based Approach for Seminological Analysis of Microcalcifications in Mammographic Images. International Journal for Intelligent Systems, 12:819–848, 1997. 5. H.-P. Chan, J. Wei, B. Sahiner, E. A. Rafferty, T. Wu, M. A. Roubidoux, R. H. Moore, D. B. Kopans, L. M. Hadjiiski, and M. A. Helvie. Computerized Detection of Masses on Digital Tomosynthesis Mammograms - A Preliminary Study. In Springer, editor, Proceedings of the 7th International Workshop on Digital Mammography, Chapel Hill, NC, 2004. 6. J.-L. Chen and J.-Y. Chang. Fuzzy Perceptron Neural Networks for Classifiers with Numerical Data and Linguistic Rules as Input. IEEE Transactions on Fuzzy Systems, 8(6):730–745, 2000. 7. H. D. Cheng, Y. M. Lui, and R. I. Freimanis. A Novel Approach to Microcalcification Detection Using Fuzzy Logic Technique. IEEE Transactions on Medical Imaging, 17(3):442–450, 1998.

322

Gero Peters et al.

8. R. Holland, M. Mravunac, J. H. Hendriks, and B. V. Bekker. So-called Interval Cancers of the Breast. Pathologic and Radiologic Analysis of Sixty-Four Cases. Cancer, 49:2527–2533, 1982. 9. J. T. Dobbins III and D. J. Godfrey. Digital X-Ray Tomosynthesis : Current State of the Art and Clinical Potential. Physics in Medicine and Biology, 48:R65–R106, 2003. 10. K. Kanzaki. The Use of Morphology and Fuzzy Set Theory in FLIR Target Segmentation and Classification. PhD thesis, Polytechnic University of New York, 1992. 11. N. Karssemeijer and G. M. te Brake. Detection of Stellate Distortions in Mammograms. IEEE Transactions on Medical Imaging, 15(5), 1996. 12. K. Kerlikowske, D. Grady, S. M. Rubin, C. Sandrock, and V. L. Ernster. Efficacy of Screening Mammography. A Meta-Analysis. The Journal of the American Medical Association, 273(2):149–154, 1995. 13. R. A. Kirsch. Resynthesis of Biological Images from Tree Structured Decomposition Data. In IFIP Working Conference on Computer Graphic Languages, pages 1–19, 1972. 14. H.-M. Lee and W.-T. Wang. A Neural Network Architecture for Classification of Fuzzy Inputs. Fuzzy Sets and Systems, 63(2):159–173, 1994. 15. S. Muller. Full-Field Digital Mammography Designed as a Complete System. European Journal of Radiology, 31(1):25–34, 1999. 16. T. Netsch and H.-O. Peitgen. Scale-Space Signatures for the Detection of Clustered Microcalcifications in Digital Mammograms. IEEE Transactions on Medical Imaging, 18(9):774–786, 1999. 17. N. Pandey, Z. Salcic, and J. Sivaswamy. Fuzzy Logic Based Microcalcification Detection. In Proceeding of the IEEE Workshop on Neural Networks for Signal Processing, volume 2, pages 662–671, 2000. 18. E. D. Pisano, C. Gatsonis, E. Hendrick, M. Yaffe, J. K. Baum, S. Acharyya, E. F. Conant, L. L. Fajardo, L. Bassett, C. D’Orsi, R. Jong, M. Rebner, and Digital Mammographic Imaging Screening Trial (DMIST) Investigators Group. Diagnostic Performance of Digital versus Film Mammography for Breast-Cancer Screening. New England Journal of Medicine, 353(17):1773–1783, 2005. 19. I. Reiser, R. M. Nishikawa, M. L. Giger, D. B. Kopans, E. A. Rafferty, R. H. Moore, and T. Wu. A Reconstruction-Independent Method for Computerized Detection of Mammographic Masses in Tomosynthesis Images. In Springer, editor, Proceedings of the 7th International Workshop on Digital Mammography, Chapel Hill, NC, 2004. 20. A. Rick, , S. Muller, S. Bothorel, and M. Grimaud. Quantitative Modeling of Microcalcification Detection in Digital Mammography. In MICCAI’99, Second International Conference, Cambridge, England, pages 32–41, September 1999. 21. A. Rick. Répresentation de la variabilité dans le traitement d’images flou. PhD thesis, Université Paris 6, 1999. 22. A. Rick, S. Bothorel, B. Bouchon-Meunier, and S. Muller. Fuzzy Techniques in Mammographic Image Processing. In E. Kerre and M. Nachtegael, editors, Studies in Fuzziness and Softcomputing: Fuzzy Techniques in Image Processing. Springer, New-York, 2000. 23. P. C. Smits, M. Mari, A. Teschioni, S. Dellapiane, and F. Fontano. Application of Fuzzy Methods to Segmentation of Medical Images. Proceeding of IPMU, (9), 1995.


323

24. G. M. te Brake and N. Karssemeijer. Single and Multiscale Detection of Masses in Digital Mammograms. IEEE Transactions on Medical Imaging, 18(7), 1999. 25. M. Unser. Fast Gabor-Like Windowed Fourier and Continuous Wavelet Transforms. IEEE Signal Processing Letters, 1(5):76–79, 1994. 26. M. Unser, A. Aldouri, and M. Eden. On the Asymptotic Convergence of βSpline Wavelets to Gabor Functions. IEEE Transactions on Information Theory, 38(2):864–872, 1992. 27. T. Wu, A. Steward, M. Stanton, T. McCauley, W. Phillips, D. B. Kopans, R. H. Moore, J. W. Eberhard, B. Opsahl-Ong, L. Niklason, and M. B. Williams. Tomographic Mammography Using a Limited Number of Low-Dose Cone-Beam Projection Images. Medical Physics, 30(3):365–380, 2003. 28. L. A. Zadeh. Fuzzy Sets. Information and Control, 8(3):338–353, 1965.

Chapter 12 Intelligent Biometric Information Fusion using Support Vector Machine Richa Singh, Mayank Vatsa and Afzel Noore West Virginia University {richas, mayankv, noore}@csee.wvu.edu

1 Introduction Biometrics is the process of verifying or identifying an individual using the person’s physiological and behavioral characteristics such as face, fingerprint, iris, signature, and gait. The main advantage of using biometrics for recognition is that it is unique for every individual and cannot be misplaced or forgotten. However, biometric systems that use a single biometric trait have to contend with noisy data, restricted degrees of freedom, failure to-enroll problems, spoof attacks, and unacceptable error rates. Various researchers have suggested that no single biometric modality can provide the protection required for high security applications [1] - [4]. To alleviate this problem and enhance the performance of a biometric system, information from different biometric sources are combined and such systems are known as multimodal biometric systems [2]. Different forms of biometric information fusion are: single biometric - multiple representation, single biometric - multiple matchers, and multiple biometrics [2]. Single biometric - multiple representation involves different representation of a single biometrics with different classifiers for each representation. The features extracted from different classifiers can be combined or the match scores of the two representations are combined to get the matching results. Single biometric - multiple matchers involves different matching algorithms and the matching scores are combined to obtain the results. Multiple biometrics refers to the information fusion from two different biometric sources. The information can be two images, or the features extracted from the two sources, or their matching scores. Thus biometric information can be combined at different levels such as, image level fusion, feature level fusion, match score level fusion, expert or decision level fusion, and rank level fusion [2]. In this chapter, the main focus is on single biometric - multiple matchers at feature level fusion, match score level fusion, and expert or decision level fusion. Several multimodal biometric algorithms have been proposed by different researchers to fuse information extracted from a single biometric source. Han R. Singh et al.: Intelligent Biometric Information Fusion using Support Vector Machine, StudFuzz 210, 325–349 (2007) c Springer-Verlag Berlin Heidelberg 2007 www.springerlink.com

326

R. Singh et al.

and Bhanu [5] proposed a feature fusion algorithm to combine statistical gait features extracted from training silhouette sequences. Lu et al. [6] used the sum rule and RBF based integration strategies to combine information extracted from face image using PCA, ICA and LDA. Researchers have also proposed several other algorithms [2, 3, 7, 8] for fusion at match score level and decision level. In [1], Kittler proposed a set of matching score fusion rules to combine the classifier which includes majority voting, sum rule, and product rule. Ross [9], Kumar [10], and Singh [11] proposed the feature level fusion algorithm, Chang [12] and Singh [11] proposed algorithm for image level fusion. Several other researchers proposed algorithms based on Support Vector Machine (SVM) [14], fuzzy clustering [15] and radial basis neural network [6, 16] to fuse the information at different levels. Many researchers claim that when two or more biometric information is combined, recognition accuracy increases [1] - [23]. However combining this does not always yield higher performance [24]. If there are two biometric features, one of which gives an accuracy of 99% and the other gives an accuracy of 85%, then it is not necessary that the performance of the multimodal system will always be higher than the two individual biometrics. The performance in this case highly depends on the multimodal algorithm applied to fuse the information. Theoretically and experimentally, it has been found that combining a weak biometric feature with a strong one does not necessarily improve the performance [24, 25]. As in the classifier fusion, several weak classifiers can be combined to form a strong classifier, similarly for biometrics the performance may improve if two weak features are combined. Considering all these factors, learning based multimodal biometric algorithm should be designed which combines multiple biometric information intelligently. In this chapter, we first describe the amplitude and phase feature based face recognition algorithm. Using the phase and amplitude features, three fusion algorithms are presented which use SVM to intelligently fuse the information at three different levels. The three levels of fusion are, feature level fusion, match score level fusion, and expert or decision level fusion. Section 2 briefly describes the amplitude and phase based face recognition algorithm. Section 3 provides an overview of SVM and Section 4 gives the details of the fusion algorithms based on SVM. Section 5 shows the experimental evaluation and comparison with standard algorithms followed by summary of the chapter.

2 Face recognition using phase and amplitude features In this section we describe the face recognition algorithm [28] which is based on the amplitude and the phase information extracted from a face image. Among the texture and shape of a face, texture provides more prominent information for recognition [31]. So, the proposed face recognition algorithm

Intelligent Biometric Information Fusion

327

extracts only the textural features for recognition. The algorithm is described as follows: Let the size of the face image F (x, y) be N × N . The face image is transformed into its log polar form F (r, θ): 2 3 2 3 2 3 2 3 N 2πr N 2πr F (r, θ) = F + θ cos , − θ sin (1) 2 s 2 s 4 5 where r = 0, ..., s − 1, θ = 0, ..., N2 − 1 and s is the factor by which the 4 5 image is sampled from 0◦ to 360◦ to produce its equivalent s × N2 polar form. The polar face image is then expressed into the 2D Fourier domain, 2πj 2πr ρ+ ϕ (2) F (r, θj )exp −i F (ρ, ϕ) = µ ν r j where θj = 2πj ν , 0 ≤ r ≤ µ, 0 ≤ j ≤ ν, µ and ν are the radial and the angular frequency resolutions respectively. The 2D Fourier transform of a face image is convolved with the 2D Fourier transform of the log polar Gabor wavelet G(ρ, ϕ). Log polar Gabor wavelet is a form of log Gabor wavelet [29] which is based on polar coordinates and the dependency of directional independent variance (σ) on the polar coordinate is realized by a logarithmic scale. Thus the functional form of 2D log polar Gabor filter can be represented as, 6 7 (3) Gr0 ,θ0 (r, θ) = exp −2π 2 σ 2 (ln(r) − ln(r0 ))2 + (ln(r)sin(θ − θ0 ))2 and the position of the filter in the Fourier domain is defined by √ 2π r00 = 2, r0i = 2i ∗ r00 , θ0i = i ∗ Nθ

(4)

where r00 is the smallest possible frequency, N is the number of filters on the unit circle, and at index L, σL and sL are further defined by , ln 2 1 (5) σL = π 2 ln(r )πsin 0

Nθ

2ln(r0 )πsin sL =

ln 2

π Nθ

,

ln 2 2

(6)

The inverse Fourier Transform (IFT) of the convolved face image is computed and the output Fg (r, θ) is a complex valued matrix containing the amplitude and the phase information:   F (ρ, ϕ) ∗ G(ρ, ϕ) (7) Fg (r, θ) = IF T  r

j

328

R. Singh et al.

The amplitude and the phase features shown in Fig. 1 are computed from the matrix Fg (r, θ) using Equation 8 and 9 respectively. 8 2 2 (8) A(r, θ) = (Re Fg (r, θ)) + (Im Fg (r, θ))

P (r, θ) = tan−1

Im Fg (r, θ) Re Fg (r, θ)

(9)

Fig. 1. Amplitude and phase features of an input face image

Amplitude features are encoded and matched using the steps described below: a) The two amplitude features to be matched, A1 and A2 are divided into z number of frames, each of size k × l. b) The correlation Vi between the two corresponding frames is computed from Equation 10. Using the frame matching threshold λ1 , which is the intermediate matching score for the frames, MsA is calculated. The final amplitude matching score M A is obtained by dividing the intermediate matching score by the number of frames. Vi = MsA =

A1i ⊕ A2i k∗l

MsA + 1, if MsA , if

MA =

MsA z

(10) Vi ≥ λ1 Vi < λ1

(11)

(12)

c) A match occurs if M A > λ2 , where λ2 is the amplitude matching threshold for the face template. Phase features are quantized using Equation 13 to generate the phase template BP .


 [1,    [0, Bp [r, θ] = [0,    [1,

1] 1] 0] 0]

if if if if

00 < Re(P (r, θ)) ≤ 900 < Re(P (r, θ)) ≤ 1800 < Re(P (r, θ)) ≤ 2700 < Re(P (r, θ)) ≤

900 1800 2700 3600

329

(13)

To match the two phase templates, a) Phase templates are divided into m frames each of size p x q. b) Corresponding frames from the two phase templates are matched using hamming distance as shown in Equation 14, 1 Bi ⊗ Bi2 Di = (14) p∗q where Bi1 and Bi2 are the ith frames of the two templates, and Di is the value of the distance between them. c) The phase matching score M P is calculated using Equations 15 and 16, P Ms + 1, if Di ≤ η1 P (15) Ms = MsP , if Di > η1 MP = 1 −

MsP m

(16)

where η1 is the frame matching threshold, MsP is the intermediate matching score for frames and m is the number of frames. d)A match occurs if the matching score M P < η2 , where η2 is the phase matching threshold for the face template. The amplitude and phase feature sets, matching scores, and decision rules proposed in this section are used for the feature fusion, match score fusion, and the expert fusion schemes respectively. The proposed face recognition algorithm handles the face images with variation in illumination, expressions, and an orientation upto 100 . Since the focus of this paper is on information fusion and not on face recognition, the experimental results presented in this chapter do not provide much details about the performance of the face algorithm on various other challenges such as rotation, accessories, and pose variation. However, we have addressed some of the challenges in our other papers in which we have proposed enhancements in the face recognition algorithm to handle the presence of single training images [28], profile images [30], and disguises [31]. Another modification of the proposed approach is discussed in [32] where the facial features are matched locally. Weights are assigned to different local features such as nose, eyes, and mouth to handle the deformation in features.

3 Overview of support vector machine Support Vector Machine, proposed by [33], is a powerful methodology for solving problems in nonlinear classification, function estimation and density

330

R. Singh et al.

estimation. The concept of decision planes that define decision boundaries is the basis of SVM. A plane that separates a set of objects belonging to different classes is called the decision plane. Figure 2 shows a simple example of a linear classifier which separates the objects belonging to two different classes, Class1 and Class2 , using a decision plane. A separating line defines the boundary that classifies the two objects on the two sides; on one side all objects belong to Class1 and on the other side all objects belongs to Class2 . Any new object (test sample) lying on the right of the boundary is classified as Class2 , otherwise it is classified as Class1 .

Class1 Class2 Test Sample

Fig. 2. A classifier that linearly separates a set of objects into distinct groups

In general, the classification tasks are not simple, and often more complex structures are required to make an optimal separation, i.e., correctly classify new objects (test samples) on the basis of training samples. Support Vector Machines are particularly suited to handle such tasks. Figure 3 shows an example of complex classification task in which the decision boundary is a complex curve, rather than a straight line.


Fig. 3. A classifier that non-linearly separates a set of objects into distinct groups

SVM starts from the goal of separating the data with a hyperplane and extends this to non-linear decision boundaries. Figure 4 shows the basic idea behind Support Vector Machines which demonstrates that using mathematical functions, the original objects are mapped into a new feature space where it is easy to separate the mapped objects. This process is known as mapping.


331


Input Space

Feature Space

Fig. 4. Mapping non-linearly separable data from the input space to a high dimensional feature space and finding the decision boundary

SVM is thus a classifier that performs classification tasks by constructing hyperplanes in a multidimensional space and separates the data points into different classes. To construct an optimal hyperplane, SVM uses an iterative training algorithm to maximize the margin between two classes. However, some researchers have shown that the margin maximization does not always lead to the minimum classification error [34]. In biometric applications, the main aim of training the classifier is to reduce classification error. To achieve this, we optimize the SVM parameters by minimizing an empirical estimate of the error through a validation set. The remaining section describes the mathematical formulation of non-linear SVM which is more useful in biometrics applications. Sometimes the training data points are not clearly separable and they are characterized as fuzzy separable data. If we think from the perspective of biometrics, fuzzy data are more common and SVM which deals with such data can provide probabilistic recommendation to the user. A two layer network can provide an adequate flexibility in the classification of fuzzy separable data. This network can be implemented by inserting a well designed nonlinear hidden-layer between the input and the output. The original linearly nonseparable data can be mapped into a new feature space, i.e., a new space represented by hidden nodes such that the mapped patterns become linearly separable. Let ϕi (x) be a nonlinear function representing hidden nodes, where i = 1, ..., N is a generalized decision function and can be written as, f (x) =

N

wi ϕi (x) + b = W ϕ(x) ¯ +b

(17)

i=1

where ϕ(x) ¯ = [ϕ1 (x), ϕ2 (x), ..., ϕN (x)]T . To obtain a nonlinear decision boundary which thus enhances the discrimination power, we can rewrite the above equation as,

332

R. Singh et al.

f (x) =

N

yi αi K(x, xi ) + b

(18)

i=1

where K(x, xi ) is a nonlinear kernel which enhances the discrimination power and αi is the Lagrangian multiplier. The basic idea behind nonlinear SVM is to use a kernel function K(x, xi ) to map the input space to the feature space so that the mapped data becomes linearly separable. One example of such kernel is the RBF kernel x − xi 2 (19) K(x, xi ) = exp 2 σ2 The Lagrange multipliers αi are determined by maximizing L(α) to N

αi yi = 0

i=1

and 0 ≤ α ≤ C, i = 1, ..., N where N

1 αi − (αi αj yi yj K(xi , xj )) L(α) = 2 i=1 j=1 i=1 N

N

(20)

and C is the factor used to control the violation of safety margin rule [33]. In practice, if the noise is very high in the training data, finding a separating hyperplane is not a trivial task. Moreover, the setting of the error penalty in the objective function is based on trial and error, which requires additional time consuming training. One alternative and intuitive approach to solve this problem is the use of ν-SVM of a soft margin variant of the optimal hyperplane which uses the ν-parameterization [35] and [36]. Here the parameter C is replaced by another parameter ν [0, 1] which is the lower bound on the fraction of support vectors and upper bound on the number of fraction of margin errors. Further, a cost-effective extension of ν-SVM is proposed by [37], and is termed as dual ν-SVM. Dual ν-SVM is designed to reduce the complexity of error penalty selection and improve the performance in terms of computation and classification. Davenport [38] shows that 2ν-SVM is an attractive alternative to SVM and offers much more natural setting for parameter selection which is a critical issue in practical applications. Dual ν-SVM (2ν-SVM) is briefly described as follows: Let {xi , yi } be a set of N data vectors with xi d , yi (+1, −1), and i = 1, ..., N . xi is the ith data vector that belongs to a binary class yi . The objective of training 2ν-SVM is to find the hyperplane that separates the two classes with the widest margins, i.e., w(x) + b = 0 subject to,

(21)


333

yi (w ϕ(x) + b) ≥ (ρ − ψi ), ψi ≥ 0

(22)

1 w2 − Ci (νρ − ψi ) 2 i

(23)

to minimize,

where ρ is the position of the margin and ν is the error parameter. ϕ(x) is the mapping function used to map the data space to the feature space, and provide generalization for the decision function that may not be a linear function of the training data. Ci (νρ − ψi ) is the cost of errors, w is the normal vector, b is the bias, and ψi is the slack variable for classification errors. Here, the slack variables are introduced to handle classes which cannot be separated by a hyperplane. ν is the error parameter that can be calculated using ν+ and ν− , which are the error parameters for training the positive and negative classes respectively. ν=

2ν+ ν− , 0 < ν+ < 1 and 0 < ν− < 1 ν+ + ν−

Error penalty Ci is calculated as, C+ , if C= C− , if

(24)

yi = +1 yi = −1

(25)

−1 ν+ C+ = n+ 1 + ν−

(26)

−1 ν− C− = n− 1 + ν+

(27)

where,

and n+ and n− are the number of training points for the positive and the negative classes respectively. 2ν-SVM training can be formulated as,     1 max(αi ) − αi αj yi yj K(xi , xj ) (28)   2 i,j where, 0 ≤ αi ≤ Ci i

i

αi yi = 0 αi ≥ ν

(29)

334

R. Singh et al.

i, j 1, ..., N and kernel function is K (xi , xj ) = ϕ(xi )ϕ(xj )

(30)

The kernel function K(xi , xj ) is the RBF kernel as described in Equation 19. To train the 2ν-SVM, we have used iterative decomposition training based optimization algorithm originally proposed by Chew et al. [37]. This optimization algorithm can be seen as pairwise decomposition method which breaks the problem down to a two decision variable problem and solves the subproblem analytically. More details about the optimization algorithm can be found in [37]. If n is the set of data vectors, then the complexity without optimization is O(n2 ) and with optimization is O(n) [37]. Applying this optimization algorithm leads to reduction in the computational complexity.

4 Information fusion in face biometrics As described earlier, there are five levels of fusion in multimodal biometrics. In this chapter, we focus on the feature level fusion, match score level fusion and expert fusion using support vector machine. For simplification and better understanding, the fusion algorithms are explained using the two feature sets of face biometrics. As explained in Section 2, these feature sets are the amplitude and phase features extracted from face images using 2D log polar Gabor wavelet [28, 30, 31, 32]. However, we can use the same fundamental concept for two or more feature sets for any other multimodal scenario. 4.1 Feature level fusion using support vector machine Feature fusion in biometrics has been addressed by several researchers [9, 11] in the literature. In [9], feature fusion is performed by concatenating two feature vectors. Singh et al. [11] have fused the eigen vectors of visible and infrared face image using genetic algorithms. Here we propose a learning based algorithm using 2ν-SVM to fuse the two feature sets. If amplitude feature set A and phase feature set P are given, our objective is to fuse these two feature sets. Here, the dimension of the two feature sets are same [28] and there is no need to normalize the two feature sets. However, if the two feature sets are different in size and are extracted using different feature extractors, then they should be normalized before fusion. The details of normalization in multimodal biometrics are described in [39]. There are two stages in the feature fusion algorithm. In the first stage we have the amplitude A and phase P which are used for training. These training features are labeled as good and bad, and using these labeled examples, 2νSVM is trained to classify the input feature points as good or bad (1, -1). The training process is defined as follows:


335

1. Training features are decomposed to l levels by the Discrete Wavelet Transform (DWT). All the 4l subbands of each feature set are used for training. 2. Activity level of the wavelet coefficients for each subband is computed over a small window by treating each coefficient separately. Activity level is computed using the window based weighted activity i.e., the average value of wavelet coefficients are computed for the window under consideration. 3. The 2ν-SVM is trained to determine the quality of every wavelet coefficient. For training and determining the quality, activity levels computed in the previous step are given as input to the 2ν-SVM. 4. At any position (x, y), the output O(x, y) of the learning algorithm is either 1 or -1. The value of O(x, y) is 1 if the features of A are classified as “good”, and the classification distance of A is greater than the classification distance of P , otherwise the output value is -1. In other words, if output is 1, the quality of wavelet coefficient for A is good and if it is -1, the quality of wavelet coefficient for P is good. In the training process, 2ν-SVM is trained to determine whether the coefficient from feature set A or feature set P should be used for a particular position in the composite feature set. Second stage is the feature fusion of the test feature sets. These test feature sets are fused using the trained 2ν-SVM output O(x, y). The feature selection can be represented by Equation 31. f eature(A(x, y)), if O(x, y) > 0 F used(x, y) = (31) f eature(P (x, y)), if O(x, y) < 0

Hi = MsF used =

F used1i ⊕ F used2i k×l

MsF used + 1, if MsF used , if

M F used =

MsF used m

(32) Hi ≥ β1 Hi < β1

(33)

(34)

Further, to match the two fused feature vectors, F used1 and F used2 , the correlation based matching technique is applied. First, the features are divided into m frames where the size of each frame is k×l. The correlation, Hi between the two corresponding frames is computed using Equation 32 where ⊕ denotes the correlation. Using an empirical frame matching threshold β1 , the intermediate matching score for the frames is obtained. The final matching score M F used is calculated by dividing the intermediate matching score MsF used by the number of frames. A person is said to be matched if M F used > β2 where β2 is the empirical fused vector matching threshold. Figure 5 illustrates the concept of feature level fusion.

336

R. Singh et al. Amplitude Features

SVM based Feature Fusion

Face Image

Fused Feature Vector

Phase Features

Fig. 5. Feature level fusion using SVM

4.2 Match score level fusion using SVM It has been suggested that the fusion of match scores of two or more classifiers gives better performance over a single classifier [1, 2]. In general, match score fusion is performed using sum rule, product rule or other statistical rules. Recently in [40], a SVM based match score fusion algorithm is proposed for fusing the match scores of fingerprint and signature. In this section, we describe a SVM learning based match score fusion algorithm. Let the training set be Z = (xi , yi ) where i = 1, ..., N , N is the number of multimodal match scores used for training and yi (1, −1), where 1 represents the genuine class and -1 represents the impostor class. 2ν-SVM is trained using these labeled training data. The training data is mapped in a higher dimension feature space such that Z → ϕ(Z) where ϕ(.) is the mapping function. The optimal hyperplane which separates the data into two different classes in the higher dimensional feature space can be obtained as the solution of Equations 21 to 23 in Section 3. In the testing phase, the fused score ft of a multimodal test pattern xt is defined as, ft = f (xt ) = wϕ(xt ) + b

(35)

The solution of this equation is the signed distance of xt from the separating hyperplane given by the SVM. Finally, an accept or reject decision is made on the test pattern xt as follows: accept, if output of SV M > 0 (36) result(xt ) = reject, if output of SV M < 0 Fig. 6 represents the steps involved in the match score level fusion using SVM. The match score fusion algorithm described above has a global training and decision scheme. However, we can also design the local and/or adaptive learning and decision schemes based on the above concept.


337

Amplitude Features M

A

SVM based Match Score Fusion

Face Image M

Fused Match Score

p

Phase Features

Fig. 6. Match score level fusion using SVM

4.3 Expert fusion In multiple expert system, there are n experts and one administrator. Experts provide their opinion about the claims of a candidate and the administrator makes the decision based on the opinions. In expert fusion, administrator makes the decision based on the expert’s opinion and confidence level. For example, if one expert is 30% confident that the person should be rejected using face biometrics and another expert is 99% confident that the person should be accepted using iris biometrics, then the administrator decides to accept the person. In another example of expert fusion where a genuine individual is suffering from soar throat and the voice expert says with 15% confidence that the person should be rejected and face biometric expert with 40% confidence says that the person should be accepted. In this case also, the administrator may accept the candidate. In cases when an impostor tries to impersonate someone’s identity using face, the face expert may accept the candidate with 60% confidence but the fingerprint expert may reject with 95% confidence. In such cases, the administrator rejects the candidate because the fingerprint expert is more confident of the decision. Based on the above three examples, we can come up with three challenges for expert fusion. • Who are the experts and how reliable are the experts? • How does an administrator make the decision based on the opinions rendered by different experts? • How do we normalize the data? In [15, 17, 41, 42], detailed experimental evaluations of multiple expert fusion algorithms are presented. Most of the research in expert fusion is based on statistical evaluation or Bayesian approach. Here we present an approach of expert fusion based on Support Vector Machine. For generality and simplification we assume that we have two different feature sets and two experts which operate on each of the feature sets. Let x1 and x2 be the two feature sets used by experts e1 and e2 respectively. Let g1 and g2 be the confidence of experts e1 and e2 ,

338

R. Singh et al.

1 1 g1 = √ , g2 = √ SR1 SR2

(37)

where SR1 and SR2 are the success rates defined as, SR = 1 −

FA + FR N

(38)

where, F A is the number of false acceptance cases, F R is the number of false rejection cases and N is the number of attempts. In the training phase, SVM for both the experts are trained using the labeled positive and negative classes as described in Section 2. g1 and g2 are calculated using both the experts training data and Equation 37. In the recognition phase, two test features are given to experts e1 and e2 respectively and the experts then try to find the decision boundary by minimizing the expression in Equation 39,  1 min  2

i=1,2

gi2 wi2 2 −

 Cj (νρ − ϕj )

(39)

j

subject to, yj (wi ϕ(x) + b) ≥ ρ − ϕj , φj ≥ 0, ∀ j and i = 1, 2

(40)

If the margin from the decision boundary for expert e1 is greater than e2 , then the output of 2ν-SVM is +1 and the administrator accepts the decision of e1 , otherwise the output is -1, and administrator accepts the decision of e2 . Here, an adaptive scheme for gi can also be performed based on the results so that the system can cope up with the changes at testing time. Figure 7 illustrates the concept of the expert fusion algorithm. Amplitude Features M

Face Image

M

A

p

Decision A Accept/Reject

Decision P Accept/Reject

Phase Features

Fig. 7. Expert fusion using SVM

SVM based Expert Fusion

Decision


339

5 Experimental results To study the performance of various levels of fusion, experiments are performed using two face databases: • Frontal face images from the colored FERET database [43]. From this database we have chosen 3000 frontal face images (variations from 0 to 10 degree in pose) from 600 individuals. • Frontal face images from the Equinox Corporation face database [44]. This database contains face images of 90 individuals in different light spectrums. We have used only the visible spectrum face image for the experiments. For training the 2ν-SVM, we have randomly chosen 2000 face images of 690 individuals (combining both the face databases) with the constraint that for every user there exists at least two images. These images are then hand labeled as genuine and impostor. The experimental results are divided into five subsections. The first subsection shows the results of amplitude and phase feature based face recognition on the two databases. Subsequent subsections show the results of the three proposed fusion algorithms and the final subsection compares the performance of the three fusion algorithms. 5.1 Experimental results of face recognition Face is first detected from the image using the triangle based face detection algorithm [45] and then the performance of the face recognition algorithm is computed for both the databases. Figure 8 shows the Receiver-Operating Characteristic (ROC) curves for the FERET and the Equinox face database respectively when amplitude and phase features are used for recognition. These plots show that the performance of both the phase and amplitude features are comparable and they outperform the standard PCA and LDA based face recognition algorithms [46]. In specific applications such as law enforcement, there may exist only one image per person for training the face recognition system. Most of the face recognition algorithms may not work well in such a scenario because they require large number of example images per class to learn the intraclass variability and the interclass similarity. We tested the amplitude and phase based face recognition algorithm for such worst case scenario. Figure 9 shows the identification accuracy of face recognition when the number of training images is decreased from three to one. With three training images, the identification accuracy of amplitude and phase features is nearly 96% for Equinox database and 92% for the FERET database. However, as the number of training images decrease, the accuracy for both the features also decreases and is around 95% using single training image for the Equinox database and around 90% for the FERET database. For the fusion experiments we have used only one face image per individual for training and the remaining images are used for testing.

340

R. Singh et al.

(a) FERET face database

(b) Equinox face database

Fig. 8. ROC plot for comparing different face recognition algorithms


341

97 96 95

Accuracy(%)

94

Amplitude − FERET Phase − FERET Amplitude − Equinox Phase − Equinox

93 92 91 90 89 88 1

2 Number of Training Image

3

Fig. 9. ROC plot on varying the number of training images

5.2 Experimental results of feature fusion algorithm Using the labeled training data, first the 2ν-SVM is trained for feature fusion. Then for each image in the recognition process (both the training and testing), the amplitude and the phase features are fused using the trained 2ν-SVM. For matching the two fused feature vectors, correlation based matching technique is used as described in Section 2.1. Figure 10 shows the ROC plot for feature fusion algorithm on FERET and Equinox face database. The Equal Error Rate (EER) of feature fusion is 2.93% on FERET database and 3.90% on Equinox database. The ROC plot also shows that the performance of proposed learning based feature fusion algorithm is better than the existing feature fusion algorithm [9]. 5.3 Experimental results of match score fusion Similar to feature fusion, the labeled training data is used to train the 2νSVM for match score fusion. In the recognition process, the fused match scores for both the training and testing data are generated based on the scores of amplitude and phase features and trained 2ν-SVM. Then the decision rule described in Section 4.2 is used for recognition. Figure 11 shows the ROC plot of the match score fusion on the two databases. This also shows the comparison with sum rule based match score fusion. The results show that on

342

R. Singh et al.

20 18

Fusion−[20] on FERET Fusion−[20] on Equinox SVM Feature Fusion on FERET SVM Feature Fusion on Equinox

False Rejection Rate(%)

16 14 12 10 8 6 4 2 0 0

2

4

6

8 10 12 False Accept Rate(%)

14

16

18

20

Fig. 10. ROC plot for feature fusion

the two databases, SVM based learning algorithm outperforms other match score fusion algorithm [20]. 5.4 Experimental results of expert fusion Labeled expert decisions of amplitude and phase features are used to train the 2ν-SVM based expert fusion algorithm. Using this trained fusion algorithm, decisions are made on the test cases. Figure 12 shows the ROC of the expert fusion on the two databases. This plot also compares the results with min/max rule based expert fusion [1]. The results show that the expert fusion using SVM performs better than the min/max rule. 5.5 Performance comparison of SVM learning based fusion algorithms In this subsection, we compare the three proposed learning based fusion algorithms with the algorithms presented in [15] and [40]. Figure 13 shows the ROC plot comparing the three proposed fusion algorithms. These ROC plots show that the feature level fusion performs best with EER of 2.93% on FERET and 3.90% on Equinox face database followed by match score fusion with EER of 6.91% on FERET and 6.35% on Equinox, and expert fusion with EER of 8.62% on FERET and 8.33% on Equinox face database. Table 1 shows the


20 18

Sum Rule on FERET Sum Rule on Equinox SVM Match Score Fusion on FERET SVM Match Score Fusion on Equinox


16 14 12 10 8 6 4 2 0 0

2

4

6

8 10 12 False Accept Rate(%)

14

16

18

20

Fig. 11. ROC plot for match score fusion

30 Min/Max Rule on FERET Min/Max Rule on Equinox SVM Expert Fusion on FERET SVM Expert Fusion on Equinox


25

20

15

10

5

0 0

5

10

15 20 False Accept Rate(%)

Fig. 12. ROC plot for expert fusion

25

30

343

344

R. Singh et al.

(a) FERET face database

(b) Equinox face database

Fig. 13. Comparison of feature fusion, match score fusion, and expert fusion


345

Table 1. Comparison of proposed and existing SVM based fusion algorithms on Equinox face database

Match Score Fusion Feature Proposed SVM

Expert Fusion

Existing

Proposed SVM

Existing

Algorithm [40]

Algorithm

Algorithm [15]

Fusion

Algorithm

Classifier

2ν-SVM

2ν-SVM

C-SVM

2ν-SVM

RBF Network

Complexity

O(n)

O(n)

O(n + 1)3

O(n)

O(n)2

Equal Error Rate 3.90%

6.35%

6.51%

8.33%

10.49%

comparison among the proposed algorithms and the algorithms presented in [15] and [40]. This comparison is based on the classifier used, computational complexity and EER. The table shows that the proposed algorithms have lower computational complexity and lower equal error rate.

6 Summary Multimodal biometric fuses the biometric information from different images of a single biometric or from different biometric traits to achieve better performance and overcome the limitations of a uni-modal biometric system. This chapter presents the concept of intelligent biometric information fusion using support vector machine. Section 2 briefly describes the amplitude and phase based face recognition algorithm. The amplitude and phase features are extracted using the 2D log polar Gabor wavelet in the frequency domain. These features are then fused using SVM based fusion algorithm at different levels. SVM is a powerful tool for solving the problems which are non-linear in nature. In SVM learning, the examples contained in the input space are mapped to a high dimensional feature space. Separation of different classes is obtained by a maximum margin hyperplane in the feature space. We briefly explained the theory of SVM in Section 3 and further extended the discussion with dual ν-SVM (2ν-SVM). In Section 4, we discussed the concept of biometric information fusion. There are five levels of fusion in biometrics: data or image level fusion, feature level fusion, match score level fusion, expert fusion or decision level fusion, and rank level fusion. In this chapter we focused on feature level fusion, match score level fusion, and expert fusion based on SVM learning approach. To validate the performance of the fusion algorithms, the amplitude and phase features based face recognition algorithms are used. Experiments are performed using the FERET and the Equinox face database. The performance is compared with standard fusion algorithms. It has been shown that SVM based learning algorithms outperform other algorithms. The performance at the three fusion levels are also compared using the pro-

346

R. Singh et al.

posed SVM based algorithms. Since the feature level fusion offers more freedom to choose the good features for classification and remove the redundant ones, the performance at the feature level is better compared to the match score and the expert fusion. There are several research issues with multimodal biometrics which has to be addressed in order to develop a robust algorithm, • What biometric modalities should be fused? How does the performance vary on combining a strong biometric with a weak biometric? What is the effect if two weak biometric traits are fused? • What is the performance with different levels of fusion and why? • How does the performance depend on the feature extractors and classifiers? • What are the tradeoffs between factors such as performance, time, cost?

7 Acknowledgment The authors wish to acknowledge the Equinox Corporation for providing the face database for this research. Portions of the research in this paper use the Color FERET database of facial images collected under the FERET program. We would also like to thank the reviewers for their valuable comments and suggestions. A portion of this research is supported through a grant (Award No. 2003-RC-CX-K001) from the Office of Science and Technology, National Institute of Justice, Office of Justice Programs, United States Department of Justice.

References 1. Kittler J., Hatef M., Duin R.P., Matas J.G. (1998) On combining classifiers. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(3):226-239 2. Ross A., Jain A.K. (2003) Information fusion in biometrics. In: Pattern Recognition Letters, 24(13):2115-2125 3. Jain A.K., Ross A. (2003) Learning user-specific parameters in a multibiometric system. In: Proceedings of the International Conference on Image Processing, 57-60 4. Jain A.K., Prabhakar S., Chen S. (1999) Combining multiple matchers for a high security fingerprint verification system. In: Pattern Recognition Letters, 20(11-13):1371-1379 5. Han J., Bhanu B. (2004) Statistical feature fusion for gait-based human recognition. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2:842-847 6. Lu X., Wang Y., Jain A.K. (2003) Combining classifiers for face recognition. In: Proceedings of the IEEE International Conference on Multimedia and Expo, III:13-16


347

7. Kittler J., Matas G., Jonsson K., Sanchez M. (1997) Combining evidence in personal identity verification systems. In: Pattern Recognition Letters, 18(9):845852 8. Tax D.M.J., Breukelen M.V., Duin R.P.W., Kittler J. (2000) Combining multiple classifiers by averaging or by multiplying? In: Pattern Recognition, 33(9):14751485 9. Ross A., Govindarajan R. (2005) Feature level fusion using hand and face biometrics. In: Proceedings of the SPIE Conference on Biometric Technology for Human Identification, II:196-204 10. Kumar A., Wong D.C.M., Shen H.C., Jain A.K. (2003) Personal verification using palmprint and hand geometry biometric. In: Proceedings of 4th International Conference on Audio and Video-based Biometric Person Authentication, 668-678 11. Singh S., Gyaourova G., Pavlidis I. (2004) Infrared and visible image fusion for face recognition. In: SPIE Defense and Security Symposium (Biometric Technology for Human Identification), 5404:585-596 12. Chang K., Bowyer K., Barnabas V., Sarkar S. (2003) Comparison and combination of ear and face images in appearance based biometrics. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(9):1160-1165 13. Ho T.K., Hull J.J., Srihari S.N. (1994) Decision combination in multiple classifier systems. In: IEEE Transaction On Pattern Analysis and Machine Intelligence, 16(1):66-75 14. Gutschoven B., Verlinde P. (2000) Multi-modal identity verification using support vector machines (SVM). In: Proceedings of the 3rd International Conference on Information fusion, 2:3-8 15. Chatzis V., Bors A.G., Pitas I. (1999) Multimodal decision-level fusion for person authentication. In: IEEE Transaction on Systems, Man and CyberneticsPart A: Systems and Humans, 29(6):674- 680 16. Ben-Yacoub S., Abdeljaoued Y., Mayoraz E. (1999) Fusion of face and speech data for person identity verification. In: IEEE Transaction on Neural Networks, 10(5):1065-1074 17. Bigun E., Bigun J., Duc B., Fischer S. (1997) Expert conciliation for multimodal person authentication systems using Bayesian Statistics. In: Proceedings of the First International Conference on Audio - Video based Personal Authentication, 291-300 18. Prabhakar S., Jain A.K. (2002) Decision-level fusion in fingerprint verification. In: Pattern Recognition, 35(4):861-874 19. Chang K.I., Bowyer K.W., Flynn P.J. (2003) Face recognition using 2D and 3D facial data. In: Proceedings Of Workshop on Multimodal User Authentication, 25-32 20. Ross A., Jain A.K., Reisman J. (2003) A hybrid fingerprint matcher. In: Pattern Recognition, 36(7):1661-1673 21. Brunelli R., Falavigna D. (1995) Person identification using multiple cues. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(10):955966 22. Hong L., Jain A.K.(1998) Integrating faces and fingerprints for personal identification. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(12):1295-1307 23. Frischholz R.W., Dieckmann U. (2000) BioID: A multimodal biometric identification system. In: IEEE Computer, 33(2):64-68

348

R. Singh et al.

24. Daugman J. Combining multiple biometrics. Cambridge University Webnote: http://www.cl.cam.ac.uk/users/jgd1000/ 25. Geruso M. (2002) An analysis of the use of iris recognition systems in U.S. Travel Document Applications. In: Journal of Engineering and Public Policy 6 (http://www.wise-intern.org/journal02/MichaelGeruso.pdf) 26. Chibelushi C.C., Mason J.S., Deravi F. (1993) Integration of acoustic and visual speech for speaker recognition. In: EUROSPEECH, 157-160 27. Dieckmann U., Plankensteiner P., Wagner T. (1997) Sesam: A biometric person identification system using sensor fusion. In: Pattern Recognition Letters, 18(9):827-833 28. Singh R., Vatsa M., Noore A. (2005) Textural feature based face recognition for single training images. In: IEE Electronics Letters, 41(11):640-641 29. Bigun J., Buf J.M.Du. (1994) N-folded symmetries by complex moments in Gabor space and their applications to unsupervised texture segmentation. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(1):80-87 30. Singh R., Vatsa M., Noore A. (2005) Performance enhancement of 2D face recognition via mosaicing. In: Fourth IEEE Workshop on Automatic Identification Advanced Technologies, 63-68 31. Singh R., Vatsa M., Noore A. (2005) Recognition of Faces with Variations in Disguise. Presented at: Summer School on Biometric Authentication: Multimodality and System Integration 32. Singh R. (2005) Unconstrained Face Recognition for Law Enforcement Applications. Master’s Thesis, West Virginia University 33. Vapnik V.N. 1995 The nature of statistical learning Theory. Springer, NY 34. Ayat N.E., Cheriet M., Suen C.Y. (2002) Optimization of the SVM kernels using an empirical error minimization scheme. In: Lecture Notes in Computer Science, 2388:354-269 35. Scholkopf B., Smola A.J., Williams R., Bartlett P. (2000) New support vector algorithms. In: Neural Computation, 12:1083-1121 36. Chen P.-H., Lin C.-J., Sch¨ olkopf B. (2005) A tutorial on nu-support vector machines. In: Applied Stochastic Models in Business and Industry, 21:111-136 37. Chew H.G., Lim C.C., R.E. Bogner. (2004) An implementation of training dualnu support vector machines. In: Qi, Teo, and Yang, editors, Optimization and Control with Applications, Kluwer 38. Davenport M.A. (2005) The 2ν-SVM: A cost sensitive extension of the ν-SVM. Technical report TREE 0504, Department of ECE, Rice University 39. Jain A.K., Nandakumar K., Ross A. (2005) Score normalization in multimodal biometric systems. In: Pattern Recognition, 38(12):2270-2285 40. Fierrez-Aguilar J., Ortega-Garcia J., Gonzalez-Rodriguez J., Bigun J. (2004) Kernel-based multimodal biometric verification using quality signals. In: Proceedings of SPIE Biometric Technology for Human Identification, 5404:544-554 41. Duc B., Maitre G., Fischer S., Bigun J. (1997) Person authentication by fusing face and speech information. In: Proceedings of the First International Conference on Audio- and Video-based Biometric Person authentication, Lecture Notes in Computer Science, 311-318 42. Verlinde P., Chollet G., Acheroy M. (2000) Multi-modal identity verification using expert fusion. In: Information Fusion Elsevier, 1(1):17-33 43. Philips P.J., Moon H., Rizvi S.A., Rauss P.J. (2000) The FERET evaluation methodology for face recognition algorithms. In: IEEE Transaction on Patterns Analysis and Machine Intelligence, 22(10):1090-1104


349

44. http://www.equinoxsensors.com/products/HID.html 45. Chiunhsiun L., Kuo-Chin F. (2000) Human face detection using geometric triangle relationship. In: Proceedings of the International Conference on the Pattern Recognition, 2:941-944 46. Belhumeur P.N., Hespanha J.P., Kriegman D.J. (1997) Eigenfaces vs. Fisherfaces: recognition using class specific linear projection. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7):711-720

Part V

Theoretical Contributions

Chapter 13 General Definition of Fuzzy Mathematical Morphology Operations. Image and Non-image Applications Antony T. Popov St. Kliment Ohridski University of Sofia Faculty of Mathematics and Informatics 5 J. Bourchier Blvd., BG-1164 Sofia, Bulgaria [email protected]

Summary Fuzzy mathematical morphology has been developed to soften the classical binary morphology so as to make the operators less sensitive to image imprecision. It can also be viewed simply as an alternative grey-scale morphological theory. In this work a generalized approach for constructing dilation-erosion adjunctions on fuzzy sets is presented. Some applications in the field of computation with uncertainties are given, more precisely in the interval arithmetic and the calculations with fuzzy numbers. Applications to image segmentation are presented as well, such as colour image processing, geodesic operations and reconstruction, texture analysis through calculation of fractal dimension.

1 Introduction In real vision systems, there is inherent imprecision in the spatial and intensity characterization of an image. At low levels of image representation, there are effects of noise in sensory transduction and of sampling frequency. At higher levels of image representation, there is more general imprecision related to the specifications of image quality. The use of fuzzy set theory in imaging is one way to model imprecision, and a fuzzy approach leads naturally to fuzzy generalization of the morphological operations commonly used in image processing and pattern recognition. There are several approaches to fuzzifying mathematical morphology, see for instance [1, 3, 30]. In our work we step on the framework of Deng and Heijmans [8] based on adjoint fuzzy logical operators - conjunctors and implicators. We generalize this definition, and thus we can define naturally fuzzy geodesic morphological operations. Also, A.T. Popov: General Definition of Fuzzy Mathematical Morphology Operations. Image and Non-image Applications, StudFuzz 210, 353–381 (2007) c Springer-Verlag Berlin Heidelberg 2007 www.springerlink.com

354

A.T. Popov

this model is applied to fuzzy arithmetic which makes possible the definition of inner addition and multiplication of fuzzy numbers. Grey - scale images can be represented naturally as fuzzy sets regarding 0 as black (background), 1 as white (foreground), and any intermediate pixel value as a degree to which the pixel belongs to an object. Colour images can be represented as 3 grey - scale images - red, green and blue component, and thus they can be processed naturally applying directly a grey - scale algorithm to any of the three colour channels - R, G and B. However, in this way when filtering noise we can change undesirably the original colours in the image. Therefore other colour models like HSV (hue, saturation, value), HLS (hue, luminance, saturation), or CIE-L*U*v* and CIE - L*a*b* in which the luminance and chrominance information is separated. Based on such kind of colour model, the algorithms have not the disadvantage to have to choose arbitrarily which of the red, green or blue channel will play dominant role in ordering. The L*a*b* model has the disadvantage that the conversion back to RGB space is not straightforward and may lead to unproper results. Therefore L*a*b* and L*u*v* models are good for segmentation and recognition tasks but not for image enhancement tasks. A model which is suitable for image enhancement and easy to compute is HSV (see for instance [23]). In this model the hue component has an angular structure, it varies from 0 to 2π such that these two values represent one and the same colour information. Therefore, a problem arises in ordering the hue values. In [18] an arbitrary used origin on the unit circle is considered, and the rest of the values are measured by their distance (the length of the shortest arc) . However, thus we cannot define suprema and infima of collections of hue values, and thus the opening and closing filters defined by this ordering are not idempotent. In [13] an interesting fuzzy ordering of the hue is defined based on psychophysical reasons. However, this ordering is not precise, and the operations are time consuming. In this work we propose a method based on YCrCb colour model [23], in which the luminance parameter Y and the chrominance parameters Cr and Cb vary between 0 and 1 by linear ordering (i.e. they can be naturally interpreted as fuzzy sets), and the transformation between RGB and YCrCb spaces is linear. In our work we show also that we can classify efficiently different textures by computing a discrete analog of the Minkowski - Bouligand fractal dimension through fuzzy morphological operations.

2 Binary mathematical morphology and the abstract lattice theory Mathematical morphology has been created as an approach in the field of image processing for extraction of geometric information from an image. The basic morphological operations — erosions and dilations — are introduced by Matheron [12] and Serra [25] to examine the structure of binary images which

Fuzzy Mathematical Morphology Operations

355

can be represented mathematically as sets. Dilations and erosions over sets are represented as Minkowski addition and subtraction by means of a fixed set, usually called a structuring element. Morphological operations have been extended to grey-scale images modeled mathematically as functions. Serra [26] and Heijmans[7] have shown that morphological operations can be extended to any complete lattice. Another intuitive clear extension of the morphological operations to grey-scale images is by using fuzzy set theory [30]. In this work we show how the fuzzy approach matches the lattice framework. A set L with a partial ordering “≤” is called a complete lattice if every subset H ⊆ L has a supremum (least upper bound) and infimum (greatest lower 0 bound)./The supremum and infimum operations are denoted respectively by H and H. The supremum of the whole lattice shall be denoted by I and infimum by O. A comprehensive discussion of the theory of morphological operations on complete lattices can be found in [7]. Let us consider the family of lattice operators L∗ = {ϕ|ϕ : L → L}. The ordering in L induces an ordering in L∗ , i.e. we say that ϕ ≤ ψ if ϕ(X) ≤ ψ(X) for every X ∈ L. In a similar way we can define supremum and infimum operations in L∗ : 1 % 1 % ϕi (X) = ϕi (X) ϕi (X) = ϕi (X). (1) i

i

i

i

Therefore the family of lattice operators has also the structure of a complete lattice. An operator ϕ : L → M, where L and M are complete lattices, is called dilation if it distributes over arbitrary suprema: 0 0 ϕ( i∈I Xi ) = i∈I ϕ(Xi ). The operator ϕ : L → M, where L and M are complete lattices, is erosion if it distributes over arbitrary infima: / / ϕ( i∈I Xi ) = i∈I ϕ(Xi ). Erosions and dilations are increasing operations, i.e. X ≤ X implies ϕ(X) ≤ ϕ(X ). An operator φ ∈ L∗ is called a closing if it is increasing, idempotent (φ2 = φ) and extensive (φ(X) ≥ X). An operator γ ∈ L∗ is called an opening if it is increasing, idempotent and anti-extensive (γ(X) ≤ X) [7]. A pair of operators (ε, δ) between two lattices, ε : M → L and δ : L → M is called an adjunction if for every two elements X ∈ L and Y ∈ M it follows that δ(X) ≤ Y ⇐⇒ X ≤ ε(Y ). In [7] it is proved that if (ε, δ) is an adjunction then ε is erosion and δ is dilation. In other hand, every dilation δ : L → M has a unique adjoint erosion ε : M → L, and vice-versa. If (ε, δ) is an adjunction, then the composition φ = εδ is a closing in L, and the composition γ = δε is an opening in M. The following two propositions are proved in [7] :

356

A.T. Popov

/ 0 • If {(εi , δi )|i ∈ I} is a set of adjunctions, then ( i εi , i δi ) is an adjunction. • If (ε1 , δ1 ) and (ε2 , δ2 ) are adjunctions, where δ1 : L → M, δ2 : M → N , ε1 : M → L and ε2 : N → M, then (ε1 ε2 , δ2 δ1 ) is an adjunction between the lattices L and N . Let us consider that L = M. Consider an abelian (i.e. commutative) group T of automorphisms of L. Let there exist a supremum generating subset l ⊆ L such that every element of L can be represented as a supremum of elements from l, and let T act transitively over l. The transitivity means that for every two elements a and b from l there exists a (unique) transformation τ ∈ T such that τ (a) = b. The lattice properties guarantee the uniqueness of τ (see [7]). Let us fix an origin o ∈ l and let τh ∈ T be the unique automorphism mapping o into h. Let B be an arbitrary element of the lattice L, and l(B) = {b ∈ l | b ≤ B} be the supremum-generating set of B. Then the operations 1 % δB = τb and εB = τb−1 (2) b∈l(B)

b∈l(B)

form an adjunction [7]. The operators δB and εB are T -invariant (which means that these operators commute with every τ ∈ T : δB τ = τ δB and εB τ = τ εB ), and they are called dilation and erosion by the structuring element B. Note that for any B ∈ L we can define an opposite element 1 ˇ = {τ −1 (o)|b ∈ l(B)}. B (3) b As an example, let us consider the lattice L with elements the subsets of a linear space E (for instance the d-dimensional Euclidean vector space R d or the space of integer vectors of length d, Z d ), i.e. L = P(E). Here and henceforth, by P(X) we denote the family of all subsets of the set X, referred to as the power set of X. The supremum operation in L is the union, and infimum is the intersection operation. Let the group of automorphisms be the group of translations {τb | b ∈ E}, with τb (A) = Ab = {x | x = a + b, a ∈ A}. Since E is a linear space, then τb−1 = τ−b for every b ∈ E. Also δA (B) = δB (A) = A ⊕ B and εB (A) = A & B. Here A ⊕ B is the Minkowski addition and A & B is the Minkowski subtraction: < τb (A), (4) A ⊕ B = {a + b | a ∈ A, b ∈ B} = b∈B

A & B = {z | τz (B) ⊆ A} =

=

τ−b (A).

(5)

b∈B

For the definition of subtraction we refer to the works of Heijmans [7] and Schneider [4] which present the original definition of Hadwiger. Some authors like Matheron [12] and Serra [25] define Minkowski subtraction as A & ˇ ⊆ A}. Both definitions produce the same result when the B = {z | τz (B) structuring element B is symmetric. If 0 ∈ B then


357

A & B ⊆ A ⊆ A ⊕ B.

(6)

For completeness we suppose that A ⊕ ∅ = ∅, E & A = E, ∅ & A = ∅, A & ∅ = E.

(7)

The closing and opening of X by the structuring element B are defined as φB (X) = (X ⊕ B) & B, γB (X) = (X & B) ⊕ B.

(8) (9)

Let denote by X c the complement of the set X, i.e. X c = E \ X. Then the following simple equalities hold [25] : (A ⊕ B)c = Ac & (−B), φB (A)c = γ(−B) (Ac ),

(10) (11)

where by −B we denote the centrally symmetric set of B, −B = {−b|b ∈ B}, ˇ given by (3). Therefore and in this case it is equal to the opposite element B opening and closing operations can be represented as follows: < γB (X) = {τh (B) |h ∈ E and τh (B) ⊆ X}, (12) φB (X) = {k ∈ E| h ∈ τk (B) ⇒ τh (−B) ∩ X = ∅}. These operations are usually referred to as classical or binary morphological operations, because they apply to binary images which could be treated mathematically as sets [25]. There are also some useful operations, which can be expressed by the operations given above. Let B be an isotropic element – for instance 3 × 3 square in the digital case – or any convex centrally symmetric compact set with nonempty interior in the continuous case (recall that a set B in the continuous space R d is said to be convex if for each of its pair of points the line segment joining these points also belongs to B). Then we can define the morphological gradient ∇B (X) = δB (X) \ εB (X)

(13)

It locates the edges of the object X. Another very useful operation is the hit-or-miss transform [25]. It is determined by a pair of structuring elements with empty intersection: πA,B (X) = {h ∈ E |τh (A) ⊆ X and τh (B) ⊆ X c }.

(14)

This operation is very well suited for locating points inside an object with certain local geometric properties, for instance finding the place of a given letter

358

A.T. Popov

in text. More generally, this operation forms the primary building block for translation-invariant mappings and is used extensively for pattern matching. A key application of morphological image processing is the restoration of degraded images. Morphological operators have proved to be very useful for this purpose [7, 25]. Sometimes the operators can be designed heuristically, whereas for complicated image-noise models satisfactory performance often requires automatic design. Openings and closings have proven to be very useful in solving certain classes of problems, in particular, when noise components have been randomly placed into the image background to form clutter or when noise holes have been subtracted from the foreground. In particular, alternating sequential filters, which are sequences of openings and closings with increasing structuring elements, have provided a useful class of filters for this purpose [27]. 2.1 Geodesic operations Let us now consider a distance function d in the linear space E, namely it satisfies the following properties:

d(x, y) = 0

d(x, y) = d(y, x) ≥ 0, if and only if x = y,

d(x, z) ≤ d(x, y) + d(y, z)

(triangle inequality) .

Having a distance function, for any subset M ⊆ E we can define a new intrinsic distance function for M named geodesic distance, i.e. for any two points x, y ∈ M we define dM (x, y) as the shortest length of a continuous curve joining x and y. It is very easy to show that this function also satisfies the upper properties of a distance, see for instance [7]. Then we can construct a geodesic ball with centre x and radius r : BM (x, r) = {y ∈ M | dM (x, y) ≤ r}. r Thus we obtain the adjunction (εrM , δM ) such as

εrM (X) = {y ∈ M | BM (y, r) ⊆ X}, r δM (X) = {y ∈ M | BM (y, r) ∩ X = ∅}

(15) (16)

called respectively geodesic erosion and geodesic dilation (see [7]). It is evident r that δM (BM (x, s)) = BM (x, r + s), and therefore r+s r s δM δM = δM . r r The composition δM εM is refered to as geodesic opening, and the composition r r εM δM - as geodesic closing. The geodesic reconstruction ρM is a dilation defined as the limit of the r when r → ∞ [7]. If the operand X is a set of markers, increasing family δM then


ρM (X) =

0}, then the dilations and erosions are the same and do not depend on the choice of the t-norm implicator pair, namely δB (X)(x) = sup {X(z) | b ∈ supp(B), b ∗ z = x},

(22)

εB (X)(x) = inf {X(z) | b ∈ supp(B), x ∗ b = z}.

(23)

The statement (22) is true, because when the conjunctor is a t-norm, then c(1, x) = x, i.e. the dilation doesn’t depend on the particular choice of the t-norm. Then having in mind that for every dilation there exists only one adjoint erosion, we conclude that the upper erosion is given by (23). Simply, min 1, xb b = 0, we can choose the implicator i(b, x) = , which is adjoint 1 b=0 to the t-norm c(b, x) = bx. Another possibility to define fuzzy T-invariant morphological operations is to use the ordinary grey-scale morphological model, as defined in [25, 7]. Let us remind that if we have two functions f, g : E → R ∪ {∞, −∞}, then we can define the adjoint operations (Dg (f ))(x) = sup[f (x − t) + g(t)] , (Eg (f ))(x) = inf [f (x + t) − g(t)], (24) t∈E

t∈E

where we denote simply τy (x) as x + y and respectively, τy−1 (x) as x − y. Let us introduce the function ψ(t) = tan

2πt − π 2

,

which is continuous and strongly increasing, and therefore invertable with inverse arctan(t) ψ −1 (t) = 0.5 + π with the assumption that ψ −1 (∞) = 1 and ψ −1 (−∞) = 0. Then if B is a fuzzy subset of E, we can define the adjunction (E B , ∆B ) by ∆B (X) = ψ −1 Dψ(B) (ψ(X) , (25) −1 Eψ(B) (ψ(X) . (26) E B (X) = ψ

364

A.T. Popov

If in E we have a transitively acting abelian group of automorphisms T , we may consider an abelian group of automorphisms for the fuzzy subsets of E defined as (θt,v )(X)(x) = ψ −1 [ψ(X(τt−1 (x)) + v] forarbitrary t ∈ E and v ∈ R . Consider also the pulse fuzzy sets Fs,t (x) = t p τs−1 (x) , where p(o) = 1 and p(y) = 0 for any other y ∈ E. Now it is evident that {θt,v } is an abelian group acting transitively on the supremum generating family of pulse fuzzy sets. In the case of flat structuring element B we obtain the same results as when we define the morphological operations by a t-norm - implicator adjoint pair, namely: ∆B (X)(x) = sup {X(z) | b ∈ supp(B), b ∗ z = x},

(27)

E B (X)(x) = inf {X(z) | b ∈ supp(B), x ∗ b = z}.

(28)

Note that the fuzzy gradient ∇B (X) = δB (X) \ εB (X) = ∆B (X) \ E B (X), because in this case we use flat symmetric structuring element B. For the computation of the gradient one can refer to [11].

5 Fuzzy geodesic morphological operations Following [24], let us say that the points x, y ∈ E are connected in the fuzzy set A if there exists a path Γ from x to y such that inf A(z) ≥ min(A(x), A(y)).

z∈Γ

Let now M be a fuzzy subset of the universal set E, which is a numerical metric space. Then if x and y are two points from E which are connected in M, we can define the following geodesic distance between x and y [2]: dM (x, y) =

len (x, y) , CM (x, y)

(29)

where len (x, y) is the length of the shortest continuous path between x and y due to the metric in E, and CM (x, y) = sup inf{M (z)| z ∈ Γ }. Γ

Here Γ denotes an arbitrary path between x and y in E. The quantity dM (x, y) satisfies the first two properites (15) but does not satisfy the triangle inequality, so it is not a real distance. However, if M is a crisp set, then it is equal to the classical geodesic distance. Now we can define a fuzzy geodesic ball


[BM (y, r)](x) =

365

1 if drM (x, y) ≤ r, 0 otherwise.

Having in mind the expressions (18) and proposition 1 we can define a fuzzy geodesic adjunction (E rM , ∆rM ) as: 1 c[(BM (x, r))(y), X(y)], ∆rM (X)(x) = y∈E

E

r M (X)(x)

=

%

i[(BM (y, r))(x), X(y)].

y∈E

Therefore in we can define fuzzy geodesic reconstruction and idempotent fuzzy geodesic openings and closings. An example of fuzzy geodesic reconstrucion is given on figure 1. Three points are used as markers to extract sclerotic lesions from a MRI image of a human brain.

Fig. 1. Detection of sclerotic lesions due to the markers shown with crosses on the left figure

Our definition of a geodesic ball differs from this in [2]. The author there works with geodesic balls with a radius, which is even nonnegative fuzzy number (for the definition of a fuzzy number one can refer to the section 6). In that case the idempotence of the closings and the openings holds evidently as well, because of the usage of our new general definition of fuzzy morphological operations given by (18). In our case the grey-scale geodesic operations are computed much easily.

366

A.T. Popov

6 Computations with intervals and fuzzy numbers Interval computations are computations using intervals with the aim to guarantee the result in particular in the presence of data uncertainties and rounding errors. Since the α-cuts of the fuzzy numbers are closed intervals, then the interval calculus is essential part of the computations with fuzzy numbers. A fuzzy number is a fuzzy subset of R , i.e. it represents a generalization of a real number r. Any fuzzy number A satisfies the following conditions ([17]): • A(x) = 1 for exactly one x; • The support of A is bounded; • The α-cuts of A are closed intervals. Following [14] we introduce the following notations: Let I(R ) be the family of all closed finite intervals of the real line. For completeness we may assume that the empty set ∅ is an element of I(R ). But in our work we shall deal only with proper intervals, i.e. such that their left endpoint is less or equal to their right endpoint. The case when any of the operands is the empty interval is covered by (7). The non-empty finite intervals from I(R ) will be denoted by Roman capitals, namely A = [a− , a+ ], i.e. a− is the left endpoint of the interval A and a+ is its right endpoint. By ω(A) we denote the length of the interval A, ω(A) = a+ − a− . Consider the outer and inner interval additive operations as studied by S. Markov in [14, 15, 16]: A + B = [a− + b− , a+ + b+ ], A +− B = [a−s + bs , as + b−s ], A − B = [a− − b+ , a+ − b− ], A −− B = [a−s − b−s , as − bs ], where

s=

+, ω(A) ≥ ω(B), −, ω(A) < ω(B).

Here and further on, as with s ∈ {+, −} denotes certain endpoint of the interval A — the left one if s = − and the right one if s = +. We define the ‘product’ st for s, t ∈ {+, −} by + + = − − = + , +− = − + = − , i.e. a++ = a−− = a+ and a+− = a−+ = a− . As proved in [20] the outer and inner interval operations in I(R ) are related to dilations and erosions as follows:

A + B = A ⊕ B = δA (B) = δB (A),

(30)


A +− B = A & (−B) ∪ B & (−A) = ε−B (A) ∪ ε−A (B).

367

(31)

Note that since the operations are symmetric, the role of the operand and the structuring element in the morphological representations can be changed. For the outer and inner subtraction we have the following relations: A − B = A + (−B) = A ⊕ (−B), A −− B = A +− (−B) = A & B ∪ −(B & A).

(32) (33)

If L is a complete lattice with a supremum generating family l, and T is an Abelian group of automorphisms of L acting transitively over l, having in mind (31) we can introduce an analog of the inner interval addition, which we will call inner morphological operation, η : L × L → L. Namely η(A, B) = εBˇ (A) ∨ εAˇ (B). / 0 −1 Since εBˇ (A) = z∈l(B) ˇ (τz (A)) = y∈l(B) (τy (A)) ≤ y∈l(B) (τy (A)) = δB (A), then we obtain evidently that /

η(A, B) = η(B, A) ≤ δB (A) = δA (B).

(34)

In many applications such as locating the roots of a polynomial with interval coefficients, it is desirable to spread also the multiplicative operations over the set of real intervals. Let L be the set of all closed finite and infinite real intervals different from the singleton [0, 0]. The order relation is the set inclusion, while the supremum is defined as the closed convex hull of the union, and the infimum is defined as intersection: 1 i

< + Hi = co (cl ( Hi )) = [inf h− i , sup hi ] % i

Hi =

=

i + Hi = [sup h− i , inf hi ].

i

+ [h− i , hi ]}

Here {Hi = is an arbitrary family of closed intervals, co (X) denotes the convex hull of the set X, while by cl (X) we mean its topological closure. A supremum generating family is l = {{x} = [x, x] | x ∈ R , x = 0}, therefore for any A ∈ L l(A) = {{x} | x ∈ A, x = 0}. There is an Abelian group of automorphisms T = {τh | h ∈ R \ {0}, τh ({x}) = {xh}}, which acts transitively on l and τh (A) = [min(a− h, a+ h), max(a− h, a+ h)].

368

A.T. Popov

Then we can define dilation and erosion operations by the structuring element A using expressions (2): 1 τa (B), δA (B) = a∈l(A)

εA (B) =

%

τ1/a (B) .

a∈l(A)

Define Z as the family of closed intervals containing 0. If A ∈ /Z +, if 0 < a− ; σ(A) = −, if a+ < 0. Outer multiplication in I(R ) (see [14]) is defined by  −σ(B) −σ(A) σ(B) σ(A) [a b ,a b ], for A, B ∈ L \ Z,    σ(A) −σ(A) σ(A) σ(A) b ,a b ], for A ∈ L \ Z, B ∈ Z, [a A×B =  for B ∈ L \ Z, A ∈ Z, [a−σ(B) bσ(B) , aσ(B) bσ(B) ],   [min(a− b+ , a+ b− ), max(a− b− , a+ b+ )], for A, B ∈ Z, The following two propositions are proved in [20]: Proposition 3. For the dilation defined above we have that δB (A) = δA (B) = A × B. Proposition 4. For intervals B such that b+ b− = 0  > −σ(B) σ(B) ? a a  for   b−σ(A) , bσ(A) ,    >  ?  −σ(B) σ(B)   for  abσ(B) , abσ(B) , εB (A) =    ∅, for      > ?    max a−+ , a+− , min a++ , a−− , for b b b b

we have that A, B ∈ L \ Z, B ∈ L \ Z, A ∈ Z A ∈ L \ Z, B ∈ Z, A, B ∈ Z.

Let us consider the inner multiplication operation “×− ” defined as [14]:  σ(B) −σ(A) −σ(B) σ(A) [a b ,a b ], = ψ(A, B), A, B ∈ L\Z,    −σ(A) −σ(A) −σ(A) σ(A) b ,a b ], A ∈ L \ Z, B ∈ Z [a − A× B =  A ∈ Z, B ∈ L \ Z, [a−σ(B) b−σ(B) , aσ(B) b−σ(B) ],   [max(a− b+ , a+ b− ), min(a− b− , a+ b+ )], A, B ∈ Z.


369

Here ψ(A, B) = + if χ(A) ≥ χ(B) and ψ(A, B) = − otherwise, where χ(A) = a− /a+ if a− + a+ ≥ 0 and χ(A) = a+ /a− if a− + a+ < 0. 7 6 Denoting B1 = b1+ , b1− , (which in this case is namely the opposite element ˇ we obtain: of B in L, i.e. B1 = B) Corollary 1. A ×− B = ε A1 (B) ∪ ε B1 (A) for any intervals A, B ∈ L \ Z. Proof. From Proposition 4 we have that ε B1 (A) = [a−σ(B) bσ(A) , aσ(B) b−σ(A) ], ε A1 (B) = [aσ(B) b−σ(A) , a−σ(B) bσ(A) ], Consider first the case when σ(B) = σ(A) = +. Then we have that ε B1 (A) = [a− b+ , a+ b− ] and ε A1 (B) = [a+ b− , a− b+ ]. Then ε A1 (B) = ∅ if and only if a+ b− > a− b+ , which holds if and only if χ(A) < χ(B), i.e. ψ(A, B) = − . Therefore ε A1 (B) ∪ ε B1 (A) = [a− b+ , a+ b− ] = A ×− B. If χ(A) > χ(B) then ψ(A, B) = +, and a+ b− > a− b+ . This means that ε B1 (A) = ∅ and ε A1 (B) ∪ ε B1 (A) = [a+ b− , a− b+ ] = A ×− B. In the case when χ(A) = χ(B), by definition ψ(A, B) = +, and both intervals ε A1 (B) and ε B1 (A) degenerate into same one-point interval — in this case the statement is trivial. Thus the proposition is proved in the case when σ(B) = σ(A) = +. The cases σ(B) = σ(A) = − and σ(B) = −σ(A) are regarded in the same way. Thus we demonstrate that the inner interval multiplication is an inner morphological operation. Now let denote by F (R ) the set of fuzzy numbers. Then we can define following operations on them using the extension principle [17]: 1 (A + B)(x) = min(A(y), B(z)); z+y=x

(A × B)(x) = (A − B)(x) =

1

1

min(A(y), B(z));

z.y=x

min(A(y), B(z)) = (A + (−B))(x);

y−z=x

1 A 1 (x) = min(A(y), B(z)) = A × (x). B B zx=y Note that every real number r could be considered as fuzzy number with membership function, which is zero on the whole real line, except in r where it takes value 1.

370

A.T. Popov

The sum, the difference and the product of fuzzy numbers are also fuzzy numbers. The division is always possible, however the result is a fuzzy number only when 0 ∈ / supp(B). Also, if A and B are fuzzy numbers then [A + B]α = [A]α + [B]α and [A × B]α = [A]α × [B]α . Now consider the group of automorhisms τb (x) in R and the fuzzy operations on F (R ) defined by G¨ odel-Brouwer conjunctor - implicator pair: (δB (A))(x) =

1

min(A(y), B(z)),

y∗z=x

(εB (A))(x) = inf h A(y) − B(τx−1 (y) (1 − A(y)) + A(y) , y∈R

where h(x) = 1 when x ≥ 0 and is zero otherwise. Now it is clear that if τb (x) = x + b

and ∗ = +

then

(δB (A)) = A + B. We can also define an inner addition operation by A +− B = ε−B (A) ∪ ε−A (B). If τb (x) = xb for b = 0 and y ∗ z = yz then (δB (A)) = A × B. In this case an inner multiplication exists as well: A ×− B = ε B1 (A) ∪ ε A1 (B). Note that in this definition we can work with fuzzy numbers which do not contain 0 in their support. Here we cannot apply equation (34), but it is not difficult to show directly that A+− B ⊆ A+B and A×− B ⊆ A×B. The last property holds true also if we use the erosions and dilations defined by (25) to define analogical outer and inner addition and multiplication operations, because in this case we can apply (34). However, the computation of ∆B (A) and E B (A) needs the usage of transcendent functions tangent and arctangent which makes the computations not very efficient. Another non - image application of fuzzy mathematical morphology related to interval - valued knowledge representation is given in [21].

7 Fuzzy mathematical morphological operations for colour images We saw that the basic morphological operations are expressed as products of the suprema and the infima of the lattice under study. When we deal with


371

colour images, we work in fact in a multidimensional space (usually R 3 or Z 3 ), where a natural ordering of the elements cannot be achieved. Therefore we try to introduce some heuristics and to compromise with the accuracy at acceptable level to guarantee the lattice properties and therefore to ensure idempotent opening and closing filtering. An useful implementation of a basic subjective colour model is the HSV (hue, saturation, value) cone [23]. It was created by A. R. Smith in 1978. It is based on such intuitive colour characteristics as tint, shade and tone (or family (hue), purety (saturation) and intensity (value)). The coordinate system is cylindrical, and the colors are defined inside a hexagonal cone. The hue value H runs from 0 to 2π. The saturation S is the degree of strength or purity and varies from 0 to 1. The saturation shows how much white is added to the color, so S=1 makes the purest color (no white). Brightness (value) V also ranges from 0 to 1, where 0 is the black. We could experience that in general the value component contains most of the information about the texture and boundary location. The hue component is usually the most homogeneous (with small variance). The saturation component allows to differentiate between different shades of the same colour. Considering the red, green and blue channel values of a pixel as a point in the 3D space, more precisely in a cube 1 × 1 × 1, we obtain a colour cube in which (white,black), (green,magenta), (red,cyan), (yellow,blue) are the pairs of opposite vertices. If this unit cube is projected onto a plane along the diagonal looking from white to black we obtain a hexagon with vertices corresponding to the red, green and blue primary colours and their complements - cyan, magenta, yellow. Decreasing the saturation (purity) of the primary numbers we obtain a smaller cube which yelds a smaller projection hexagon. If we stack these hexagonal projections we obtain the hexagonal cone. We may slightly enlarge this cone to a cone with circular base. Then the HSV colour model values for a colour image are defined as follows: VALUE increases over the axis of the cone. Without lack of generality we may consider the height of the cone to be 1. The SATURATION is given as the distance to the axis, therefore it varies also from 0 to 1. The HUE is measured as the polar angle in a plane through the saturation segment and orthogonal to the cone axis. This angle is measured counterclockwise from red. Instead of [0, 2π) we may use the interval [0, 1) dividing the angle by 2π. If S=0 the hue is undefined, and therefore the colour is achromatic, namely a shade of grey. The fully saturated primary colours occur when S=1. On figure 2 one can see a fuzzy decomposition of the hue value by 7 triangular fuzzy numbers, two of them represent the red colour, and the rest - the yellow, green,cyan, blue and magenta colours. However, when we use the HSV model, the main obstacle is the fact that the hue is measured as an angle, and it is not defined for the levels of grey. Therefore there are not clear mathematical reasons for its ordering. How-

372

A.T. Popov

1 R

G

Y

C

B

M

R

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

0

0.1667

0.3333

0.5

0.6667

0.8333

1

Fig. 2. Fuzzy decomposition of the hue

ever, from psychophysiological point of view one may order the colours in the following way - red, magenta, blue, yellow, cyan, green, based on the way humans perseive the hue of the colour. Red is considered to be the smallest, since it stimulates the eye less than the other colours. Contrary, green mostly stimulates the eye [13]. This ordering can be done by triangular fuzzy numbers. Therefore we can present an invertable function HO that converts the true hue values in numbers in [0, 1] in partially linear way following the psychophysiological order. For clarity we use the syntax of C programming language. float HO(float x) { if (x < 1/12.) return 1/12. else if (x < 1/4.) return else if (x < 5/12.) return else if (x < 7/12.) return else if (x < 3/4.) return else if (x < 11/12.) return else return x - 11/12.; }

+ x; 1/2. 5/6. 2/3. 1/3. 1/6.

+x - 1/12.; +x - 1/4.; + x - 5/12.; + x - 7/12.; + x - 3/4.;

//larger red //yellow //green //cyan //blue //magenta //smaller red


373

An alternative of the HSV model is to use the YCrCb colour model [23]. To obtain the parameters Y, Cr and Cb we use a linear combination of R,G, B values. Note, that Y represents the lightness (luminance), and should not be mistaken with the yellow colour in the RGB model notation. The parameter Cr encodes the red-cyan sensation, with value ≈ 0 for the cyan colour and ≈ 1 for the red. The parameter Cb encodes the yellow-blue sensation with ≈ 0 indicating yellow and ≈ 1 indicating blue. Without lack of generality we can assume that R,G and B values are represented as points in an unit cube, namely 0 ≤ R, G, B ≤ 1. The YCrCb colour space is also a unit cube with transformation formulas:

Y = 0.299 R + 0.577 G + 0.114 B, Cr = 0.5 R − 0.411 G − 0.081 B + 0.5,

(35) (36)

Cb = 0.5 B − 0.169 R − 0.325 G + 0.5 .

(37)

Given the matrix



 0.9988 1.4026 −0.0005 M =  1.0181 −0.7271 −0.3499  0.9993 0.0015 1.7724

and the column vector z = (−0.7010, 0.5385, −0.8869)T , it is easy to show that     R Y  G  = M  Cr  + z. (38) B Cb Henceforth it is clear that the transformation between RGB and YCrCb models is linear and easy to compute. When we use the HSV model usually we give priority to the value (V), since if anyone looks at the V-map of a colour image, he can usually distinguish the different objects on the image as looking on a black -and- white TV. Much better grey-scale representation of a colour image is produced by its Y - map. Generally, less priority is given to the chrominance maps - V,S in HSV model, or Cr, Cb in YCrCb. When working with HSV model, the next priority is given to the hue, because it contains mostly the colour information. The least priority is given to the saturation because it is correlated with the other two components and its role as a parameter in image processing tasks is criticized – one can refer for instance to the works [6] and [28]. This is another reason to prefer the YCrCb model, where the components Cr and Cb have equal weights. 7 i Let us divide the interval [0, 1] into N equal pieces Ii = i−1 N , N . (0 ∈ I1 ). Let us also suppose that for a pixel x Cb(x) ∈ Ij and 1−Cr(x) ∈ Ii . Note that

374

A.T. Popov

we use the negation of Cr to obtain an ordering closer to the one presented in [13]. On table 1 one can see a N × N table with a zigzag path going through its cells. Table 1. Zigzag tracing of the CrCb parameter space. 1 2 9 10 ...

4 3 8 ... ...

5 6 7 ... ...

... ... ... ... ...

Thus we code approximately with accuracy 1/N the Cr and Cb values by the number of the step at which we visit the respective cell. Let us consider the cell (i, j) and let n = max(i, j) and m = min(i, j). Then for the number of the step we have 2 n − even and n = i, or n − odd and n = j n −m+1 T (i, j) = otherwise n2 − 2n + m + 1 Further on, for simplicity, for any pixel x we will denote the respective integer number from the table by T (x). Then if given a colour image X, we define the transformation N 2 [(N 2 − 1)Y (x)] + T (x) − 1 , (χ(X))(x) = N4 − 1 which is a real number between 0 and 1. Then it is clear that having χ(X), we can find Y (x) with accuracy 1/N 2 and Cr(X) and Cb(x) with 1/N simply by taking the quotient and the reminder of the division of [χ(X)(N 4 − 1)] by N 2 . Here [t] means the whole part of t. Similar approach can be apllied to the HSV model: Then ˜ − 1) + s˜ − 1 N 2 [(N 2 − 1)V (x)] + N (h , (χ(X))(x) = 4 N −1 ˜ and s˜ are the number of the intervals with length N containing the where h ordered hue and the saturation. Then the colour fuzzy morphology when given an adjoint conjunctor implicator pair can be represented as:

δB (X)(x) = χ−1

1 y

−1

εB (X)(x) = χ

c(B(x − y), (χ(X))(y)) ,

% y

(39)

i(B(y − x), (χ(X))(y)) .

(40)


375

Operations with non-flat structuring elements are rarely used, because it is not clear a priori how an incremental operation will affect the colours. They can be used in texture analysis by fractal dimension estimation as shown in section 8. On the figure 3 one can see a colour picture of flowers. Next its dilation, erosion, opening and closing by 3 × 3 flat square structuring element are presented. Note that in this case the choice of the conjunctor - implicator pair is not essential.

Fig. 3. From top to bottom and left to right: original, dilation, erosion; opening, closing

There is no visible difference between the results when we use YCrCb or HSV model. On figure 4 a collage photo of the author is given, and the results of edge detection algorithms (Laplasian of Gaussian and Sobel) and morphological gradient using our approach. To construct the morphological gradient, the dilation and erosion of the colour picture are calculated again using 3 × 3 flat structuring element.

376

A.T. Popov

Fig. 4. Authors portrait: original, morphological gradient, Laplacian of Gaussian filter, Sobel filter

8 Morphological operations and an efficient way to compute fractal dimension The topological dimension of a set in the three dimensional physical space is related with notions like length, area and volume. The sets having only length are refered to as one - dimensional, those who have area but no volume are said to be two-dimensional, and those having volume - three dimensional. Let us for instance measure a the length of a curve having structures at all scales like the coastline of Norway [33]. This length can be measured by a yardstick of length L. The resulting length is the number of yardsicks needed to walk from the beginning to the end of the curve. For a curve having finite length (so called rectifiable curve, the C 2 -smooth curves like conics, splines, Bezier curves etc. are rectifiable) as L tends to zero, the measured length tends to the actual length of the curve. This does not happen for the fractal curves when L tends to zero the measured length increases without limit to infinity. This fact is formalized by introducing the so called fractal dimension, which is one for rectifiable curves, and between one and two for fractal curves. There are different definitions of fractal dimension [33]. The Hausdorff - Besicovitch dimension is difficult to compute; much easier to compute is those proposed by Minkowski, who estimates the length of the curve at resolution L by replacing each point of the curve by a disk of diameter L and dividing the area of the resulting ribbon by L. Note that the replacement of the curve with the ribbon is equivalent to a dilation with a disk with diameter L. The same idea can be applied to the surfaces in R 3 . The surface is dilated by a ball with diameter L and the volume of the obtained sandwich is divided by L to find an estimate of the surface area. Therefore for arbitrary compact set X in R n Bouligand defined the following limit: log µn (X ⊕ Bδn ) D(X) = lim DT (X) − , (41) δ→0 log δ


377

where DT (X) means the topological dimension of X, µk means the k-dimensional volume (Lebesgue measure) and Brk is the closed ball with radius r in R k . D is known now as Minkowski - Bouligand dimension of the set X, and for fractal sets (i.e. sets with self similarity or statistical self-similarity) DT (X) < D(X) ≤ DT (X) + 1, while for nonfractal sets D = DT . Let us now consider the interesting from signal processing point of view case, when X is the graph of a bounded function f : P → R , and P is compact connected subset of R n , and its boundary ∂P is measurable, i.e. µn−1 (∂P ) < ∞. Then from the sigma - additivity of the Lebesgue measure, it follows that in (41) we may substitute the volume of X ⊕ Bδn+1 by the volume of the truncated parallel body of X with thicknes δ, defined as (X ⊕ Bδn+1 ) ∩ (P × (−∞, ∞)). If f is continuous, then the truncated parallel body of its graph is a compact set in R n+1 . Then, as proved in [10] (Theorem 1), if f : P → R is a continuous function defined on such a set P as described above, and bε is the function defined as bε (x) = max{y|(x, y) ∈ Bεn+1 (x)} with domain the ball Bεn (0), then the n + 1-dimensional volume of the truncated parallel body of the graph of f with thickness ε can be calculated through Lebesgue integration as P (f ⊕ bε − f & bε ). The proof is made for one dimensional signals, but it can be repeated without any problems and the result holds true for any Euclidean space R n – see for instance [33] for the two - dimensional case. As seen above, the theory of fractal dimension works based on the ordinary grey-scale morphological operations as defined by (24) in [7, 25]: f ⊕ g = Dg (f ) , f & g = Eg (f ). However in practical applications, when we measure discrete analog of the Minkowski - Bouligand dimension, we can overgo the limits of the pixel values. Thus we try to substitute in practice f ⊕ bε by δB (A) and f & bε by εB (A), where A is the fuzzy-set representation of the image f , and B is a symmetric structuring element with B(0) = 1. Another reason for doing this is the note of P. Soille in [33] that in practice, the intensity surface of an image has intensity units along the grey-scale axis and spatial along the image plane. As a consequence, flat structuring elements rather than spheres might be conidered. It is easy to show that if in the conjunctor - implicator pair the conjunctor is a t-norm, then δB (A)(x) ≥ A(x) since c(1, y) = y. For the adjoint implicators of the t-norms considered in section 3.2 we have that i(1, y) = y, and therefore δB (A)(x) ≥ A(x) ≥ εB (A)(x), so the substitution is plausible. Many natural textures present structures at all scales or show a high degree of roughness. Metric measurements of textured images are scale dependent and tend towards infinity when the size of the probe decreases toward 0 [34]. Then it is clear that the fractal dimension is an elusive concept since it encompasses

378

A.T. Popov Table 2. Dimension of the natural textures on figure 5. Fractal Dimension structuring element B1 B2 left 2.2050 2.2113 right 2.1735 2.1860

different noninteger dimensions. So, in practice we need efficient procedures to calculate analogs of the fractal (Minkowski - Bouligand) dimension for discrete sets, because the image plane conists of finite number of pixels. The usage of fuzzy operations in calculating the fractal dimension leads to results which do not depend on lighting conditions. Below one can see two microscopic pictures of two sphalerite samples. On table 2 one can see the respective values of the fractal dimension calculated using Lukasiewicz conjunctor- implicator pair   0 0.6 0 with two different structuring 3 × 3 elements nonflat B1 =  0.4 1 0.4  0 0.6 0   111 and flat B2 =  1 1 1  . 111

Fig. 5. Microscopic images of two different sphalerite probes

Note that the fractal dimension quantifies the human perception of roughness of the material. This experiment shows that the usage of fuzzy operations for computing fractal dimension of textured image makes sence. Moreover, a commercial software based on ordinary grey-scale morphological operations produces erroneous results on the specific sphalerite images due to floating point errors. It is also clear,that generalized fuzzy morphological operations could be ap-


379

plied directly on images with integer pixel values in the interval [0, L] in the case of morphological operators generated by Lukasiewicz and G¨ odel-Brouwer conjunctor - implicator pairs. Usually in practice we have L = 255, and the usage of integer arithmetic can make the operations much faster. Simply, we can substitute in conjunctor and implicator formulas the 1-s by L-s. Further studies might be possible to analyze the dependence of the fractal dimension on the specific physical properties and the morphogenesis of the minerals and ores. Our approach gives the opportunity also to analyze and segment directly textured colour images.

References 1. Bloch I. and Maˆıtre H. (1995) Fuzzy mathematical morphologies: a comparative study, Pattern Recognition, Vol. 28, No. 9, 1341–1387 2. Bloch I. (2000) Geodesic balls in a fuzzy set and fuzzy geodesic mathematical morphology, Pattern Recognition Vol. 33, No. 6, 897–905 3. De Baets B. (1997) Fuzzy morphology: a logical approach, in: Uncertainty Analysis in Engineering and the Sciences (B. Ayyub and M.M. Gupta, eds.), Kluwer Academic Publishers, Norwell 4. Schneider R. (1993) Convex bodies: the Brunn - Minkowski theory (Encyclopedia of Math. and its Appl., Vol. 44), Cambridge University Press, Cambridge UK 5. Dougherty E.R., Sinha D. and Sinha. P (1994) Fuzzy morphological algorithms, in: Mental and Fuzzy Systems (S. Mitra, M.M. Gupta and W.F. Kraske, eds.), SPIE Press, Bellingham WA 6. Hanbury A. and Serra J. (2001) Mathematical morphology in the HLS colour space, T. Cootes, C. Taylor (eds.), Proceedings of the British Machine Vision Conference 2001, Springer, 451–460. 7. Heijmans H.J.A.M. (1994) Morphological image operators, Academic Press, Boston MA 8. Deng T.-Q. and Heijmans H. J. A. M. (2000) Grey-scale morphology based on fuzzy logic, CWI Report PNA-R0012, Amsterdam, October 2000 9. Kaufmann A. and Gupta M. (1988) Fuzzy mathematical models in engineering and management science, North-Holland, Amsterdam 10. Maragos P., Fang-Kuo S. (1993) Measuring the fractal dimension of signals: Morphological covers and iterative optimization. IEEE Trans. Signal Process., Vol. 41, No.1, 58–67 11. Maragos P., Tzouvaras V., and Stamou G. (2003) Lattice fuzzy signal operators and generalizred image gradients, Proc. 10th International Fuzzy System Association World Congress (IFSA03), Istanbul, Turkey, July 2003, LNCS Vol. 2715, 412–419 12. Matheron G. (1975) Random sets and integral geometry, Wiley, New York 13. Louverdis G., Andreadis I., and Tsalides Ph. (2002) New fuzzy model for morphological colour image processing, IEE Proceedings – Vision, Image and Signal Processing, Vol. 149, No. 3, 129–139

380

A.T. Popov

14. Markov S. M. (1977) Extended interval arithmetic, Compt. rend. Acad. Bulg. Sci., Vol. 30, No. 9, 1239–1242 15. Markov S. M. (1992) On the presentation of ranges of monotone functions Using interval arithmetic, Interval Computations, Vol. 4, No. 6, 19–31 16. Markov S. M. (1997) Isomorhic embeddigs of abstract interval systems, Reliable Computing, Vol. 3, 199–207 17. Nguyen H.T. and Walker E. A. (2000) A first course in fuzzy logic (2nd edition) CRC Press, Boca Raton FL 18. Peters II, R. A. (1997) Mathematical morhology for angle - valued images, SPIE Proceedings Vol. 3026 - Non-linear image processing VIII, SPIE Press, Bellingham WA 19. Popov A.T. (1997) Convexity indicators based on fuzzy morphology, Pattern Recognition Letters, Vol. 18, No. 3, 259–267 20. Popov A.T. (1998) A relation between morphological and interval operations, Reliable Computing, Vol. 4, No. 2, 167–178 21. Popov A.T., Nguyen H.T, Reznik L. K. (1998) An application of fuzzy mathematical morphology to interval - valued knowledge representation: A remark, Reliable Computing, Vol. 4, No. 3, 283–290 22. Popov A. T. (2000) Aproximate connectivity and mathematical morphology, in: J. Goutsias, L. Vincent, D. S. Bloomberg (eds.), Mathematical Morohology and its Applications to Image and Signal Processing, Kluwer, 149–158 23. Rogers D. F. (1998) Procedural elements for computer graphics (2nd edition), WCB McGraw - Hill 24. Rosenfeld A. (1984) The fuzzy geometry of image subsets, Pattern Recognition Letters, Vol. 2, 311–317 25. Serra J. (1982) Image analysis and mathematical morphology, Academic Press, London 26. Serra J. (1988) Mathematical morphology for complete lattices, in: Image analysis and mathematical morphology, Vol. 2 (J. Serra, ed.), Academic Press, London 27. Serra J. (1988) Alternating sequential filters, in: Image analysis and mathematical morphology, Vol. 2 (J. Serra, ed.), Academic Press, London, 1988 28. Serra J. (2005) Colour images representations and segmentation, Tutorial at IASTED International Conference on Visualization, Imaging and Image Processing (VIIP 2005), Benidorm, Spain, Sept. 7–9, 2005 (preprint) 29. Sinha D. and Dougherty E.R. (1992) Fuzzy mathematical morphology, Visual Communication and Image Representation, Vol. 3, No. 3, 286–302 30. Sinha D. and Dougherty E.R. (1993) Fuzzification of set inclusion, theory and applications, Fuzzy Sets and Systems, Vol. 55, 15–42 31. Sinha D. and Dougherty E.R. (1995) A general axiomatic theory for intrinsically fuzzy mathematical morphologies, IEEE Transactions on Fuzzy Systems, Vol. 3, No. 4, 389–403 32. Sinha D., Sinha P., Dougherty E.R. and Batman S. (1997) Design and analysis of fuzzy morphological algorithms for image processing, IEEE Transactions on Fuzzy Systems, Vol. 5, No. 4, 570–584 33. Soille P. and Rivest J.F. (1996) On the validity of fractal dimension measurements in image analysis, Journal of Visual Communication and Image Representation, Vol. 7, No. 3, 217–229 34. Soille P., (2002) Morphological image analysis (2nd edition), Springer-Verlag, Berlin.


381

35. Werman M. and Peleg S. (1985) Min - max operators in texture analysis, IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 7, No. 2, 730–733 36. Zadeh L. (1975) Fuzzy sets and their applications to cognitive processes, Academic Press, London

Chapter 14 Intuitionistic Fuzzy Image Processing Ioannis K. Vlachos and George D. Sergiadis Aristotle University of Thessaloniki Faculty of Technology Department of Electrical & Computer Engineering Telecommunications Laboratory University Campus, GR–54124 Thessaloniki, Greece [email protected], [email protected]

Summary The purpose of this chapter is to establish a flexible mathematical framework for image processing, based on the concepts of intuitionistic fuzzy sets theory. The images in the intuitionistic fuzzy environment are composed of components corresponding to membership and indeterminacy functions, associated with image properties. Heuristic and analytical methods are presented that model the inherent uncertainty of images from different points of departure. The modelling is carried out by exploiting and interpreting the inherent ambiguity and vagueness, carried by the image itself, in terms of elements of intuitionistic fuzzy sets. The efficiency of the proposed intuitionistic fuzzy framework is demonstrated in the context of contrast enhancement and edge detection, yielding satisfactory and promising results.

1 Introduction Fuzzy sets (FSs) theory [39] has been successfully applied to many image processing and pattern recognition problems. Fuzzy techniques offer a nonlinear and knowledge-driven framework for coping with the ambiguity and vagueness often present in digital images, associated mostly with the definition of various elements such as edges and regions, or the notions of brightness and contrast. Defining a suitable membership function to describe an image property is not a trivial task, since it depends on various factors that introduce different types of uncertainties, which finally translate into additional uncertainties about the membership function itself [21]. Therefore, we must carefully seek more flexible, or even intuitive, ways in order to model uncertainty. I.K. Vlachos and G.D. Sergiadis: Intuitionistic Fuzzy Image Processing, StudFuzz 210, 383–414 (2007) c Springer-Verlag Berlin Heidelberg 2007 www.springerlink.com

384

I.K. Vlachos and G.D. Sergiadis

Among the various notions of higher-order FSs, intuitionistic fuzzy sets (IFSs) proposed by Atanassov [1, 2, 3] provide a flexible, yet solid, mathematical setting to cope, besides the presence of vagueness, with the hesitancy originating from imperfect or/and imprecise information. IFSs reflect better the aspects of human behavior. A human being who expresses the degree of belongingness of a given element to a set, does not often express correspondingly degree of non-belongingness as the complement to one. This psychological fact states that linguistic negation does not always coincides with logical negation [16]. A survey of the research on IFSs can be found in [22, 23]. This work aims at providing a flexible environment for image processing, based on the elements of IFSs theory. Images are susceptible of bearing uncertainties associated with the intensity levels of pixels. The origin of this type of uncertainty can be traced back to the acquisition chain or to the noisy environment and the imaging mechanisms. The problem of analyzing/synthesizing the image in/from its corresponding intuitionistic fuzzy components is posed and heuristic, as well as analytical ways are proposed for solving it. Heuristic approaches involve modelling of the acquisition and imaging mechanisms by considering their intrinsic limitations and weaknesses. Additionally, analytical techniques optimize the informational content of the image. Moreover, a new operator is defined and its properties are thoroughly investigated. Theoretical results are also supported by various examples from different imaging domains. Finally, from the numerous possible image processing applications, we demonstrate the efficiency of the proposed framework for contrast enhancement and edge detection.

2 Elements of intuitionistic fuzzy sets theory Extending the notion of FS proposed by Zadeh [39], Atanassov [1, 2, 3] introduced the concept of the IFS. However, recently Dubois et al. [12] posed the issue of terminological difficulties in FSs theory and especially in the case of IFSs. Therefore, in order to avoid any misunderstanding between the intuitionistic fuzzy logic of Takeuti and Titani [35] and the IFSs proposed by Atanassov [1, 2, 3], we will hereinafter use the term A–IFSs to refer to Atanassov’s concept of IFSs. A–IFSs are described using two characteristic functions expressing the degree of membership (belongingness) and the degree of non-membership (nonbelongingness) of elements of the universe to the A–IFS. The membership and non-membership values induce an indeterminacy index, which models the hesitancy of deciding the degree to which an object satisfies a particular property. In fact, it is this additional degree of freedom that provides us with a flexible mathematical framework to cope with imperfect or/and imprecise information. In 1993, Gau and Buehrer [15] proposed the notion of vague sets. Bustince and Burillo [6] proved that the concept of vague sets coincides with that of A–IFSs. Finally, it should be mentioned that A–IFSs and interval-

Intuitionistic Fuzzy Image Processing

385

valued fuzzy sets (IVFSs) are isomorphic generalizations of FSs, as proved by Atanassov and Gargov [4] and also emphasized by [9, 11, 13]. Therefore, the presented image processing framework can also be viewed under the scope of IVFSs. 2.1 Preliminaries Definition 1. An FS A˜ defined on a universe X may be given as [39] A˜ = {x, µA˜ (x)|x ∈ X},

(1)

˜ The membership value where µA˜ : X → [0, 1] is the membership function of A. ˜ µA˜ (x) describes the degree of belongingness of x ∈ X in A. Definition 2. An A–IFS A defined on a universe X is given by [1, 2, 3] A = {x, µA (x), νA (x)|x ∈ X},

(2)

where µA : X → [0, 1]

and

νA : X → [0, 1],

(3)

with the condition 0 µA (x) + νA (x) 1,

(4)

for all x ∈ X. The numbers µA (x) and νA (x) denote the degree of membership and the degree of non-membership of x to A respectively. For an A–IFS A in X we call the intuitionistic index of an element x ∈ X in A the following expression πA (x) = 1 − µA (x) − νA (x).

(5)

We can consider πA (x) as a hesitancy degree of x to A [1, 2, 3]. From (5) it is evident that 0 πA (x) 1,

(6)

for all x ∈ X. FSs can also be represented using the notation of A–IFSs. An FS A˜ defined on X can be represented as the following A–IFS A = {x, µA˜ (x), 1 − µA˜ (x)|x ∈ X},

(7)

with πA (x) = 0 for all x ∈ X. A geometrical interpretation of the connection between FSs and A–IFSs can be found in [29]. Throughout this paper by I F S (X) we denote the set of all the A–IFSs defined on X. Similarly, F S (X) is the set of all FSs on X.

386


2.2 Basic operations on A–IFSs Atanassov in [1, 2, 3] defined a set of operations between two A–IFSs A and B. Definition 3. The union operator ∪ between A and B is given by A ∪ B = {x, max{µA (x), µB (x)}, min{νA (x), νB (x)}|x ∈ X} .

(8)

Definition 4. The intersection operator ∩ between A and B is given by A ∩ B = {x, min{µA (x), µB (x)}, max{νA (x), νB (x)}|x ∈ X} .

(9)

Definition 5. The complementary set Ac of A is defined as Ac = {x, νA (x), µA (x)|x ∈ X} .

(10)

Definition 6. A refines B, denoted as A B, iff µA (x) µB (x) or

and

νA (x) νB (x),

for

µB (x) νB (x)

µA (x) µB (x)

and

νA (x) νB (x),

for

µB (x) νB (x)

(11)

for all x ∈ X. 2.3 Intuitionistic fuzzy entropy De Luca and Termini [20] proposed a nonprobabilistic entropy for FSs and formulated the axiomatic requirements an entropy measure should comply with. Szmidt and Kacprzyk [31] extended the axioms of De Luca and Termini, proposing the following definition for an entropy measure in the setting of A– IFSs theory. Definition 7 (Szmidt and Kacprzyk [31]). An entropy on I F S (X) is a real-valued functional E : I F S (X) → [0, 1], satisfying the following axiomatic requirements: • E1: E(A) = 0 iff A is a crisp set; that is µA (xi ) = 0 or µA (xi ) = 1 for all xi ∈ X • E2: E(A) = 1 iff µA (xi ) = νA (xi ) for all xi ∈ X; that is A = Ac • E3: E(A) E(B) if A refines B; i.e. A B • E4: E(A) = E(Ac ) Moreover, in [31], an intuitionistic fuzzy entropy measure was defined as 1 max Count(Ai ∩ Aci ) , n i=1 max Count(Ai ∪ Aci ) n

E(A) =

(12)


387

where n = Cardinal(X) and Ai denotes the single-element A–IFS corresponding to the i-th element of the universe X and is described as Ai = {xi , µA (xi ), νA (xi )}. In other words, Ai is the i-th “component” of A. Moreover, max Count(A) denotes the biggest cardinality of A and is given by max Count(A) =

n

(µA (xi ) + πA (xi )) .

(13)

i=1

3 Towards intuitionistic fuzzy image processing 3.1 Intuitionistic fuzzy model of an image Let us consider an image A of size M ×N pixels, having L gray levels g ranging between 0 and L − 1. When applying FSs for image processing, an image can be considered as an array of fuzzy singletons. Each element of the array denotes the membership value µA˜ (gij ) of the gray level gij , corresponding to the (i, j)-th pixel, with respect to a predefined image property such as brightness, edginess, homogeneity [24, 25, 26]. As a generalization of this approach, we introduce the following representation of an image A in the intuitionistic fuzzy environment. Definition 8. An image A is described by the A–IFS A = {gij , µA (gij ), νA (gij )|gij ∈ {0, . . . , L − 1}} ,

(14)

with i ∈ {1, . . . , M } and j ∈ {1, . . . , N }, where µA (gij ) and νA (gij ) denote respectively the degrees of membership and non-membership of the (i, j)-th pixel to the set A associated with an image property. The functions µA and νA correspond to the membership and non-membership components of the image. If instead of A–IFSs we consider FSs, then Definition 8 reduces to the one given in [24, 25, 26]. 3.2 A heuristic approach to intuitionistic fuzzification of images Image processing techniques based on FSs theory, provide a flexible mathematical framework to cope with “qualitative” properties, such as the contrast or edginess of a region, by modelling, in terms of membership functions, the ambiguity and vagueness often present in digital images. Depending on the desired application, different fuzzification schemes have been proposed in the literature. For example, in the case of contrast enhancement one may consider the property “brightness” of gray levels and construct the FS A˜ = “Bright pixel ” by a simple gray-level normalization or by applying more complex methods, such as the entropy maximization of fuzzy events [8].

388


Gray-Level Domain

Fuzzy Domain

Intuitionistic Fuzzy Domain

Membership component

Input image

Modification

Modification

Gray-Level Domain

Processed fuzzy image

Output image

New membership component

Fuzzy image

Hesitancy component

Fuzzy Domain

New hesitancy component

Fig. 1. Overview of the intuitionistic fuzzy image processing framework

Moving one step further, the task of describing an image in terms of elements of A–IFSs theory is a little more complicated, since it also requires the definition of the non-membership or the hesitancy component of the A–IFS. This additional requirement forces us to seek intuitive ways for intuitionistic fuzzification of images, which will reflect the inherent imprecision or/and imperfection of information carried by the image itself. In the quest for an intuitionistic fuzzy image processing (IFIP) framework, the question that naturally arises, when attempting to define the A–IFS of brightness of pixels, can be stated as: “How can we define the membership and non-membership functions of gray levels to describe an image in the intuitionistic fuzzy domain (IFD)? ”. Or in order to be more consistent with human perception: “How bright is a gray level and how sure are we of it being that bright? ”. Therefore, it seems that defining the membership and hesitancy functions instead of the membership and non-membership ones, suites better our intuitive comprehension of the brightness of an intensity level. So, in which intuitive way can we intuitionistically fuzzify an image and on what factors should hesitancy depend on? 3.3 From images to intuitionistic fuzzy sets Hesitancy in images originates from various factors, which in their majority are due to the inherent weaknesses of the acquisition and the imaging mechanisms. Limitations of the acquisition chain, such as the quantization noise, the suppression of the dynamic range, or the nonlinear behavior of the mapping system, affect our certainty on deciding whether a pixel is “gray” or “edgy” and therefore introduce a degree of hesitancy associated with the corresponding pixel. Therefore, hesitancy should encapsulate the aforementioned sources of indeterminacy that characterize digital images. Defining the membership component of the A–IFS that describes the brightness of pixels in an image, is a more straightforward task that can be carried out in a similar manner as in traditional fuzzy image processing systems. In the presented heuristic framework, we consider the membership value of a gray level g to be its normalized intensity level; that is


(a)

(b)

389

(c)

Fig. 2. Gray-scale test images of size 256×256 pixels with 8 bits-per-pixel gray-tone resolution

µA (g) =

g , L−1

(15)

where g ∈ {0, . . . , L − 1}. It should be mentioned that any other method for calculating µA can also be applied. 3.4 A fuzzy histogram approach to hesitancy modelling Quantization noise is an inextricable part of any physical system that involves analog-to-digital conversion. Consequently, any imaging system is susceptible to quantization errors springing from possible miss-mappings of a gray level g as (g + 1) or (g − 1), without however any significant change for the human perception. In order to model this type of imprecision of gray levels, the concept of fuzzy histogram based on fuzzy numbers was proposed in [17, 18]. A fuzzy number g˜ : R → [0, 1] is an FS of the real line that is normal and convex. We restrict our selection to symmetrical fuzzy numbers, which are conceptually suitable to represent the notion of gray level “approximately g”. A symmetrical triangular fuzzy number can be defined as . |x − g| , (16) µg˜ (x) = max 0, 1 − p where the positive real parameter p controls the shape of the number. By considering gray levels as fuzzy numbers, the notion of histogram can be extended into a fuzzy setting [17, 18]. The fuzzy histogram of a digital image A is a sequence hfA (g) with g ∈ {0, . . . , L − 1} and is defined as hfA (g) ||{(i, j), µg˜ij (g)|i ∈ {1, . . . , M }, j ∈ {1, . . . , N }||,

(17)

where || · || stands for the cardinality of an FS. Moreover, hfA (g) represents the frequency of occurrence of gray level “approximately g”. However, due to

I.K. Vlachos and G.D. Sergiadis 0.03

0.03

0.02

0.02

0.02

0.01

˜ f (g) h A

0.03

˜ f (g) h A

˜ cA (g) h

390

0.01

0

0 0

50

100

150

200

250

0.01 0

0

50

100

150

200

250

0

50

100

150

Gray level g

Gray level g

Gray level g

(a)

(b)

(c)

200

250

Fig. 3. (a) Normalized crisp histogram for the image of Fig. 2(a). Corresponding fuzzy histograms with (b) p = 3 and (c) p = 9

its definition, the fuzzy histogram fails to be a probability density function. Therefore, a normalized fuzzy histogram, that is also a probability density function, can be obtained as f ˜ f (g) = hA (g) , h A L−1 f g=0 hA (g)

(18)

with g ∈ {0, . . . , L − 1}. Figure 3 depicts corresponding crisp and fuzzy normalized histograms for the image of Fig. 2(a) using various values of parameter p. One can observe that as p increases the resulting fuzzy histogram becomes smoother. The impact of quantization errors can be be seen by comparing the crisp and fuzzy histograms. In the case of “hard” first-order statistics there exists a number of gray levels with zero or almost zero frequencies of occurrence due to quantization noise, while gray levels in their vicinity possess high frequencies [17]. This is not the case when considering the fuzzy histogram. Therefore, in order to model the indeterminacy originating out of quantization noise, the hesitancy corresponding to a gray level of the image should be proportional to the normalized absolute difference between the normalized crisp and fuzzy histograms; that is πA (g) ∝

˜ c (g) − h ˜ f (g)| |h A @A A, c ˜ ˜ f (g)| maxg |h (g) − h A

(19)

A

˜ c is the normalized crisp histogram of the image. where h A Since the quantity at the r.h.s. of (19) takes values in [0, 1] and taking into account that by setting νA (g) = 0 into (5), for any particular gray level g, the max (g) that hesitancy can obtain is maximum value πA max (g) = 1 − µA (g), πA

(20)

we deduce that the maximum hesitancy of a gray level decreases as its membership value increases. Therefore, hesitancy must also be proportional to max (g); i.e. πA


0.4

0.6 0.4 0.2

0.5 0.4 πA (g)

0.5

πA (g)

1 0.8

0.3 0.2 0.1

0 50

100

150

200

250

0.3 0.2 0.1

0 0

391

0 0

50

100

150

200

250

0

50

100

150

Gray level g

Gray level g

Gray level g

(a)

(b)

(c)

200

250

Fig. 4. (a) Membership (dashed line) and non-membership (solid line) functions with p = 5 and k = 0.9 for the image of Fig. 2(a). Corresponding degrees of hesitancy for (b) k = 0.9 and (c) k = 0.1 1

0.03

0.5

0.02

0.4 πA (g)

˜ cA (g) h

0.8 0.6 0.4

0.01

0.2 0 50

100

150

200

250

0.2 0.1

0 0

0.3

0 0

50

100

150

200

250

0

50

100

150

Gray level g

Gray level g

Gray level g

(a)

(b)

(c)

200

250

Fig. 5. (a) Normalized crisp histogram for the image of Fig. 2(b). Corresponding (b) membership (dashed line) and non-membership (solid line) functions with p = 5 and k = 0.9 and (c) hesitancy function

πA (g) ∝ (1 − µA (g)),

(21)

which ensures that the constraint imposed by (20) is satisfied. The aforementioned mathematical constraint turns out to describe adequately a very interesting property of physical systems. Due to presence of noise in any imaging system, lower gray levels are more affected than higher ones. Therefore, hesitancy associated with lower gray levels should by definition be larger than the one corresponding to higher levels. This fact is well modelled by the assumption described by (21). Finally, the hesitancy index associated with a gray level should also be a decreasing function of the normalized dynamic range of the image, since the more suppressed the dynamic range the less certain we are regarding the actual grayness of the intensity levels. The normalized dynamic range is given by ∆r =

gmax − gmin , L−1

(22)

where gmin and gmax are the minimum and maximum gray levels of the image respectively. Combining all the aforementioned assumptions, we model the hesitancy πA (g) corresponding to gray level g of an image A, defined as in (14), with

392

I.K. Vlachos and G.D. Sergiadis 1

0.1

0.5

0.8 πA (g)

0.4

˜ cA (g) h

0.6 0.4

0 0

50

100

150

200

250

0.2 0.1

0.2 0

0.3

0 0

50

100

150

200

250

0

50

100

150

Gray level g

Gray level g

Gray level g

(a)

(b)

(c)

200

250

Fig. 6. (a) Normalized crisp histogram for the image of Fig. 2(b). Corresponding (b) membership (dashed line) and non-membership (solid line) functions with p = 5 and k = 0.9 and (c) hesitancy function

the following formula πA (g) = (1 − µA (g))

˜ c (g) − h ˜ f (g)| |h A @A A (1 − k∆r), c ˜ ˜ f (g)| maxg |h (g) − h A

(23)

A

where k ∈ (0, 1). Parameter k controls the overall influence of the dynamic range to the hesitancy function. One can easily verify that the hesitancy degree computed by (23) satisfies the constraint described by (6). Consequently, the A–IFS associated with the image A is defined considering the membership and hesitancy components of (15) and (23) respectively. In order to demonstrate the concepts of the heuristic environment for image processing in the intuitionistic fuzzy setting, we considered the gray-scale images of Fig. 2. Fig. 4(a) illustrates the membership (dashed line) and nonmembership (solid line) functions corresponding to the image of Fig. 2(a) with p = 5 and k = 0.9. The membership function was calculated using the normalization of the intensity levels as described by (15). Moreover, Figs. 4(b) and 4(c) depict the hesitancy functions for k = 0.9 and k = 0.1 respectively. Furthermore, let us consider the image of Fig. 2(b) with a corresponding normalized crisp histogram as shown in Fig. 5(a). For p = 5 and k = 0.9 the membership and non-membership functions are illustrated in Fig. 5(b), while the hesitancy index is shown in Fig. 5(c). Finally, a more intense impact of inherent hesitancy is demonstrated in Fig. 6 for the low-contrasted image of Fig. 2(c). The increased degree of hesitancy originates from the low dynamic range and the large concentration of pixels around a single bin of the histogram.

4 The intuitionistic fuzzy image processing framework In the previous section we dealt with the problem of analyzing an image in its intuitionistic fuzzy components, by modelling the factors of the acquisition and imaging chains that introduce uncertainty in the image under processing. The purpose of the following analysis concerns the same problem, but from


393

another point of view. Given an image of unknown origin and without an a priori knowledge of the imaging mechanisms and their nonlinear nature, we seek and propose optimal ways for defining the corresponding A–IFS, as well as for embedding the modelled hesitancy in the processed image. 4.1 Fuzzification: from gray levels to membership functions The first stage of the IFIP framework, illustrated in Fig. 1, involves the transfer of the image from the gray-level domain (GLD) to the fuzzy domain (FD) by defining a suitable membership function µA˜ . As also stated in Sect. 3.3, different fuzzification schemes can be applied, from a simple intensity normalization to the intuitive definition of membership by human-expert knowledge. 4.2 Intuitionistic fuzzification: the quest for hesitancy Having specified the FS associated with an image, the answer to the question of defining the A–IFS corresponding to the image is now relaxed: “Which A–IFS, from the multitude that can be generated, is optimum for the specific image? ”. In [7] the concept of intuitionistic fuzzy generators (IFGs) was introduced and methods for constructing A–IFSs from FSs were presented. Intuitionistic fuzzy generators Definition 9 (Bustince et al. [7]). A function ϕ : [0, 1] → [0, 1] will be called intuitionistic fuzzy generator (IFG) if ϕ(x) 1 − x

(24)

for all x ∈ X. The first characterization theorem for IFGs is stated as follows. Theorem 1 (Bustince et al. [7]). Let ϕ : [0, 1] → [0, 1]. Then, ϕ is a continuous IFG if and only if there exists a continuous function f : [0, 1] → [0, 1] such that • f (x) x for all x ∈ [0, 1], • ϕ(x) = (f ◦ N )(x) for all x ∈ X, where N denotes the standard negation, N : [0, 1] → [0, 1] given by N (x) = 1 − x for all x ∈ X. Furthermore, an A–IFS can be constructed from an FS and an IFG according to the following theorem. Theorem 2 (Bustince et al. [7]). Let A˜ be an FS on the universe X = ∅, and let ϕ be an IFG. Then, the set A = {xi , µA˜ (xi ), ϕ(µA˜ (xi ))|xi ∈ X} is an A–IFS on X.

(25)

394


However, a procedure for obtaining the optimal A–IFS, corresponding to the FS A˜ constructed in the fuzzification stage, is required. Optimization of intuitionistic fuzzy entropy Let us consider the continuous function f (x) = xλ , with λ 1. It is evident that since λ 1, it holds that f (x) x and f : [0, 1] → [0, 1] for all x ∈ [0, 1]. Hence, according to Theorem 1 and employing the standard negation N , the following IFG can be constructed ϕ(x) = (1 − x)λ ,

(26)

where x ∈ [0, 1] and λ 1. The selection of the proposed IFG is justified in terms of satisfying the intuitive assumptions considered in Sect. 3, regarding the nonlinear behavior of the imaging mechanisms. For the intuitionistic fuzzification of the image, the FS A˜ specified in the fuzzification stage of the IFIP framework will serve as the basis for constructing the corresponding A–IFS. However, defining the proper A–IFS to model the hesitancy present in a digital image, must encompass optimization of the ˜ as well as suitable non-membership function generation. Motivated by FS A, (26) and taking into account the involution property of fuzzy complements, the membership function of the FS A˜ is tuned according to

µ Â (g) = 1 − (1 − µA˜ (g))λ ,

(27)

where λ 0. Using the same parameter λ in both (26) and (27) and considering also µ Â as a function of λ, (27) can be rewritten as µ Â (g; λ) = 1 − (1 − µA˜ (g))λ−1 ,

(28)

with λ 1. According to Theorem 2 and considering the IFG of (26), it holds that the non-membership function is given by νÂ (g; λ) = ϕ (ˆ µA (g; λ)) = (1 − µA˜ (g))λ(λ−1) ,

(29)

for all g ∈ {0, . . . , L − 1}, where λ 1. By varying the parameter λ different A–IFSs can be generated. Therefore, in the case of image A, the intuitionistic fuzzy entropy of (12) can also be consider as a function of λ; that is E(A; λ) =

L−1 1 − max {ˆ µA (g; λ), νÂ (g; λ)} 1 c , hA˜ (g) M N g=0 1 − min {ˆ µA (g; λ), νÂ (g; λ)}

(30)

where hcA˜ is the crisp histogram of the image after fuzzification. Applying the maximization of intuitionistic fuzzy entropy principle, the entropy of (30) will serve as the optimization criterion in order to select the optimal parameter λopt that produces the A–IFS that best models the image; i.e.


(a)

(b)

(c)

395

(d)

Fig. 7. (a) Gray-scale test image with its corresponding optimal (b) membership, (c) non-membership, and (d) hesitancy components 1

0.03

E(A; λ)

˜ c˜ (g) h A

0.8 0.02 0.01

0.6 0.4 0.2

0

0 0

50

100 150 200 Gray level g

0

250

5

10

15

20

25

30

λ (b)

(a) 1

0.8

0.8 πAopt (g)

1

0.6 0.4 0.2

0.6 0.4 0.2

0

0 0

50


0

250

50


250

(d)

(c)

Fig. 8. (a) Normalized crisp histogram for the image of Fig. 7(a). (b) Intuitionistic fuzzy entropy curve, used for obtaining the optimal parameter λopt for transferring the image in the IFD. (c) Membership (dashed line), non-membership (solid line), and (d) hesitancy optimal functions describing the brightness of gray levels in the image

λopt = arg max{E(A; λ)}. λ1

Therefore, the optimal A–IFS Aopt associated with image A is

(31)

396


B C Aopt = g, µAopt (g), νAopt (g) |g ∈ {0, . . . , L − 1} ,

(32)

where µAopt (g) = 1 − (1 − µA˜ (g))

λopt −1

(33)

and νAopt (g) = (1 − µA˜ (g))

λopt (λopt −1)

.

(34)

It should be mentioned that also other IFGs with different characteristics can be used to obtain an optimal A–IFS Aopt , with respect to the IFG employed. Figure 7 depicts a gray-scale image with its corresponding membership, non-membership and hesitancy components. The intuitionistic fuzzy entropy curve used for obtaining λopt is illustrated in Fig. 8(b), while Figs. 8(c) and 8(d) show the optimal functions of the A–IFS Aopt describing the brightness of gray levels in the image. 4.3 Modification of intuitionistic fuzzy components After obtaining the optimal representation of the image in the IFD, the modification of its membership and non-membership components is carried out and the new components, denoted as µAopt and νAopt , are derived. The selection of the appropriate transformation functions is application-dependent. 4.4 Intuitionistic defuzzification: embedding hesitancy In order to transfer the image from the IFD back to the FD, the proper FS corresponding to the modified set Aopt must be obtained. Atanassov [3] proposed an operator for the “de-construction” of an A–IFS into an FS. Definition 10. If A ∈ I F S (X), then Dα : I F S (X) → F S (X), where Dα (A) = {xi , µA (xi ) + απA (xi ), νA (xi ) + (1 − α)πA (xi )|xi ∈ X},

(35)

with α ∈ [0, 1]. It should be mentioned that the family of all FSs associated with A by the operator Dα , denoted by {Dα (A)}α∈[0,1] , constitutes a totally ordered family of FSs. We will call Dα the Atanassov’s operator . Different values of α generate different FSs and therefore different representations of the processed image in the FD are possible. Thus, a criterion must be employed in order to select the optimal parameter αopt . In image processing it is sometimes useful to increase the grayness ambiguity by increasing the number of gray levels [37]. Images with high fuzziness are considered to be more suitable in terms of human brightness perception. The linear index of fuzziness [19] is a measure of the intrinsic fuzziness of an


397

FS. Moreover, since fuzzy entropy is also a measure of fuzziness, increasing the fuzziness of an image results in increasing its informational content. In ˜ the linear index of fuzziness is defined as the case of an FS A, |X|

˜ = γl (A)

1 min {µA˜ (xi ), 1 − µA˜ (xi )} , 2|X| i=1

(36)

where |X| = Cardinal(X). Substituting the min t-norm with the product operator yields |X|

˜ = γ(A)

1 µ ˜ (xi )(1 − µA˜ (xi )). 4|X| i=1 A

(37)

The index of fuzziness will serve as the optimization criterion to obtain the optimal FS corresponding to the A–IFS Aopt . From (35) and (37) we obtain γ Dα (Aopt ) =

L−1 1 c h (g)µDα (Aopt ) (g)(1 − µDα (Aopt ) (g)). 4M N g=0 A˜

(38)

Setting dγ Dα (Aopt ) = 0, dα

(39)

yields that L−1 αopt =

g=0

hcA˜ (g)πAopt (g) 1 − 2µAopt (g) . L−1 2 2 g=0 hcA˜ (g)πA (g)

(40)

opt

Moreover, since MN d2 γ Dα (Aopt ) 1 c 2 = − h (g)πA (g) 0, opt dα2 2M N g=0 A˜

(41)

it is evident that the extremum of (38) is a global maximum. However, (40) does not guarantee that aopt will lie in the [0, 1] interval, since (38) is possible to attain a maximum outside of [0, 1]. Consequently, the optimal parameter αopt used to transfer the image from the IFD to the FD is obtained as   if αopt < 0, 0, αopt = αopt , if 0 αopt (42) 1,   1, if αopt > 1. An important proposition that is going to be used when applying the IFIP framework for contrast enhancement is stated and proved.

398


Proposition 1. Let A = {x, µA (x), νA (x)|x ∈ X} be an A–IFS, with µA (x) and νA (x) being increasing/decreasing and decreasing/increasing functions of x respectively. Then, the membership function of the FS Dα (A) retains the monotonicity of µA (x). Proof. From (35) and taking into account (5), the membership function of Dα (A) can be written as µDα (A) (x) = α + (1 − α)µA (x) − ανA (x).

(43)

The derivative of (43) with respect to x is dµDα (A) (x) dµA (x) dνA (x) = (1 − α) −α . dx dx dx

(44)

Let us suppose that µA (x) and νA (x) are increasing and decreasing functions A (x) 0 (increasing function) of x respectively. Then, since α ∈ [0, 1] and dµdx dνA (x) and dx 0 (decreasing function), it follows immediately that dµDα (A) (x) 0. dx

(45)

Hence, µDα (A) is an increasing function of x. Similarly in the case when µA (x) + and νA (x) are decreasing and increasing functions of x respectively. * Finally, the optimal FS corresponding to the modified A–IFS Aopt is defined as @ A Aõpt = g, µDαopt (Aopt ) (g)|g ∈ {0, . . . , L − 1} . (46) 4.5 Defuzzification: from sets to images In the defuzzification stage of the IFIP framework, the image, from its representation using FSs, is transferred back to the GLD. Different defuzzification schemes can be applied, described in general as > ? (47) g = t µDαopt (Aopt ) (g) , where g are the new gray levels of the image and t[·] a transformation function t : [0, 1] → {0, . . . , L − 1}.

5 Intuitionistic fuzzy histogram hyperbolization Contrast enhancement refers to a wide family of algorithms, which aim to increase the contrast in order for the image to be more suitable for human perception or for a specific application. One of the most common approaches


399

to contrast enhancement is histogram equalization, in which we seek for a gray-level transformation in order for the histogram of the resulting image to approximate to a uniform distribution over the gray-level range. However, due to the nonlinear nature of human brightness perception, the requirement of a uniform histogram is not very suitable. Based on this fact, Frei [14] proposed the idea of histogram hyperbolization, where the underlying model for equalization is logarithmic. Extending the histogram hyperbolization into the fuzzy setting, Tizhoosh [38] proposed the method of fuzzy histogram hyperbolization (FHH). In this approach, for an image A the membership values µA˜ (g) of the gray levels are modified by a logarithmic function, in order to generate the new gray levels; that is ? L − 1 > −(µA˜ (g))β e −1 , (48) g = −1 e −1 where g is the new gray level and β is the fuzzifier controlling the enhancement procedure. However, the selection of β depends on the image and may often lead to over- or under-enhancement. Typical values of β are in the range of [0.5, 2]. As already mentioned, assigning membership values to gray levels is a task characterized by uncertainty. Using the proposed IFIP framework seems to be a promising and flexible alternative to raise this inherent hesitancy. In the first stage of the proposed intuitionistic fuzzy histogram hyperbolization (IFHH) technique, the image is fuzzified according to µA˜ (g) =

g − gmin , gmax − gmin

(49)

where gmin and gmax are the minimum and maximum gray levels of the image respectively. Using (49) to transfer the image to the FD has the advantage of stretching out the membership function over the total gray level range. Using the maximum intuitionistic fuzzy entropy principle described in Sect. 4.2, we obtain the intuitionistic fuzzy representation of the image as the A–IFS Aopt . One can observe that in the fuzzy approach to histogram hyperbolization, the membership values are modified according to µA˜ (g) = (µA˜ (g)) , β

(50)

for all g ∈ {0, . . . , L − 1}. In [10], De et al. proposed an operator An : I F S (X) → I F S (X) given by n

n

An = {x, (µA (x)) , 1 − (1 − νA (x)) |x ∈ X} ,

(51)

where n is any positive real number. In the case that A is an FS, the operator of (51) coincides with the one used for modifying the membership values in the FHH approach. Therefore, in the proposed IFHH method, the modification of

400


(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Fig. 9. (a) Initial highly low-contrasted image (adopted from [36]) and (e) its histogram equalized version. Images obtained using the FHH approach for (b) β = 0.5, (c) β = 1, and (d) β = 2. Resulting images using the proposed IFHH approach for (f) β = 0.5, (g) β = 1, and (h) β = 2

the A–IFS describing the image in the IFD is carried out using the operator An ; that is @ A β β Aopt = g, µAopt (g) , 1 − 1 − νAopt (g) |g ∈ {0, . . . , L − 1 . (52) The image is transferred back to the FD by selecting the optimal parameter αopt of Atanassov’s operator that maximizes the index of fuzziness of the resulting image. Finally, the image in the GLD is obtained as g =

? L − 1 > −µDαopt (Aopt ) (g) − 1 . e e−1 − 1

(53)

It should be mentioned that in contrast enhancement the employed transformation of the gray levels should be an increasing function, in order to preserve their ordering. Since, the operator of (51) preserves monotonicity of its arguments and the functions (28) and (29), used to obtain the image in the IFD, are increasing and decreasing functions of g respectively, then from Proposition 1 it is ensured that the produced FS, from which the gray level transformation will result, will preserve the ordering of intensity levels.


401

In order to demonstrate the efficiency of the proposed framework, we applied the IFHH scheme to the highly low-contrasted image of Fig. 9(a) adopted from [36]. Figs. 9(b)–9(d) depict images obtained from the FHH approach for β = 0.5, 1 and 2 respectively, while the images derived using its intuitionistic fuzzy extension are illustrated in Figs. 9(f)–9(h). Moreover, the result of the histogram equalization is shown in Fig. 9(e). Comparing the images, one may observe that the proposed intuitionistic fuzzy scheme significantly enhances the contrast of the initial image without being seriously affected by the choice of the value of β. Different image regions and high-frequency edges have been drastically enhanced, revealing hidden image details. On the contrary, the FHH method is seriously affected by the selection of β. But also even if the tuning of β is carried out manually to produce the best outcome, the results of the IFHH continue to outperform the ones derived from FHH and histogram equalization, without any adjustment of β. This result confirms the adaptability and efficiency of the proposed IFIP framework and also demonstrates that the proposed algorithm is not parameter-sensitive.

6 Intuitionistic fuzzy contrast intensification 6.1 Minimization of fuzziness contrast enhancement Image fuzziness is a measure of grayness ambiguity associated with image pixels. It is some times required to reduce the amount of fuzziness present in an image, in order to enhance the contrast between bright and dark regions. A contrast enhancement algorithm based on minimization of fuzziness was proposed in [25]. In this approach, the image is fuzzified according to µA˜ (g) =

gmax − g 1+ Fd

−Fe ,

(54)

where gmax denotes the maximum gray level desired, and Fe , Fd are the exponential and denominational fuzzifiers respectively, which control the ambiguity in the fuzzy plane. Fuzzifier Fd can be determined by the crossover point as Fd =

gmax − g◦ , 1 −1 Fe − 1 2

(55)

with g◦ being the gray level associated with the crossover point. The modification of the membership values is carried out using the intensification operator [40] given by 2 2 (µA˜ (g)) if 0 µA˜ (g) 12 , T1 (µA˜ (g)) = (56) 2 1 − 2 (1 − µA˜ (g)) if 12 µA˜ (g) 1.

402


Successive applications of the intensification operator T are possible, according to the following scheme Tr (µA˜ (g)) = T1 {Tr−1 (µA˜ (g))} ,

(57)

where r = 1, 2, . . . , which result in further decreasing the fuzziness of the image. In the limiting case of r → ∞, Tr produces a two-level (binary) image. After the modification of the membership values has taken place, the defuzzification is performed according to   if g¯ < 0, 0 g = g¯ (58) if 0 g¯ 255,   255 if g¯ > 255, where g is the new gray level and g¯ is obtained from the inverse of the membership function as −1 F e g¯ = gmax − Fd (µA (g)) −1 . (59) In order to exploit the ambiguity present in digital images and the difficulties in precisely defining the membership function, an intensification-like operator for A–IFSs is introduced and its properties are stated and proved. 6.2 A novel A–IFS operator for contrast enhancement Analogously to FSs, we define the crossover point of a set A ∈ I F S (X) to be that point of the universe for which µA (x) = νA (x). The purpose is to construct an intuitionistic fuzzy operator that alters membership and nonmembership values with respect to the crossover point. Theorem 3. Let a set A ∈ I F S (X). Then, Am = {xi , µAm (xi ), νAm (xi )|xi ∈ X}, where

and

 m µA (xi )    m−1 νA (xi ) µAm (xi ) = m (1 − µA (xi ))   1 − m−1 (1 − νA (xi ))  m (1 − νA (xi ))   1 − m−1 (1 − µA (xi )) νAm (xi ) = m ν (x )  i A   m−1 µA (xi )

(60)

if µA (xi ) νA (xi ), (61) if µA (xi ) νA (xi ),

if µA (xi ) νA (xi ), (62) if µA (xi ) νA (xi ),

with m 0 being a real parameter, is an A–IFS.


403

Proof. In order for Am to be an A–IFS it should satisfy the constraints of (3) and (4). Let us assume that µA (xi ) νA (xi ). Clearly, µAm (xi ) 0. m Moreover, since µA (xi ) νA (xi ) and m 0, it holds that µm A (xi ) νA (xi ). m−1 m−1 m Dividing both sides by νA (xi ) yields µA (xi )/νA (xi ) νA (xi ) 1; that is µAm (xi ) 1. In a similar manner, for the non-membership function it holds that m

(1 − νA (xi )) / (1 − µA (xi ))

m−1

1 − νA (xi ) 1,

from which it follows that νAm (xi ) 0. Moreover, since m

(1 − νA (xi )) / (1 − µA (xi ))

m−1

1,

it is evident that νAm (xi ) 1. Therefore, µAm (xi ), νAm (xi ) ∈ [0, 1]. In order for Am to be an A–IFS, it should also satisfy the constraint of (4); that is µAm (xi ) + νAm (xi ) 1, which can be rewritten as 1 1 m m [µA (xi ) (1 − µA (xi ))] [νA (xi ) (1 − νA (xi ))] , 1 − µA (xi ) νA (xi )

(63)

for µA (xi ) νA (xi ). Since µA (xi ) + νA (xi ) 1 and also the function f (x) = xm is increasing for m 0, it suffices to prove that µA (xi ) (1 − µA (xi )) νA (xi ) (1 − νA (xi )) . (64) Moreover, (64) can be rewritten as µA (xi ) − 12 νA (xi ) − 12 , which obviously holds since µA (xi ) νA (xi ) and µA (xi ) + νA (xi ) 1. + The same can be proved if µA (xi ) νA (xi ). * Clearly, it holds that A0 = Ac

A1 = A.

and

(65)

1

1

0.8

0.8

0.6 0.4 0.2

πAm (x)

1 0.8 νAm (x)

µAm (x)

Fig. 10 illustrates the effect of the -operator on the membership, nonmembership and hesitancy functions of an A–IFS. The proposed -operator offers an interesting property to the entropy of an A–IFS, as shown by the following proposition.

0.6 0.4 0.2

0 2

4

6

8

10

0.4 0.2

0 0

0.6

0 0

2

4

6

8

10

0

2

4

6

x

x

x

(a)

(b)

(c)

8

10

Fig. 10. Behavior of the -operator on the (a) membership, (b) non-membership, and (c) hesitancy functions of an A–IFS (solid line) for m = 2 (dashed line) and m = 4 (dotted line)

404


Proposition 2. For any set A ∈ I F S (X) it holds that E(Am ) E(A),

(66)

with m 1. Proof. From property E4 of entropy, it suffices to prove that Am A. Assume that µA (xi ) νA (xi ). Then, since m 1 we have that m−1 µm−1 (xi ) νA (xi ). A

Multiplying both sides by µA (xi ) yields m−1 µm (xi ) µA (xi ); A (xi )/νA

that is µAm (xi ) µA (xi ). The same can be proved for the non-membership function. Similarly, if µA (xi ) νA (xi ) it can easily be proved that µAm (xi ) µA (xi ) and νA (xi ) νA (xi ). Hence, Am A.

+ *

Lemma 1. The family of all A–IFSs associated with A by the -operator, denoted by {Am }m1 , constitutes a family of ordered A–IFSs according to their intuitionistic fuzzy entropy. 6.3 Application to contrast intensification Let us consider an image A of size M × N pixels having L gray levels g ranging between 0 and L − 1. The image is transferred to the FD using the membership function µA˜ of (49). The optimal A–IFS Aopt associated with image A is constructed using the optimization of intuitionistic fuzzy entropy method described in Sect. 4.2. Applying the proposed -operator iteratively, the modification of the image in the IFD is carried out using the following recursive scheme B C m (67) r Am opt = 1 r−1 Aopt , where r = 1, 2, . . . denotes the number of successive applications of the operator. The image is then transferred back to the FD using Atanassov’s operator and the maximization of index of fuzziness criterion. The representation of the processed image in the FD is given as the FS with membership function µA˜ (g) = µr Am (g) + αopt πr Am (g), opt opt

(68)


(a)

(b)

(c)

405

(d)

Fig. 11. (a) Initial highly low-contrasted image. Images obtained using (b) histogram equalization, (c) minimization of fuzziness, and (d) proposed intuitionistic fuzzy contrast intensification technique for m = 1.5 and one iteration

for all g ∈ {0, . . . , L − 1}. Finally, the contrast-enhanced image in the GLD is obtained as g = (L − 1)µA˜ (g),

(69)

where g are the transformed gray levels of the image. Figure 11 illustrates the results obtained using the proposed approach (Fig. 11(d)), its fuzzy counterpart (Fig. 11(c)), as well as histogram equalization technique (Fig. 11(b)). Since in general in contrast enhancement there is not an a priori knowledge regarding the nature of the images under processing, in the case of the minimization of fuzziness method the crossover point was associated with the midpoint of the gray-level range; that is g◦ = 128. The fuzzifier Fe was set to Fe = 2 and gmax = 255. Moreover, in both fuzzy and intuitionistic fuzzy approaches we considered one iteration, while the parameter of the -operator was set to m = 1.5. This selection of m decreases the intuitionistic fuzzy entropy of the optimal A–IFS and thus the overall hesitancy and fuzziness carried by the image. Larger values of m have the effect of more intensively moving image gray levels towards higher and lower intensity levels. From the comparison of the images of Fig. 11, one may observe that the proposed method successfully enhances the contrast of the initial highly low-contrasted image. 6.4 Application to mammographic imaging Mammography is a specific type of medical imaging that uses a low-dose X– ray source and high resolution film, for breast cancer diagnosis. Mammograms often suffer from low contrast and also from a complicated structured background. It is therefore important for the diagnosis procedure to enhance the contrast of images in order for the human expert to discriminate between normal and suspected malignant tissue. Additionally, the presence of artifacts

406


originating from identification labels, radiopaque markers, as well as from wedges, often drastically influence the contrast enhancement procedure. Due to all the aforementioned factors, the uncertainty associated with mammographic images is increased. Therefore, it seems that applying the proposed contrast intensification technique based on A–IFSs theory, will be suitable for the pre-processing of mammograms. A study for fuzzy techniques in mammographic image processing was presented in [27]. In order to evaluate the proposed method using the intuitionistic intensification operator, different test images from the Digital Database for Mammographic Image Analysis (MIAS) were selected [28]. The proposed method was compared with its fuzzy counterpart, as well as with the histogram equalization technique. In the fuzzy approach, a critical point is the selection of the gray level g◦ associated with the crossover point. This selection soundly affects the enhancement procedure. Figs. 12(b) and 12(c) depict the results of the minimization of fuzziness algorithm for two extreme values of g◦ = 10 and g◦ = 240. One can observe that the resulting images are strongly affected by the choice of g◦ . Additionally, the variability in tissue density and its inhomogeneous nature, combined also with the fact that no gray-level reference is known a priori, suggests that a safe initial selection of g◦ would be the midpoint of the gray-level range; that is g◦ = 128. Figs. 13 and 14 illustrate mammographic images processed with different techniques, namely the histogram equalization, the minimization of fuzziness algorithm, and its intuitionistic fuzzy extension. From the results one can observe that the proposed method based on the newly defined -operator, successfully enhances the contrast of the initial images. Different regions and high-frequency edges, have been drastically increased, resulting in images that are more appropriate for examination by the human expert. Moreover, the flexibility of the pro-

(a)

(b)

(c)

(d)

Fig. 12. (a) Initial mammographic image (MIAS database name: mdb010) with fatty background tissue and a well-defined circumscribed benign mass. Images obtained using the minimization of fuzziness algorithm with Fe = 2, gmax = 255 and one iteration for (b) g◦ = 10 and (c) g◦ = 240. (d) Image obtained using the proposed intuitionistic fuzzy scheme for m = 2 and one iteration


(a)

(b)

(c)

407

(d)

Fig. 13. (a) Initial mammographic image (MIAS database name: mdb015) with fatty background tissue and a well-defined circumscribed benign mass. Images obtained using (b) histogram equalization, (c) minimization of fuzziness with Fe = 2, gmax = 255, g◦ = 128, and one iteration. (d) Image obtained using the proposed intuitionistic fuzzy scheme for m = 2 and one iteration

(a)

(b)

(c)

(d)

Fig. 14. (a) Initial mammographic image (MIAS database name: mdb179) with dense background tissue and spiculated malignant masses. Images obtained using (b) histogram equalization, (c) minimization of fuzziness with Fe = 2, gmax = 255, g◦ = 128, and one iteration. (d) Image obtained using the proposed intuitionistic fuzzy scheme for m = 2 and one iteration

posed IFIP framework in handling adequately different types of images is also demonstrated.

7 Edge detection driven by hesitancy Tracing the edges between an object and its background is an important task in image segmentation, as well as in image interpretation and understanding. However, edge detection techniques are sensitive to noise. Therefore, smoothing is often applied in the pre-processing stage, which has also the side-effect of destroying edges. Hence, there is a tradeoff between detailed edge extraction and noise cancellation.

408


Answering the question of whether a pixel is edgy or not, undoubtedly bears a degree of hesitancy. The underlying idea of the proposed intuitionistic fuzzy edge detection scheme, originates from the fact that different edge detectors treat edges in different ways. For an image A of size M × N having L gray levels g ∈ {0, . . . , L − 1}, let us consider two edge maps G1 and G2 , with G1 , G2 ∈ [0, 1], as the outcome of two different edge detectors. An “ideal” edge should be clearly identified by both edge detectors. The opposite must hold for regions of constant intensity. It is therefore meaningful and intuitively correct to interpret the absolute difference between the assessment of the two edge detectors for a specific pixel, as the hesitancy associated with that pixel belonging to an edge or not. Consequently, we propose to model the hesitancy component of the A–IFS A = “Edgy pixel ” as πA (gij ) = |G1 (gij ) − G2 (gij )|.

(70)

It is evident that πA (gij ) ∈ [0, 1]. Having specified the hesitancy function of the A–IFS A the problem of constructing its membership component is still open. Burillo and Bustince [5] proposed the following theorem for constructing an A–IFS from an FS and a hesitancy function. Let A˜ ∈ F S (X). Let us further consider the mappings πA˜ : X → [0, 1] with πA˜ (x) 1 − µA˜ (x) for all x ∈ X. With these conditions: Theorem 4 (Burillo and Bustince [5]). Let φ : F S (X) → I F S (X), with ˜ = {x, µ ˜ (x), ν ˜ (x), |x ∈ X} φ(A) φ(A) φ(A)

(71)

such that ˜ (i) µφ(A) ˜ (x) with fixed a, b ∈ R for all A ∈ F S (X), ˜ (x) = a + bµA (ii) πφ(A) ˜ (x) for all x ∈ X, ˜ (x) = πA ˜ = A. ˜ (iii) if πA˜ (x) = 0 for all x ∈ X, then φ(A) Then, (i’) µφ(A) ˜ (x), ˜ (x) = µA (ii’) νφ(A) ˜ (x) − πA ˜ (x) for all x ∈ X ˜ (x) = 1 − µA and conversely, that is, if (i’) and (ii’) are satisfied, then (i)–(iii) are satisfied as well. For the membership function of the A–IFS to capture the information carried by both edge detectors involved, we chose to calculate the membership as the product between edge detector G1 and G2 . More generally, instead of their product operator, we can consider the membership value of the (i, j)-th pixel to be their intersection modelled by a t-norm T (·, ·); that is µA (gij ) = T (G1 (gij ), G2 (gij )) .

(72)


409

According to Theorem 4, in order for the membership and hesitancy functions of (70) and (72) respectively to determine a proper A–IFS, they must satisfy the constraint πA (gij ) 1 − µA (gij ) for all i ∈ {1, . . . , M } and j ∈ {1, . . . , N }. ˜ be two FSs in X and T a t-norm. Then, Theorem 5. Let A˜ and B ˜ ˜ ˜ ˜ |A(x) − B(x)| + T A(x), B(x) 1

(73)

(74)

for all x ∈ X. ˜ ˜ Proof. Let us assume that A(x) B(x). Then, from (74) we obtain that ˜ ˜ ˜ ˜ T (A(x), B(x)) 1 + A(x) − B(x). However, for any t-norm it holds that ˜ ˜ ˜ ˜ ˜ ˜ T (A(x), B(x)) min{A(x), B(x)} for all x ∈ X. Then, if A(x) B(x), we ˜ ˜ ˜ ˜ derive that −A(x) + B(x) + T (A(x), B(x)) 1. Similarly, in the case when ˜ ˜ ˜ ˜ ˜ ˜ A(x) B(x), we obtain that A(x) − B(x) + T (A(x), B(x)) 1. * + Therefore, based on Theorem 5, the membership and hesitancy functions of (70) and (72) respectively, define a proper A–IFS. In the same manner, one can verify that constructing the hesitancy function based on the edge detectors G1 and G2 as πA (gij ) = τ |G1 (gij ) − G2 (gij )|,

(75)

where τ ∈ [0, 1], also defines a proper A–IFS. Parameter τ controls the amount of hesitancy employed when constructing the A–IFS. Evidently, if τ = 0 then we obtain an FS. According to Proposition 2, for values of m 1 the intuitionistic fuzzy entropy decreases. Moreover, from (65) and property E3 of entropy, it is deduced that the -operator for m = 0 and m = 1 delivers A–IFSs with the same entropy. Therefore, it is expected that values of m in the range of (0, 1), will increase the entropy of the A–IFS corresponding to the image, and thus its informational content. The task of edge detection can be considered as a decision-making problem, in which we are faced with the question whether a pixel is edgy or not. In [32, 33, 34], a similarity measure for A–IFSs was proposed using a geometrical representation of A–IFSs. For two sets A, B ∈ I F S (X), the similarity between them is defined as dIF S (A, B) , (76) Sim(A, B) = dIF S (A, B c ) where dIF S (A, B) is their normalized Hamming distance, given by [30] |X|

1 |µA (xi ) − µB (xi )| + |νA (xi ) − νB (xi )| 2|X| i=1 + |πA (xi ) − πB (xi )| ,

dIF S (A, B) =

(77)

410


where |X| = Cardinal(X). The similarity measure of (76) has the property of selecting objects that are more similar than dissimilar [34], since it involves a ratio of similarity and dissimilarity measures. An “ideal ” edge in terms of A–IFSs theory can be represented by the A– IFS “Ideal edge” = I = {gij , 1, 0|gij ∈ {0, . . . , L−1}}, for all i ∈ {1, . . . , M } and j ∈ {1, . . . , N }. Therefore, applying the similarity measure of (76) for comparing the sets A and I , will yield an estimation of how edgy a pixel is and therefore will produce the edge map of the image under processing, considering the properties of both edge detectors and the nonlinear behavior of the similarity measure. The application of the similarity measure is carried out pixel-wisely; that is Sim(Aij , I¯), where Aij denotes the (i, j)-th component of the A–IFS A and I¯ = {gij , 1, 0} for all i ∈ {1, . . . , M } and j ∈ {1, . . . , N }. From (76) we have that 0 Sim(A, B) ∞. However, as suggested in [33], when comparing two objects, in this case the sets “Edgy pixel ” and “Ideal edge”, only values of the similarity measure in [0, 1] are meaningful and thus of interest. Moreover, if Sim(Aij , I¯) = 0 means the identity of Aij and I¯, while Sim(Aij , I¯) = 1 means that Aij is to the same extent similar to I¯ and I¯c . Therefore, the image is transferred to the FD using the following rule m ¯ ¯ 1 − Sim(Am ij , I ) if 0 Sim(Aij , I ) 1, µA˜ (gij ) = (78) ¯ 1 if Sim(Am ij , I ) 1. Finally, the edge map of the image in the GLD is obtained as Edge(gij ) = (L − 1)µA˜ (gij ),

(79)

for all gij , with i ∈ {1, . . . , M } and j ∈ {1, . . . , N }. In order to evaluate the performance of the proposed intuitionistic fuzzy edge detection scheme, we considered the image of Fig. 15(a). The image was contaminated by “salt & pepper” noise with density 0.04, as well as with Gaussian white noise with zero mean and variance 0.01. Fig. 15(b) depicts the outcome of the Sobel edge detector applied to the filtered initial image, using an adaptive Wiener filter of size 3 × 3 pixels. This result is considered as the edge map G1 . Similarly, Fig. 15(c) is the edge map G2 obtained after median filtering in a 3 × 3 pixels window and application of the Sobel operator. The membership and hesitancy components of the A–IFS representing the edges of the image are illustrated in Figs. 15(d) and 15(e) respectively, after applying the -operator with m = 0.3. It should be mentioned that different values of m can be selected from the range (0, 1), in order to increase the hesitancy and thus make the algorithm more adaptive to noise. From Fig. 15(e) one can observe that the proposed method successfully assigns high hesitancy values to pixels where false edges occur due to the presence of noise. At the same time, low hesitancy values are assigned to pixels treated as edges or constantintensity regions by both edge maps. The edge map in the GLD is derived as shown in Fig. 15(f), for min as the t-norm operator and τ = 1. Using parametric t-norm operators, one can adjust the influence of the edge maps


(a)

(b)

(c)

(d)

(e)

(f)

411

Fig. 15. (a) Image contaminated by Gaussian white noise and “salt & pepper” noise. Edge maps obtained using Sobel operator and pre-processing with (b) adaptive Wiener filtering and (c) median filter in a 3 × 3 window. (d) Membership and (e) hesitancy components of the corresponding A–IFS, for τ = 1, after applying the -operator with m = 0.3. (f) Resulting edge map by proposed intuitionistic fuzzy framework

G1 and G2 to the membership function of the A–IFS. From the resulting edge map, one can observe that the proposed hesitancy-driven edge detection scheme successfully extracts edges, while suppressing noise, even if the initial image is highly contaminated by different types of noise.

8 Conclusions In this chapter, we presented a novel intuitionistic fuzzy framework for image processing. Heuristic and analytical methods for analysis and synthesis of images to and from their intuitionistic fuzzy components were also introduced. The proposed IFIP environment successfully models the uncertainty

412


associated with image pixels, originating from various sources, such as the nonlinear behavior of the acquisition and imaging mechanisms or/and the unknown physical origin of the image itself. Moreover, each stage of the proposed framework was thoroughly analyzed and their particular characteristics were investigated in depth. Additionally, a novel operator for A–IFSs was introduced that selectively adjusts the entropy of the A–IFSs. The efficiency of the proposed approach to IFIP was demonstrated using various examples in the field of contrast enhancement and edge detection, yielding promising results. We believe that the proposed intuitionistic fuzzy environment introduces a new potential in carrying out various image processing tasks, since it offers a flexible and adaptable way of treating the hesitancy present in digital images.

References 1. Atanassov K.T. Intuitionistic fuzzy sets. Fuzzy Sets Syst., vol. 20, pp. 87–96, 1986. 2. Atanassov K.T. More on intuitionistic fuzzy sets. Fuzzy Sets Syst., vol. 33, pp. 37–45, 1989. 3. Atanassov K.T. Intuitionistic Fuzzy Sets: Theory and Applications. Studies in Fuzziness and Soft Computing. Physica–Verlag, Heidelberg, 1999. 4. Atanassov K.T. and Gargov G. Interval valued intuitionistic fuzzy sets. Fuzzy Sets Syst., vol. 31, pp. 343–349, 1989. 5. Burillo P. and Bustince H. Construction theorems for intuitionistic fuzzy sets. Fuzzy Sets Syst., vol. 84, pp. 271–281, 1996. 6. Bustince H. and Burillo P. Vague sets are intuitionistic fuzzy sets. Fuzzy Sets Syst., vol. 79, pp. 403–405, 1996. 7. Bustince H., Kacprzyk J., and Mohedano V. Intuitionistic fuzzy generators— application to intuitionistic fuzzy complementation. Fuzzy Sets Syst., vol. 114, pp. 485–504, 2000. 8. Cheng H.D. and Chen J.R. Automatically determine the membership function based on the maximum entropy principle. Inf. Sci., vol. 96, pp. 163–182, 1997. 9. Cornelis C., Atanassov K.T., and Kerre E.E. Intuitionistic fuzzy sets and interval-valued fuzzy sets: A critical comparison. In: Proc. Third European Conf. on Fuzzy Logic and Technol. (EUSFLAT’03), pp. 159–163. Zittau, Germany, 2003. 10. De S.K., Biswas R., and Roy A.R. Some operations on intuitionistic fuzzy sets. Fuzzy Sets Syst., vol. 114, pp. 477–484, 2000. 11. Deschrijver G. and Kerre E.E. On the relationship between some extensions of fuzzy set theory. Fuzzy Sets Syst., vol. 133, pp. 227–235, 2003. 12. Dubois D., Gottwald S., Hajek P., Kacprzyk J., and Prade H. Terminological difficulties in fuzzy set theory—The case of “Intuitionistic Fuzzy Sets”. Fuzzy Sets Syst., vol. 156, pp. 485–491, 2005. 13. Dubois D., Ostasiewicz W., and Prade H. Fuzzy sets: History and basic notions. In: D. Dubois and H. Prade (Eds.), Fundamentals of Fuzzy Sets, The Handbooks of Fuzzy Sets Series, pp. 21–124. Kluwer, Dordrecht, 2000. 14. Frei W. Image enhancement by histogram hyperbolization. Comput. Graphics Image Process., vol. 6, pp. 286–294, 1977.


413

15. Gau W.L. and Buehrer D.J. Vague sets. IEEE Trans. Syst. Man Cybern., vol. 23, pp. 610–614, 1993. 16. Grzegorzewski P. Distances between intuitionistic fuzzy sets and/or intervalvalued fuzzy sets based on the Hausdorff metric. Fuzzy Sets Syst., vol. 148, pp. 319–328, 2004. 17. Jawahar C.V. and Ray A.K. Fuzzy statistics of digital images. IEEE Signal Process. Lett., vol. 3, pp. 225–227, 1996. 18. Jawahar C.V. and Ray A.K. Incorporation of gray-level imprecision in representation and processing of digital images. Pattern Recognit. Lett., vol. 17, pp. 541–546, 1996. 19. Kaufmann A. Introduction to the Theory of Fuzzy Subsets—Fundamental Theoretical Elements, vol. 1. Academic Press, New York, 1975. 20. Luca A.D. and Termini S. A definition of a nonprobabilistic entropy in the setting of fuzzy sets theory. Inf. Control, vol. 20, pp. 301–312, 1972. 21. Mendel J.M. and John R.I.B. Type-2 fuzzy sets made simple. IEEE Trans. Fuzzy Syst., vol. 10, pp. 117–127, 2002. 22. Nikolova M., Nikolov N., Cornelis C., and Deschrijver G. Survey of the research on intuitionistic fuzzy sets. Advanced Studies in Contemporary Mathematics, vol. 4, pp. 127–157, 2002. 23. Nikolova M., Nikolov N., Cornelis C., and Deschrijver G. Survey of the research on intuitionitsic fuzzy sets. In: Proc. of the 6th Joint Conference on Information Science, (JCIS’02), pp. 117–120, 2002. 24. Pal S.K. and King R.A. Image enhancement using fuzzy set. Electron. Lett., vol. 16, pp. 376–378, 1980. 25. Pal S.K. and King R.A. Image enhancement using smoothing with fuzzy sets. IEEE Trans. Syst. Man Cybern., vol. 11, pp. 495–501, 1981. 26. Pal S.K. and King R.A. A note on the quantitative measure of image enhancement through fuzziness. IEEE Trans. Pattern Anal. Mach. Intell., vol. 4, pp. 204–208, 1982. 27. Rick A., Bouchon-Meunier B., Muller S., and Rifqi M. Fuzzy techniques in mammographic image processing. In: E.E. Kerre and M. Nachtegael (Eds.), Fuzzy Techniques in Image Processing, pp. 308–336. Physica–Verlag, 2000. 28. Suckling J., Parker J., Dance D., Astley S., Hutt I., Boggis C., Ricketts I., Stamatakis E., Cerneaz N., Kok S., Taylor P., Betal D., and Savage J. The mammographic image analysis society digital mammogram database. International Congress Series, vol. 1069, pp. 375–378, 1994. 29. Szmidt E. and Kacprzyk J. A fuzzy set corresponding to an intuitionistic fuzzy set. Int. J. Uncertainty Fuzziness Knowl.-Based Syst., vol. 6, pp. 427–435, 1998. 30. Szmidt E. and Kacprzyk J. Distances between intuitionistic fuzzy sets. Fuzzy Sets Syst., vol. 114, pp. 505–518, 2000. 31. Szmidt E. and Kacprzyk J. Entropy for intuitionistic fuzzy sets. Fuzzy Sets Syst., vol. 118, pp. 467–477, 2001. 32. Szmidt E. and Kacprzyk J. A concept of similarity for intuitionistic fuzzy sets and its use in group decision making. In: Proc., 2004 IEEE International Conference on Fuzzy Systems, pp. 1129–1134, 2004. 33. Szmidt E. and Kacprzyk J. A similarity measure for intuitionistic fuzzy sets and its application in supporting medical diagnostic reasoning. In: L. Rutkowski, J.H. Siekmann, R. Tadeusiewicz, and L.A. Zadeh (Eds.), Proc. 7th International Conference ICAISC 2004, Lecture Notes in Computer Science, vol. 3070, pp. 388–393. Springer, 2004.

414


34. Szmidt E. and Kacprzyk J. Similarity of intuitionistic fuzzy sets and the Jaccard coefficient. In: Proc. IPMU’04, pp. 1405–1412, 2004. 35. Takeuti G. and Titani S. Intuitionistic fuzzy logic and intuitionistic fuzzy set theory. J. Symb. Log., vol. 49, pp. 851–866, 1984. 36. Tizhoosh H.R. http://pami.uwaterloo.ca/tizhoosh/fip.htm. 37. Tizhoosh H.R. Fuzzy image enhancement: An overview. In: E.E. Kerre and M. Nachtegael (Eds.), Fuzzy Techniques in Image Processing, pp. 137–171. Springer–Verlag, 2000. 38. Tizhoosh H.R. and Fochem M. Image enhancement with fuzzy histogram hyperbolization. In: Proc. of EUFIT’95, vol. 3, pp. 1695–1698, 1995. 39. Zadeh L.A. Fuzzy sets. Inf. Control, vol. 8, pp. 338–353, 1965. 40. Zadeh L.A. A fuzzy-set-theoretic interpretation of linguistic hedges. J. Cybern., vol. 2, pp. 4–34, 1972.

Chapter 15 Integer Programming Methods in Image Processing and Bayes Estimation Boris A. Zalesky United Institute of Informatics Problems National Academy of Sciences Surganov Street 6, Minsk, 220012, Belarus [email protected]

To my mother, my wife and my sons

1 Introduction For the last few decades significant progress in image processing has been achieved. Nowadays various types of images are widely employed in medical and industrial application, sciences and engineering, surveillance systems etc. The progress can be explained not only by rapid development of computers and their hardware but also by advances in computational techniques. Usually, computations to enhance quality of pictures, to segment them or to recognize image objects require enormous calculations. The real use of the image estimators is restricted by opportunity to calculate or to evaluate satisfactory their values. Special methods and techniques have been developed to provide that. Results that we present in this chapter are based on new integer programming and combinatorial optimization methods that were specially proposed to solve practical problems of image processing. Although, the integer programming and the combinatorial optimization are successfully used to solve a wide variety of other practical tasks, including different scheduling problems, production and electricity generation planning, classic knapsack and traveling salesman problems etc. [13, 17]. Often image processing problems are reduces to the following general integer programming task. For an original digital picture we need to find its estimate, which is segmented, classified or filtered image etc., minimizing some specific criterion. At that, the originals image and its estimates are supposed to have, in general, different pixel values. For instance, for image segmentation problems pixels of resulting images take values in appropriate finite labeling sets, whereas pixels of recognized pictures usually take only two values. B.A. Zalesky: Integer Programming Methods in Image Processing and Bayes Estimation, StudFuzz 210, 415–444 (2007) c Springer-Verlag Berlin Heidelberg 2007 www.springerlink.com

416

B.A. Zalesky

In many cases the Bayes estimation, including the Bayes estimation of images, comes to the same discrete optimization problem. Indeed, all our collections of images are finite sets. Therefore, any Bayes model can be presented in the exponential Gibbs form, but a Gibbs maximum a posteriori estimate is a resulting image that minimizes the energy function of the Gibbs model. On the other hand, the Bayes approach is a convenient tool to estimate statistical reliability of results. The main goal of the paper is to present new integer minimization techniques that are particularly actual for image processing and the Bayes estimation. They are methods and algorithms of integer minimization of modular functions – the MFM method, square submodular polynomials – the SPM method, the method of minimization of functions that can be represented by submodular polynomials (the MSFM method) and the method of minimization of submodular polynomials via representing them by graphs. The first, the second and the fourth methods are based on the graph cut technique and the third one is close to the SFM algorithm published in [9, 11]. We would like to note that the MFM method solves the problem posed by Greig, Porteous and Seheult in [8]. To find MFM and SPM estimates special multiresolution version of the minimum graph cut algorithm, which we call the MGMC-algoritm, has been developed. This parallel algorithm is especially efficient to estimate images. For instance, it was successfully exploited to find the Ising model estimates of binary pictures. Besides, the MGMC-algoritm is used as part the MFM and the SFM methods. All methods do not require a local Markov dependence of coordinates of variables but they are especially efficient when such type dependence presents. In particular, they are fast for image processing since pictures are usually characterized by a local dependence of pixels. The algorithms are polynomial on input data and can be implemented in a concurrent mode since they use parts of the input image to find corresponding parts of its estimate. Each algorithm is adapted to minimize functions its specific type. Though, the third method minimization of submodular functions, which we call the MSFM method, is applicable to minimize modular polynomials and square submodular polynomials MFM and SPM methods minimize modular functions and submodular square polynomials a lot more efficiently. So, in the worst case the MFM and SPM methods need O(n3 ) operations to minimize an appropriate functions whereas MSFM-method requires, in general, O(n8 ). Actually, numerical tests showed practically linear dependence of computational time of the MFM and the SPM methods on input data to segment large size 2D and 3D images. Results of image processing are urgent for medicine applications, including 2D and 3D ultrasonic imaging and tomography, for stereo vision, for pattern recognition and other problems. Specialists of many branches prepared and provided progress in the area. References to integer minimization methods and image processing by discrete optimization technique can be found in [1, 2, 4, 5, 6, 8, 9, 10, 11, 12, 15, 16, 18, 19].

Integer Programming Methods in Image Processing

417

In spite of existence of many different fruitful approaches to the problem the methods of integer programming presented in this paper are very recent. The chapter consists of description of their main ideas, formulations and examples of some applications.

2 Image processing as discrete optimization problem In the Section we formulate a general framework of problems of image processing in a form of a integer minimization task, introduce common notations and discuss some specific problems of image analysis. For the finite set of pixels S = {1, . . . , N } we understand an image I as a vector with coordinates Ij , j ∈ S that take values in some ordered finite set F = {f1 , . . . , fL }, (f1 ≤ . . . ≤ fL ). Our approach is independent of an image dimension. Instead, we take into account an image neighborhood system that are specified by the graph G = (S, A) with nodes-pixels S and edges A = {(i, j), i, j ∈ S} connecting neighbor pixels. By other words, an image is considered as a graph G with nodes labeled by a vector I. Examples of systems of neighborhoods of 2D and 3D images commonly used in image processing are shown in Fig.1. However, since for different goals of image processing various image values and neighborhood systems are preferable we do not impose restrictions on the finite ordered set F and the graph G. For instance, for binary images F = {0, 1}, for grayscale images – F = {0, . . . , 255} but for image recognition problems F can be another set of image characteristics. We assume that a resulting image DI is with the same graph G but, in general, it takes values in another finite ordered set M = {m1 , . . . , mK }, (m1 ≤ . . . ≤ mK ). So, to lighten segmentation of a grayscale image with F = {0, . . . , 255} one can use some sparse set, for instance, M = {0, 10, 20, . . . , 240, 250}.

Fig. 1. Neighborhoods of 2D and 3D images commonly used in image processing

Various problems in image processing and the Bayes estimation are reduced to the following task: for an image I, which takes a value f ∈ F, with the graph G to determine its estimate D = argminm∈MN U (f , m) m

(1)

for some function U (f , m) that in the following will be denoted Uf (m). In spite D of clear statement the problem is known as hard because computation of m

418

B.A. Zalesky

usually requires enormous number of operations. Moreover, for some functions U it is NP-hard, i.e. it needs, in principle, exponential number of calculations. Till recently it was only possible to compute estimates of a resulting image D (see, [6, 8]). m We developed the polynomial time algorithm to find the minimum of Uf (m) for modular functions of the form λi |fi − mi | + βi,j |mi − mj |, λi ≥ 0, βi,j ≥ 0. (2) Umod,f (m) = i∈V

(i,j)∈E

It is particularly efficient for image processing problems and can be implemented in a concurrent mode. The algorithm solves the problem, which was posed by Greig, Porteous and Seheult in [8]. The problem turnes out reducible to several separate problems of determining minimum graph cuts. At that, a solution of each previous graph cut problem determines part of the minimum cut of consequent. The special graph cut technique, which allows finding parts D = argminm∈MN Umod,f (m) by minimum cuts of coordinates of a solution m of appropriate subgraphs of the graph G, has been created, as well. The decomposition graph cut technique has been developed to minimize submodular second order polynomials. The condition of submodularity is necessary here because in general the problem of minimization of second order polynomials is NP-hard. Submodular square polynomials can be written into the form λi (fi − mi )2 + βi,j (mi − mj )2 , λi ≥ 0, βi,j ≥ 0. Usec,f (m) = i∈V

(i,j)∈E

The problem of their minimization is reduced to the determining minimum graph cut, which can be solved by means of the developed graph cut technique. Note that for bitmap images with F = M = {0, 1} functions Umod = Usec . In order to solve task (1) for functions U of a general form, the technique has been offered to present U as a Boolean polynomial, so that vector minD Also, the polynoimizing the Boolean polynomial determines a solution m. mial time MSFM algorithm was developed to minimize submodular Boolean polynomials. Like MFM and SPM methods it is based on decomposition of the initial submodular Boolean polynomial by sub-polynomials of a few Boolean variables. Separate minimization of those submodular sub-polynomials enables to find parts of coordinates of a Boolean vector that minimizes the D of the resulting image. initial Boolean polynomial and, then, to find a value m The algorithms presented in the chapter demonstrated possibility to solve real image processing problems. For instance, to segment 3D ultrasonic images of thyroid gland of size 200 × 200 × 300 with 26-pixels neighborhood systems by the non-parallel version of the MFM algorithm the personal computer with Pentium 4 1.6GH CPU spent 14 min. Fast computations of image estimates can provide creation of new models giving chances for their determination or evaluation.


419

3 Multiresolution network flow minimum cut algorithm We now describe the multiresolution graph minimum cut algorithm (shortly, MGMC algorithm). For some types of graphs this highly parallel method turns out more speedy and more efficient in comparison with known maximum network flow algorithms. It was successfully used for identification of minimum cuts of large graphs, for instance, for determination of the Gibbs estimates of binary images, while classical methods were not able to solve the problem. The detailed description of the algorithm can be found in [18]. 3.1 Preliminaries Assume that G is an arbitrary network1 not necessary induced by an image or a Bayes model. The essence of the MGMC algorithm is the following. The network G is partitioned into several subnetworks of appropriate sizes. Every subnetwork is modified in a special way and evaluated two times. The first time all boundary arcs going from the outside into the subnetwork are considered as going from the source (to the same nodes) and all boundary arcs going from within the subnetwork outside are ruled out of the consideration. The minimum cut of the modified subnetwork is determined by known algorithms [7, 2]. It is proven that all nodes of the network, which are connected with the sink by a directed path, will be connected with the sink in the solution of the original network by the same path. The second time all boundary arcs going from the outside into the subnetwork are excluded from the consideration and all boundary arcs going from within the subnetwork outside are supposed to be connected with the sink. The minimum cut of the modified subnetwork is identified once more. This time all directed paths going from the source are determined. The paths identified are excluded from the further consideration. The reduced network G1 is divided into several subnetworks again. The procedure of identification of arcs belonging to G1 but not belonging to the minimal cut is repeated now with one difference – we take in account arcs connected with excluded nodes. The arcs not belonging to the minimal cut of G1 are found and are removed anew. We go to higher levels until we obtain the network Gk that can be solved by a usual maximum network flow algorithm. A) consist of n + 2 numbered In more details. Let the network G = (S, ˜ nodes S = {0, 1, 2 . . . , n, n + 1}, where s = 0 is the source, t = n + 1 is the sink and S = {1, 2 . . . , n} are usual nodes. The set of directed arcs is ˜ Capacities of arcs are denoted by di,j > 0. Suppose A = {(i, j) : i, j ∈ S}. the network G satisfies the condition: (G) 1

For every usual node i ∈ S there is either an arc (s, i) ∈ A connecting the source s with i or an arc (i, t) ∈ A connecting i with the sink t but not both of these arcs.

Remind that a network is a directed graph with to special nodes called by the source and the sink

420

B.A. Zalesky

Remark 1. The condition (G) does not restrict the use of the MGMC algorithm. It has been done to simplify writing down notations and proofs. Any network G can be easily modified to satisfy (G) for O(n) operations (one look through usual network nodes S). The modified network will have the same minimum cut. A network, which corresponds to a binary image, has all white pixels connected with the source and all black ones attached to the sink (see left picture on Fig. 2).

Fig. 2. Left network satisfies condition (G); Central network corresponds to function Umod,f ; Right network presents function Usec,f

To connect properties of network cuts with properties of Boolean polynoE mials consider two sets W, B that partition S ( W, B ⊂ S, W B = ∅ and F W B = S). The Boolean vector x : S → {0, 1}|S| with coordinates xi = 1, if i ∈ W , and xi = 0 otherwise is the indicator vector of the set W . The set of all such indicator vectors x is denoted by X. The capacity of the s − t cut C(x) is defined as the sum of capacities of all forward arcs going from the set W ∪ {s} to the set B ∪ {t} [13], i.e. di,j . C(x) = i∈W ∪{s} j∈B∪{t}

Let y be the vector with coordinates yi = 1, if (s, i) ∈ A, or yi = 0, if (i, t) ∈ A (remind that the condition (G) is valid). Set λi = ds,i , if (s, i) ∈ A, or λi = di,t , if (i, t) ∈ A. The capacities of usual arcs (i, j), i, j ∈ S will be denoted by βi,j = di,j . It is not hard to see that for the function U (x) = λi (1 − 2yi )xi + βi,j (xi − xj )xi (3) i∈S

(i,j)∈S×S

n λi yi [14]. Functions C(x) and U (x) are the s − t-cut C(x) = U (x) + i=1 n distinguished only by the constant i=1 λi yi . Therefore, the solutions x∗ = argminx∈X U (x) entirely identify minimum network flow cuts.


421

Example 1. As we mentioned above there is a clear correspondence between networks satisfying condition (G) and binary images. So, the vector y can be understood as an original binary image and the vector x∗ as its estimate that minimizes the quadratic function U (x). For an arbitrary subset of usual nodes E ⊂ S define two functions UE (x) = λi (1 − 2yi )xi + βi,j (xi − xj )xi (4) (i,j)∈(E×E)∪(E×E c )∪(E c ×E)

i∈E

and VE (x) =

λi (1 − 2yi )xi +

βi,j (xi − xj )xi ,

(i,j)∈(E c ×E c )

i∈E c

satisfying the equality U (x) = UE (x) + VE (x). Also, define restriction xE of x onto the set E, which is the vector xE = (xi )i∈E , such that x = (xE , xE c ). Since the function VE (x) depends only on xE (i.e. VE (x) = VE (xE )) the following simple proposition is valid. o

Proposition 1 If xE minimizes the function φ(xE ) = U (xE , xE c ) for some o fixed xE c , then for any set D ⊂ E the restriction xD of xE minimizes the o function ψ(xD ) = UD (xD , xE\D , xE c ) and vice versa. For vectors xE , zE the ordering relationship xE ≤ zE means xi ≤ zi , i ∈ E; xE < zE , if xi ≤ zi , i ∈ E and there is at least one node j ∈ E such that xj < zj ; xE ≤ zE , if there are nodes i, j ∈ E such that xi < zi and xj > zj . Our method is based on monotone (in some sense) dependence of restrictions x∗E of solutions x∗ = argminx∈X U (x) on values x∗E c . 3.2 Description and ground of algorithm In spite of the MGMC algorithm generality it has been developed first to process images. Some useful characteristics of graph cuts were exhibited by geometrical properties of binary images minimizing the function U (x) of (3). o Let us consider the property of the monotony of x∗E in details. For fixed xE c o o o and zE c set xE = argminxE U (xE , xE c ) and zE = argminxE U (xE , zE c ). It is o o clear that every set {xE }x c , {zE }z c can consist of more than one element. o

E

o

E

In general, in the case xE c ≤zE c the inequality xE ≤ zE is not satisfied, but without fail there exist solutions xE and zE such that xE ≤ zE . Theorem 2 Following properties are valid. o

o

(i) If fixed frontier vectors satisfy the condition xE c ≤zE c , then for any soluo tion xE = argminxE U (xE , xE c ) there exists a solution o zE = argminxE U (xE , zE c ) such that xE ≤ zE .

422

B.A. Zalesky o

o

(ii) If fixed frontier functions satisfy the condition xE c ≤zE c , then for any o solution zE = argminxE U (xE , zE c ) there exists a solution o xE = argminxE U (xE , xE c ) such that xE ≤ zE . o

(iii) For any frontier condition xE c the set {xE }xo c has the minimal (the E maximal) element xE (xE ). o o (iv) If xE c ≤zE c , then xE ≤ zE and xE ≤ zE . The detailed proof of Theorem 2 can be found in [18]. It is based on properties of the function of 3. Proposition 1 and Theorem 2 allow to validate theoretically the MGMC algorithm. The main idea of the MNFC is to estimate at least parts of restrictions E x∗Ei ofEsolutions x∗ = argminx U (x) for a suitable fixed partition Ei = S, Ei Ej = ∅. For this purpose the monotone dependence in the sense of Theorem 2 of local solutions x∗Ei on the frontier values of x∗E c is exploited.

i

The parts of x∗Ei estimated by the special local solutions xEi are ruled out of the further consideration. It significantly reduces computational expenses. To simplify the explanation let us consider an arbitrary subset E ∈ S. It follows from Proposition 1 that the vector xE = argminxE U (xE , x∗E c ) = x∗E for some solution x∗ , and the solution xE minimizes the function UE (xE , x∗E c ) for the frontier condition x∗E c . Theorem 2 guarantees existence of two solutions that can be found by known maximum network flow algorithms [2, 7] x0,E = argminxE U (xE , 0E c ) and x1,E = argminxE U (xE , 1E c ), which satisfy the inequality x0,E ≤ xE ≤ x1,E .

(5)

Denote the sets on nodes B = {k ∈ E | x1,E,k = 0} and W = {k ∈ E | x0,E,k = 1}. The equalities 0B = x1,B = x∗B and 1W = x0,W = x∗W is inferred from (5). If sets B and W are not empty we found coordinates x∗B∩W of some solution x∗ of the minimum cut. Actually, we needFto repeat this F procedure for F each subset Ei of our partition. For sets B = i Bi , W = i Wi , R = B W it gives coordinates x∗R of x∗ . The formal description of the MGMC algorithm can be done in the following form. Step 1 (Initialization). Assign: the level number l = 1, the set not estimated nodes S(l) = S, the set of estimated nodes R = ∅, and the set of arcs A(l) = A. Step 2 (Partition of the network). Consider the F network F {S(l), {s}, {t}, A(l)} with nodes S(l) = S\R and arcs A(l) = D(l) F (l) H(l) of three types D(l) = {(µ, ν) : (µ, ν) ∈ A, µ ∈ S(l), ν ∈ S(l)} , F (l) = {(µ, t) : µ ∈ S(l)} , H(l) = {(s, ν) : ν ∈ S(l)}


423

having capacities cap(µ, ν) = dµ,ν ,

(µ, ν) ∈ D(l),

cap(µ, t) = dµ,t +

dµ,ν ,

(µ, t) ∈ F (l),

ν∈B(l), µ∼ν

cap(s, ν) = ds,ν +

dµ,ν ,

(s, ν) ∈ H(l)

µ∈W(l), µ∼ν

where dµ,ν are capacities of the of the original network G and the relation µ ∼ ν neans that µ and ν are neighbors in G. If the maximum flow in the network {S(l), {s}, {t}, A(l)} can be efficiently identified by a usual network minimum cut algorithm, we do not need its partition. Assign the number of partition elements kl = 1, and E1 (l) = S(l). Identify the minimum cut in {S(l), {s}, {t}, A(l)}, STOP. Otherwise, partition the set S(l) by not intersected sets E1 (l), E2 (l), . . . , Ekl (l), Ei (l)

=

Ej (l) = ∅,

kl
0 denote by     D mod = argminm∈MV λi m |fi − mi | + βi,j |mi − mj | (6)   i∈S

(i,j)∈A

and by D sec = argminm∈MV m

  

i∈S

λi (fi − mi )2 +

(i,j)∈A

βi,j (mi − mj )2

  

.

(7)

Since functions Umod,f (m) and Usec,f (m) depend on differences of fi − mi without loss of generality we suppose that all fi , mi > 0.

426

B.A. Zalesky

D mod 4.1 Method of integer minimization to find m The idea of the method consists in representation of the vector of image values m as integer-valued combination of Boolean vectors and, then, reduction of the problem of integer minimization to the problem of Boolean optimization. The last problem can be solved efficiently by the MGMC or other graph cut algorithms. Am example of representation of the functional Umod,f (m) by a graph is drawn at the center of Fig. 2. Nodes of the graph correspond to pixels of binary images that decompose some 4-level gray-scale image. = 1, if µ ≥ ν, Let µ, ν be integers and the indicator function 1 µ≥ν

and be zero otherwise. For any µ ∈ M and ordered Boolean variables x(l) = that satisfy the inequality 1 µ≥m(l)

x(1) ≥ x(2) ≥ . . . ≥ x(k) the relationship µ = m(0) +

K

(m(l) − m(l − 1))x(l),

(8)

l=1

holds true. Vice versa, any non-increasing sequence of Boolean variables x(1) ≥ x(2) ≥ . . . ≥ x(k) determines integer µ ∈ M according to (8). Similarly, image values fi ∈ F can be represented as sums fi =

L

fi (τ )

τ =1

of non-increasing sequences of Boolean variables fi (1) ≥ fi (2) ≥ . . . ≥ fi (L). Let x(l) = (x1 (l), x2 (l), . . . , xN (l)), l = 1, . . . , K be Boolean vectors. Vectors z(l) are with coordinates 1 zi (l) = m(l) − m(l − 1)

m(l)

fi (τ ).

(i = 1, . . . , N, l = 1, . . . K)

τ =m(l−1)+1

and |x| =

N

|xi |

i=1

is the norm of the vector. The following Proposition is valid. Proposition 3 For all integer ν ∈ M and all fi ∈ F


427

m(0) |ν − fi | = m(0) − fi (τ ) + τ =1

K l=1

fi (τ ) +

m(l)

− (m(l) − m(l − 1))1 ν≥m(l)

τ =m(l−1)+1

L

fi (τ ) (9)

τ =m(k)+1

Thus, for any image value f ∈ F N and any value m ∈ MN of the estimate the function Umod,f (m) can be represented in the following form Umod,f (m) =

K

(m(l) − m(l − 1))u(l, x(l)) + const,

(10)

l=1

m(0) where const = N m(0) − τ =1 fi (τ ), N -dimensional Boolean vectors x(l) = (1

m1 ≥m(l)

, 1

m2 ≥m(l)

, . . . , 1

mn ≥m(l)

)

and functions u(l, b) =

λi |zi (l) − bi | +

i∈S

βi,j |bi − bj | .

(i,j)∈A

for N -dimensional Boolean vectors b. Denote by ˇ (l) = argminb u(l, b), x

(l = 1, . . . , K)

(11)

the Boolean solutions that minimize u(l, b) and note that z(1) ≥ z(2) ≥ . . . ≥ z(K). Moreover, zi (1) = zi (2) = . . . = zi (ν − 1) = 1, 0 ≤ zi (ν) ≤ 1, zi (ν + 1) = . . . = zi (K) = 0 ˇ (l) are in general for some integer 1 ≤ ν ≤ K. It is easy to see that solutions x ˇ l1 ≥ x ˇ l2 can exist. unordered in a sense that for ll < l2 unordered solutions x Nevertheless, there always exists non-increasing sequence ˇ (1) ≥ x ˇ (2) ≥ . . . ≥ x ˇ (K) x of solutions of problem (11). The following properties, which have been proven in [19], hold true. Theorem 4 If 1 ≤ l < l ≤ K are integers and a sequence of vectors z(1) ≥ z(2) ≥ . . . ≥ z(k) are ordered, then

428

B.A. Zalesky

ˇ (l ) is an arbitrary solution of problem (11), there exists a solution (i) if x ˇ (l ) such that x ˇ (l ) ≥ x ˇ (l ). Conversely, if x ˇ (l ) is an arbitrary solution x ˇ (l ) ≥ x ˇ (l ); ˇ (l ) such that x of problem (11), then there exists a solution x (ii) given 1 ≤ l ≤ k, the set of solutions {ˇ x(l)} has a minimal x(l) and maximal x(l) elements; (iii) the sets of minimal and maximal elements are ordered, that is x(1) ≥ . . . ≥ x(k), x(1) ≥ . . . ≥ x(k). Equality (10) of Proposition 3 and Theorem 4 allow to conclude that one of solutions of original problem (6) is of the form D mod = m(0) + m

K

(m(l) − m(l − 1))ˇ x(l)

(12)

l=1

for n-dimensional vector m(0) = (m(0), . . . , m(0)) and non-increasing seˇ (1) ≥ x ˇ (2) ≥ . . . ≥ x ˇ (K) of vectors. quence x ˇ (l) can be found by minimum The solutions x 0 / cut algorithms including the MGMC algorithm and then ordered by or operations. However, there is more efficient strategy. To determine a solution of problem (6) one can execute the following sequence of computations: ˇ (1) by the MGMC or another minimum graph cut algoStep 1. To find x rithm. ˇ (l + 1), l = 1, . . . , k − 1 set x Step 2. Before computation of x ˇi (l + 1) = 0 for all i ∈ B(l) = {i ∈ S : x ˇi (l) = 0} and consider the reduced problem ˇ Bc (l) (l + 1) = argminxBc (l) u(l, 0B(l) , xBc (l) ) x just as it was done in the MGMC algorithm. D mod of the original problem by the formula Step 3. Calculate the solution m (12). In a case of gray-scale or color images this strategy diminishes essentially number of operation in comparison with immediate minimization of u(l, b). As we have already mentioned in the Introduction segmentation of grayscale 3D ultrasonic images of thyroid gland of size 200×200×300 with 26-pixels neighborhood systems by the non-parallel version of the MFM algorithm along with offered strategy takes 14 min of computations of the personal computer with Pentium 4 1.6GH CPU. Results of processing a noisy image of a car by the median filter and by the function Umod,f (m) are drawn in Fig. 4. The bottom left image of Fig. 3 was D mod estimate done by the median filter whereas the bottom right picture is m of the noisy copy. D sec 4.2 Method of integer minimization to find m D sec of (7) we again represent integer–valued vector variables m To find m by a integer combination of ordered Boolean vectors. Then, the problem of


429

D mod -estimate images Fig. 4. Original, noisy, filtered by median and m

integer minimization of Usec,f (m) will be reduced to the problem of Boolean optimization. However, now one connected graph with K × N + 2 nodes will be used instead of K separate graphs with N +2 nodes each. The right picture on Fig. 2 is an example of a graph that represents the function Usec,f (m). Denote by gi = fi − m(0), i ∈ S,

al = m(l) − m(l − 1), l = 1, . . . , K.

According to (8) the vector m can be represented as the integer linear comK bination m(0) + l=1 al x(l) of ordered Boolean vectors x(l) = (x1 (l), x2 (l), . . . , xN (l)) l = 1, . . . , K, and the function Usec,f (m) is written as follows Usec,f (m) =

2 K λ i gi − al xi (l) +

i∈S

l=1

(i,j)∈A

βi,j

K

al xi (l) − xj (l)

i∈S

where the polynomial of the Boolean variables P (x(1), . . . , x(K)) =

K 2 = al − 2gi al xi (l) λi −

l=1

(i,j)∈A

βi,j

1≤τ 0 satisfy the condition ai,j +

n−2

b+ l1 ,l2 ,... ,i,... ,j,... ,lm ≤ 0, ∀i, j ∈ S.

m=1 l1

Soft Computing in Image Processing: Recent Advances (Studies in Fuzziness and Soft Computing)

Soft Computing: Methodologies and Applications (Advances in Soft Computing) (Advances in Intelligent and Soft Computing)

Soft Computing Applications in Business (Studies in Fuzziness and Soft Computing)

Fuzzy Probability and Statistics (Studies in Fuzziness and Soft Computing)

Speech Processing and Soft Computing

Fuzzy Probability and Statistics (Studies in Fuzziness and Soft Computing)

Wavelets in soft computing

Simulating Continuous Fuzzy Systems (Studies in Fuzziness and Soft Computing)

Simulating Continuous Fuzzy Systems (Studies in Fuzziness and Soft Computing)

Recent Advances in Memetic Algorithms (Studies in Fuzziness and Soft Computing)

Fuzzy Mathematics: Approximation Theory (Studies in Fuzziness and Soft Computing)

Fuzzy Implications (Studies in Fuzziness and Soft Computing)

Consensual Processes (Studies in Fuzziness and Soft Computing, 267)

Hybrid Intelligent Systems (Studies in Fuzziness and Soft Computing)

Uncertainty Theory (Studies in Fuzziness and Soft Computing) 3rd Edition

Fuzzy Optimization: Recent Advances and Applications (Studies in Fuzziness and Soft Computing 254)

Soft Methods for Integrated Uncertainty Modelling (Advances in Soft Computing)

Modeling Uncertainty with Fuzzy Logic: With Recent Theory and Applications (Studies in Fuzziness and Soft Computing)

Quantitative Logic and Soft Computing: Vol 2 (Advances in Intelligent and Soft Computing)

Knowledge processing with interval and soft computing

Applied Soft Computing Technologies: The Challenge of Complexity (Advances in Intelligent and Soft Computing)

Production Engineering and Management under Fuzziness (Studies in Fuzziness and Soft Computing, Volume 252)

Soft Computing in Ontologies and Semantic Web

Soft Computing in Economics and Finance

Soft Computing in Humanities and Social Sciences

Applied Soft Computing Technologies: The Challenge of Complexity (Advances in Soft Computing)

Artificial Intelligence and Soft Computing

Artificial intelligence and soft computing

Intelligent Multimedia Processing with Soft Computing

Fuzzy and Rough Techniques in Medical Diagnosis and Medication (Studies in Fuzziness and Soft Computing)

On Collective Intelligence (Advances in Intelligent and Soft Computing, 76)

Soft Computing in Image Processing: Recent Advances (Studies in Fuzziness and Soft Computing)

Soft Computing: Methodologies and Applications (Advances in Soft Computing) (Advances in Intelligent and Soft Computing)

Soft Computing Applications in Business (Studies in Fuzziness and Soft Computing)

Fuzzy Probability and Statistics (Studies in Fuzziness and Soft Computing)

Speech Processing and Soft Computing

Fuzzy Probability and Statistics (Studies in Fuzziness and Soft Computing)

Wavelets in soft computing

Simulating Continuous Fuzzy Systems (Studies in Fuzziness and Soft Computing)

Simulating Continuous Fuzzy Systems (Studies in Fuzziness and Soft Computing)

Recent Advances in Memetic Algorithms (Studies in Fuzziness and Soft Computing)

Fuzzy Mathematics: Approximation Theory (Studies in Fuzziness and Soft Computing)

Fuzzy Implications (Studies in Fuzziness and Soft Computing)

Consensual Processes (Studies in Fuzziness and Soft Computing, 267)

Hybrid Intelligent Systems (Studies in Fuzziness and Soft Computing)

Uncertainty Theory (Studies in Fuzziness and Soft Computing) 3rd Edition

Fuzzy Optimization: Recent Advances and Applications (Studies in Fuzziness and Soft Computing 254)

Soft Methods for Integrated Uncertainty Modelling (Advances in Soft Computing)

Modeling Uncertainty with Fuzzy Logic: With Recent Theory and Applications (Studies in Fuzziness and Soft Computing)

Quantitative Logic and Soft Computing: Vol 2 (Advances in Intelligent and Soft Computing)

Knowledge processing with interval and soft computing

Applied Soft Computing Technologies: The Challenge of Complexity (Advances in Intelligent and Soft Computing)

Production Engineering and Management under Fuzziness (Studies in Fuzziness and Soft Computing, Volume 252)

Soft Computing in Ontologies and Semantic Web

Soft Computing in Economics and Finance

Soft Computing in Humanities and Social Sciences

Applied Soft Computing Technologies: The Challenge of Complexity (Advances in Soft Computing)

Artificial Intelligence and Soft Computing

Artificial intelligence and soft computing

Intelligent Multimedia Processing with Soft Computing

Fuzzy and Rough Techniques in Medical Diagnosis and Medication (Studies in Fuzziness and Soft Computing)

On Collective Intelligence (Advances in Intelligent and Soft Computing, 76)

Recommend Documents