Foundations of Computational Intelligence Volume 2: Approximate Reasoning

Aboul-Ella Hassanien, Ajith Abraham, and Francisco Herrera (Eds.) Foundations of Computational Intelligence Volume 2 ...

Author: Aboul-Ella Hassanien | Ajith Abraham | Francisco Herrera

55 downloads 766 Views 10MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Aboul-Ella Hassanien, Ajith Abraham, and Francisco Herrera (Eds.) Foundations of Computational Intelligence Volume 2

Studies in Computational Intelligence, Volume 202 Editor-in-Chief Prof. Janusz Kacprzyk Systems Research Institute Polish Academy of Sciences ul. Newelska 6 01-447 Warsaw Poland E-mail: [email protected] Further volumes of this series can be found on our homepage: springer.com Vol. 181. Georgios Miaoulis and Dimitri Plemenos (Eds.) Intelligent Scene Modelling Information Systems, 2009 ISBN 978-3-540-92901-7 Vol. 182. Andrzej Bargiela and Witold Pedrycz (Eds.) Human-Centric Information Processing Through Granular Modelling, 2009 ISBN 978-3-540-92915-4 Vol. 183. Marco A.C. Pacheco and Marley M.B.R. Vellasco (Eds.) Intelligent Systems in Oil Field Development under Uncertainty, 2009 ISBN 978-3-540-92999-4 Vol. 184. Ljupco Kocarev, Zbigniew Galias and Shiguo Lian (Eds.) Intelligent Computing Based on Chaos, 2009 ISBN 978-3-540-95971-7 Vol. 185. Anthony Brabazon and Michael O’Neill (Eds.) Natural Computing in Computational Finance, 2009 ISBN 978-3-540-95973-1 Vol. 186. Chi-Keong Goh and Kay Chen Tan Evolutionary Multi-objective Optimization in Uncertain Environments, 2009 ISBN 978-3-540-95975-5 Vol. 187. Mitsuo Gen, David Green, Osamu Katai, Bob McKay, Akira Namatame, Ruhul A. Sarker and Byoung-Tak Zhang (Eds.) Intelligent and Evolutionary Systems, 2009 ISBN 978-3-540-95977-9 Vol. 188. Agustín Gutiérrez and Santiago Marco (Eds.) Biologically Inspired Signal Processing for Chemical Sensing, 2009 ISBN 978-3-642-00175-8 Vol. 189. Sally McClean, Peter Millard, Elia El-Darzi and Chris Nugent (Eds.) Intelligent Patient Management, 2009 ISBN 978-3-642-00178-9 Vol. 190. K.R. Venugopal, K.G. Srinivasa and L.M. Patnaik Soft Computing for Data Mining Applications, 2009 ISBN 978-3-642-00192-5 Vol. 191. Zong Woo Geem (Ed.) Music-Inspired Harmony Search Algorithm, 2009 ISBN 978-3-642-00184-0

Vol. 192. Agus Budiyono, Bambang Riyanto and Endra Joelianto (Eds.) Intelligent Unmanned Systems: Theory and Applications, 2009 ISBN 978-3-642-00263-2 Vol. 193. Raymond Chiong (Ed.) Nature-Inspired Algorithms for Optimisation, 2009 ISBN 978-3-642-00266-3 Vol. 194. Ian Dempsey, Michael O’Neill and Anthony Brabazon (Eds.) Foundations in Grammatical Evolution for Dynamic Environments, 2009 ISBN 978-3-642-00313-4 Vol. 195. Vivek Bannore and Leszek Swierkowski Iterative-Interpolation Super-Resolution Image Reconstruction: A Computationally Efficient Technique, 2009 ISBN 978-3-642-00384-4 Vol. 196. Valentina Emilia Balas, János Fodor and Annamária R. Várkonyi-Kóczy (Eds.) Soft Computing Based Modeling in Intelligent Systems, 2009 ISBN 978-3-642-00447-6 Vol. 197. Mauro Birattari Tuning Metaheuristics, 2009 ISBN 978-3-642-00482-7 Vol. 198. Efrén Mezura-Montes (Ed.) Constraint-Handling in Evolutionary Optimization, 2009 ISBN 978-3-642-00618-0 Vol. 199. Kazumi Nakamatsu, Gloria Phillips-Wren, Lakhmi C. Jain, and Robert J. Howlett (Eds.) New Advances in Intelligent Decision Technologies, 2009 ISBN 978-3-642-00908-2 Vol. 200. Dimitri Plemenos and Georgios Miaoulis Visual Complexity and Intelligent Computer Graphics Techniques Enhancements, 2009 ISBN 978-3-642-01258-7 Vol. 201. Aboul-Ella Hassanien, Ajith Abraham, Athanasios V. Vasilakos, and Witold Pedrycz (Eds.) Foundations of Computational Intelligence Volume 1, 2009 ISBN 978-3-642-01081-1 Vol. 202. Aboul-Ella Hassanien, Ajith Abraham, and Francisco Herrera (Eds.) Foundations of Computational Intelligence Volume 2, 2009 ISBN 978-3-642-01532-8

Aboul-Ella Hassanien, Ajith Abraham, and Francisco Herrera (Eds.)

Foundations of Computational Intelligence Volume 2 Approximate Reasoning

123

Prof. Aboul-Ella Hassanien

Prof. Francisco Herrera

Cairo University Faculty of Computers and Information Information Technology Department 5 Ahmed Zewal St. Orman, Giza E-mail: [email protected] http://www.fci.cu.edu.eg/abo/

Soft Computing and Intelligent Information Systems Department of Computer Science and Artificial Intelligence ETS de Ingenierias Informática y de Telecomunicación University of Granada E-18071 Granada Spain E-mail: [email protected]

Prof. Ajith Abraham Machine Intelligence Research Labs (MIR Labs) Scientific Network for Innovation and Research Excellence P.O. Box 2259 Auburn, Washington 98071-2259 USA E-mail: [email protected]

ISBN 978-3-642-01532-8

e-ISBN 978-3-642-01533-5

DOI 10.1007/978-3-642-01533-5 Studies in Computational Intelligence

ISSN 1860949X

Library of Congress Control Number: Applied for c 2009 Springer-Verlag Berlin Heidelberg This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typeset & Cover Design: Scientific Publishing Services Pvt. Ltd., Chennai, India. Printed in acid-free paper 987654321 springer.com

Preface

Foundations of Computational Intelligence Volume 2: Approximation Reasoning: Theoretical Foundations and Applications

Human reasoning usually is very approximate and involves various types of uncertainties. Approximate reasoning is the computational modelling of any part of the process used by humans to reason about natural phenomena or to solve real world problems. The scope of this book includes fuzzy sets, Dempster-Shafer theory, multi-valued logic, probability, random sets, and rough set, near set and hybrid intelligent systems. Besides research articles and expository papers on theory and algorithms of approximation reasoning, papers on numerical experiments and real world applications were also encouraged. This Volume comprises of 12 chapters including an overview chapter providing an up-to-date and state-of-the research on the applications of Computational Intelligence techniques for approximation reasoning. The Volume is divided into 2 parts: Part-I: Approximate Reasoning – Theoretical Foundations Part-II: Approximate Reasoning – Success Stories and Real World Applications Part I on Approximate Reasoning – Theoretical Foundations contains four chapters that describe several approaches of fuzzy and Para consistent annotated logic approximation reasoning. In Chapter 1, “Fuzzy Sets, Near Sets, and Rough Sets for Your Computational Intelligence Toolbox” by Peters considers how a user might utilize fuzzy sets, near sets, and rough sets, taken separately or taken together in hybridizations as part of a computational intelligence toolbox. In multi-criteria decision making, it is necessary to aggregate (combine) utility values corresponding to several criteria (parameters). The simplest way to combine these values is to use linear aggregation. In many practical situations, however, linear aggregation does not fully adequately describe the actual decision making process, so non-linear aggregation is needed. From the purely mathematical viewpoint, the next natural step after linear functions is the use of quadratic

VI

Preface

functions. However, in decision making, a different type of non-linearities is usually more adequate than quadratic ones: fuzzy-type non-linearities like OWA or Choquet integral that use min and max in addition to linear combinations. In Chapter 2, “Fuzzy Without Fuzzy: Why Fuzzy-Related Aggregation Techniques Are Often Better Even in Situations Without True Fuzziness” by Nguyen et al. gives a mathematical explanation for this empirical phenomenon. Specifically, the authors show that approximation by using fuzzy methodology is indeed the best (in some reasonable sense). In Chapter 3, “Intermediate Degrees are needed for the World to be Cognizable: Towards a New Justification for Fuzzy Logic Ideas” Nguyen et al. prove that intermediate degrees are needed to describe real-world processes and it provides an additional explanation for the success of fuzzy techniques (and other techniques which use intermediate degrees) – which often goes beyond situations in which the intermediate degrees are needed to describe the experts’ uncertainty. Chapter 4, “Paraconsisitent annotated logic program Before After EVALSPN and its applications” by Nakamatsu, proposes a paraconsistent annotated logic program called EVALPSN. In EVALPSN, an annotation called an extended vector annotation is attached to each literal. In addition, the author introduces the bfEVALPSN and its application to real-time process order control and its safety verification with simple examples. Part II on Approximate Reasoning – Success Stories and Real World Applications contains eight chapters that describe several success stories and real world applications on approximation reasoning. In Chapter 5, “A Fuzzy Set Approach to Software Reliability Modeling” Zeephongsekul provides a discussion of a fuzzy set approach, which is used to extend the notion of software debugging from a 0-1 (perfect/imperfect) crisp approach to one which incorporates some fuzzy sets ideas. Chapter 6, “Computational Methods for Investment Portfolio: the Use of Fuzzy Measures and Constraint Programming for Risk Management” by Majoc et al. present a state of the art on computational techniques for portfolio management, that is, how to optimize a portfolio selection process and propose a novel approach involving utility-based multi-criteria decision making setting and fuzzy integration over intervals. In Chapter 7, “A Bayesian Solution to the Modifiable Areal Unit Problem” Hui explores how the Modifiable Areal Unit Problem (MAUP) can be described and potentially solved by the Bayesian estimation (BYE). Specifically, the scale and the aggregation problems are analyzed using simulated data from an individualbased model. In chapter 8, “Fuzzy Logic Control in Communication Networks” by Chrysostomou and Pitsillides discuss the difficulty of the congestion control problem and review the control approaches currently in use. The authors motivate the utility of Computational Intelligence based control and then through a number of examples, illustrate congestion control methods based on fuzzy logic control.

Preface

VII

In Chapter 9, “Adaptation in Classification Systems” Bouchachia investigates adaptation issues in learning classification systems from different perspectives. Special attention is given to adaptive neural networks and the most visible incremental learning mechanisms. Adaptation is also incorporated in the combination of incremental classifiers in different ways so that adaptive ensemble learners are obtained. These issues are illustrated by means of a numerical simulation. In Chapter 10, “Music Instrument Estimation in Polyphonic Sound Based on Short-Term Spectrum Match” Jiang et al. provide a new solution to an important problem of instrument identification in polyphonic music: There is loss of information on non-dominant instruments during the sound separation process due to the overlapping of sound features. Experiments show that the sub-patterns detected from the power spectrum slices contain sufficient information for the multiple-timbre estimation tasks and improve the robustness of instrument identification. In Chapter 11, “Ultrasound Biomicroscopy Glaucoma Images Analysis Based on Rough Set and Pulse Coupled Neural Network” El-Dahshan et al. present rough sets and pulse coupled neural network scheme for Ultrasound Biomicroscopy (UBM) glaucoma images analysis. The Pulse Coupled Neural Network (PCNN) with a median filter was used to adjust the intensity of the UBM images. This is followed by applying the PCNN-based segmentation algorithm to detect the boundary of the interior chamber of the eye image. Then, glaucoma clinical parameters are calculated and normalized, followed by application of a rough set analysis to discover the dependency between the parameters and to generate set of reducts that contains minimal number of attributes. In Chapter 12, “An overview of fuzzy c-means based image clustering algorithm” Huiyu Zhou and Gerald Schaefer provide an overview of several fuzzy cmeans based image clustering concepts and their applications. In particular, we summarize the conventional fuzzy c-means (FCM) approaches as well as a number of its derivatives that aim at either speeding up the clustering process or at providing improved or more robust clustering performance. We are very much grateful to the authors of this volume and to the reviewers for their great efforts by reviewing and providing interesting feedback to authors of the chapter. The editors would like to thank Dr. Thomas Ditzinger (Springer Engineering Inhouse Editor, Studies in Computational Intelligence Series), Professor Janusz Kacprzyk (Editor-in-Chief, Springer Studies in Computational Intelligence Series) and Ms. Heather King (Editorial Assistant, Springer Verlag, Heidelberg) for the editorial assistance and excellent cooperative collaboration to produce this important scientific work. We hope that the reader will share our joy and will find it useful! December 2008

Aboul Ella Hassanien, Cairo, Egypt Ajith Abraham, Trondheim, Norway Francisco Herrera, Granada, Spain

Contents

Part I: Approximate Reasoning - Theoretical Foundations and Applications Approximate Reasoning - Theoretical Foundations Fuzzy Sets, Near Sets, and Rough Sets for Your Computational Intelligence Toolbox . . . . . . . . . . . . . . . . . . . . . . . . . James F. Peters

3

Fuzzy without Fuzzy: Why Fuzzy-Related Aggregation Techniques Are Often Better Even in Situations without True Fuzziness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hung T. Nguyen, Vladik Kreinovich, Fran¸cois Modave, Martine Ceberio

27

Intermediate Degrees Are Needed for the World to Be Cognizable: Towards a New Justification for Fuzzy Logic Ideas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hung T. Nguyen, Vladik Kreinovich, J. Esteban Gamez, Fran¸cois Modave, Olga Kosheleva

53

Paraconsistent Annotated Logic Program Before-after EVALPSN and Its Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kazumi Nakamatsu

75

Part II: Approximate Reasoning - Success Stories and Real World Applications A Fuzzy Set Approach to Software Reliability Modeling . . . . . 111 P. Zeephongsekul

X

Contents

Computational Methods for Investment Portfolio: The Use of Fuzzy Measures and Constraint Programming for Risk Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 Tanja Magoˇc, Fran¸cois Modave, Martine Ceberio, Vladik Kreinovich A Bayesian Solution to the Modifiable Areal Unit Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 C. Hui Fuzzy Logic Control in Communication Networks . . . . . . . . . . . . 197 Chrysostomos Chrysostomou, Andreas Pitsillides Adaptation in Classification Systems . . . . . . . . . . . . . . . . . . . . . . . . . 237 Abdelhamid Bouchachia Music Instrument Estimation in Polyphonic Sound Based on Short-Term Spectrum Match . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259 Wenxin Jiang, Alicja Wieczorkowska, Zbigniew W. Ra´s Ultrasound Biomicroscopy Glaucoma Images Analysis Based on Rough Set and Pulse Coupled Neural Network . . . . 275 El-Sayed A. El-Dahshan, Aboul Ella Hassanien, Amr Radi, Soumya Banerjee An Overview of Fuzzy C-Means Based Image Clustering Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295 Huiyu Zhou, Gerald Schaefer Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311

Part I

Fuzzy Sets, Near Sets, and Rough Sets for Your Computational Intelligence Toolbox James F. Peters Computational Intelligence Laboratory, Department of Electrical & Computer Engineering, University of Manitoba, E1-390 Engineering Building, 75A Chancellor’s Circle, Winnipeg, Manitoba R3T 5V6 Canada [email protected]

Summary. This chapter considers how one might utilize fuzzy sets, near sets, and rough sets, taken separately or taken together in hybridizations as part of a computational intelligence toolbox. These technologies offer set theoretic approaches to solving many types of problems where the discovery of similar perceptual granules and clusters of perceptual objects is important. Perceptual information systems (or, more concisely, perceptual systems) provide stepping stones leading to nearness relations and properties of near sets. This work has been motivated by an interest in finding a solution to the problem of discovering perceptual granules that are, in some sense, near each other. Fuzzy sets result from the introduction of a membership function that generalizes the traditional characteristic function. Near set theory provides a formal basis for observation, comparison and classification of perceptual granules. Near sets result from the introduction of a description-based approach to perceptual objects and a generalization of the traditional rough set approach to granulation that is independent of the notion of the boundary of a set approximation. Near set theory has strength by virtue of the strength it gains from rough set theory, starting with extensions of the traditional indiscernibility relation. This chapter has been written to establish a context for three forms of sets that are now part of the computational intelligence umbrella. By way of introduction to near sets, this chapter considers various nearness relations that define partitions of sets of perceptual objects that are near each other. Every perceptual granule is represented by a set of perceptual objects that have their origin in the physical world. Objects that have the same appearance are considered perceptually near each other, i.e., objects with matching descriptions. Pixels, pixel windows, and segmentations of digital images are given by way of illustration of sample near sets. This chapter also briefly considers fuzzy near sets and near fuzzy sets as well as rough sets that are near sets.

This author gratefully acknowledges the insights and suggestions by Christopher Henry, Piotr Wasilewski and Andrzej Skowron concerning topics in this paper. This research has been supported by the Natural Sciences & Engineering Research Council of Canada (NSERC) grant 185986.

A.-E. Hassanien et al. (Eds.): Foundations of Comput. Intel. Vol. 2, SCI 202, pp. 3–25. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com

4

J.F. Peters

The main contribution of this chapter is the introduction of a formal foundation for near sets considered in the context of fuzzy sets and rough sets. Keywords: Description, fuzzy sets, near sets, perceptual granule, perceptual system, rough sets. Near To How near to the bark of a tree are drifting snowflakes, swirling gently round, down from winter skies? How near to the ground are icicles, slowly forming on window ledges? –Fragment of a Philosophical Poem [27]. –Z. Pawlak & J.F. Peters, 2002.

1 Introduction This chapter considers how one might utilize fuzzy sets, near sets, and rough sets, considered separately and taken together as part of a computational intelligence toolbox. Near set theory provides a formal basis for observation, comparison and classification of perceptual granules. Near sets and the perception of nearness of objects were inspired by images in a philosophical poem written in 2002 [14]. Since that time, a considerable number of papers have been written about near set theory [21, 20, 29] and its applications [2, 4, 5, 3, 26, 25]. Near sets result from the introduction of a description-based approach [23] to the identification and analysis of perceptual objects and a generalization of the traditional rough set approach to granulation that is independent of the notion of the boundary of a set approximation. Perceptual information systems (or, more concisely, perceptual systems) provide stepping stones leading to nearness relations and properties of near sets. This work has been motivated by an interest in finding a solution to the problem of discovering perceptual granules that are, in some sense, near each other. Near set theory provides a formal basis for observation, comparison and classification of perceptual granules. A perceptual granule is defined by a collection of objects that are graspable by the senses or by the mind. This is made clear in this article by considering various nearness relations that define partitions of sets of perceptual objects that are near each other. Every perceptual granule is represented by a set of perceptual objects that have their origin in the physical world. Objects that have the same appearance are considered perceptually near each other, i.e., objects with matching descriptions. Pixels, pixel windows, and segmentations of digital images are given by way of illustration of sample near sets. This chapter also briefly presents near sets arising from fuzzy sets and rough sets. The main contribution of this chapter is an overview of the basics of near sets considered separately and in the context of fuzzy sets and rough sets. This chapter has the following organization. The basic notion of a perceptual system is presented in Sect. 2. Definitions and illustration of indiscernibility, weak

Fuzzy Sets, Near Sets, and Rough Sets

5

indiscernibility and weak tolerance relations are given in Sect. 3. Building on these relations, nearness relation are defined and illustrated in Sect. 4. Nearness relations lead to the introduction of perceptual near sets in Sect. 5. A short introduction to fuzzy near sets and near fuzzy sets is given in Sect. 6. The strong ties between near sets and rough sets are briefly discussed in Sect. 7.

2 Perceptual Systems The word perception indicates a direction rather than a primitive function. It is known that the uniformity of apparent size of objects at different distances, or of their colour in different lights, is more perfect in children than in adults. –Phenomenology of Perception. –Maurice Merleau-Ponty, 1945. Thus the near thing, as ‘the same’, appears from now from this ‘side’, now from that; and the ‘visual perspectives’ change–also, however, the other manners of appearance (tactile, acoustic, and so forth), as we can observe by turning our attention in the right direction. –Cartesian Meditations. –Edmund Husserl, 1929.

This section briefly presents the basis for perceptual systems that hearkens back to the original notion of a deterministic information system introduced by Zdzisław Pawlak [13] and elaborated in [11, 10]. 2.1

Perceptual Object Descriptions

Perceptual objects are known by their descriptions. An object description is defined by means of a tuple of function values φ (x) associated with an object x ∈ X (see Table 1). The important thing to notice is the choice of functions φi ∈ B used to describe an object of interest. Assume that B ⊆ F (see Table 1) is a given set of functions representing features of sample objects X ⊆ O and F is finite. Let φi ∈ B, where φi : O −→ R. In combination, the functions representing object features provide a basis for an object description φ : O −→ Rl , a vector containing measurements (returned values) associated with each functional value φi (x) for x ∈ X, where |φ | = l, i.e. the description length is l. Object Description: φ (x) = (φ1 (x), φ2 (x), . . . , φi (x), . . . , φl (x)). The intuition underlying a description φ (x) is a recording of measurements from sensors, where each sensor is modelled by a function φi .

6

J.F. Peters Table 1. Description Symbols Symbol Interpretation R O X x F B φ l i φi φ (x) X, F

Set of real numbers, Set of perceptual objects, X ⊆ O, set of sample objects, x ∈ O, sample object, A set of functions representing object features, B ⊆ F, φ : O → Rl , object description, l is a description length, i ≤ l, φi ∈ B, where φi : O −→ R, probe function, φ (x) = (φ1 (x), . . . , φi (x), . . . , φL (x)), description, φ (x1 ), . . . , φ (x|X | ), i.e., perceptual information system.

Let X ,Y ⊆ O denote sets of perceptual objects. Sets X ,Y ⊆ O are considered near each other if the sets contain perceptual objects with at least partial matching descriptions. A perceptual object x ∈ O is something presented to the senses or knowable by the mind [9]. In keeping with the approach to pattern recognition suggested by Pavel [12], the features of an object such as contour, colour, shape, texture, bilateral symmetry are represented by probe functions. A probe function can be thought of as a model for a sensor. A probe makes it possible to determine if two objects are associated with the same pattern without necessarily specifying which pattern (classification). A detailed explanation about probe functions vs. attributes in the classification of objects is given in [19]. 2.2

Perceptual Systems: Specialized Deterministic Systems

For representing results of a perception, the notion of a perceptual system is briefly introduced in this section. In general, an information system is a triple S = Ob, At, {Val f } f ∈At where Ob is a set of objects, At is a set of functions representing either object features or object attributes, and each Val f is a value domain of a function f ∈ At, where f : Ob −→ P(Val f ) (P(Val f ) is a power set of Val f ). If f (x) = 0/ for all x ∈ Ob and f ∈ At, then S is total. If card( f (x)) = 1 for every x ∈ Ob and f ∈ At, then S is deterministic. Otherwise S is non-deterministic. In the case, when f (x) = {v}, {v} is identified with v. An information system S is real valued iff Val f = R for every f ∈ At. Very often a more concise notation is used: Ob, At,especially when value domains are understood, as in the case of real valued information systems. Since we discuss results of perception, as objects we consider perceptual objects while f ∈ At are interpreted as probe functions. Two examples of perceptual systems are given in Table 2. Definition 1. Perceptual System A perceptual system O, F is a real valued total deterministic information system where O is a non-empty set of perceptual objects, while F a countable set of probe functions.


7

Table 2. Sample perceptual information systems Sys. 1 X x1 x2 x3 x4 x5 x6 x7 x8 x9 x10

φ1 0 0 1 1 0 0 1 1 0 1

φ2 1 1 2 3 1 2 2 3 1 1

φ3 0.1 0.1 0.05 0.054 0.03 0.02 0.01 0.01 0.5 0.5

φ4 0.75 0.75 0.1 0.1 0.75 0.75 0.9 0.1 0.1 0.25

1.1: SCC leaf 1.2: Shubert stomata 10x

d 1 0 0 1 1 0 1 0 1 1

Sys. 2 Y y1 y2 y3 y4 y5 y6 y7 y8 y9 y10 y11 y12 y13 y14

φ1 0 1 1 1 1 1 0 0 0 1 1 1 1 1

φ2 2 1 1 3 4 4 2 3 3 2 4 4 1 4

1.3: Pin leaf

φ3 0.2 0.25 0.25 0.5 0.6 0.6 0.4 0.5 0.5 0.7 0.6 0.7 0.25 0.6

φ4 0.01 0.01 0.01 0.55 0.75 0.75 0.2 0.6 0.6 0.4 0.8 0.9 0.01 0.75

d 0 0 0 0 1 1 0 1 1 0 1 0 0 1

1.4: Pin stomata 100x

Fig. 1. Sample Percepts

The notion of a perceptual system admits a wide variety of different interpretations that result from the selection of sample perceptual objects contained in a particular sample space O. Perceptual objects are known by their descriptions. 2.3

Sample Perceptual System

By way of an illustration, let P, φ denote a perceptual system where P is a set of microscope images and φ is a probe function representing luminance contrast1 , respectively. A sample Shubert choke cherry leaf and Native Pin choke cherry leaf are shown in Figures 1.1 and 1.3. Leaf stomata (minute pores in the epidermis of a leaf) are shown in the microscope images magnified by 10x in Fig. 1.2 for the 1

In digital images, luminance contrast can be controlled by converting irradiance (amount of light per unit area) into a grey value g using a function g(E) = E γ , where E denotes irradiance level and luminance varies non-linearly with γ (gamma) typically having a value of 0.4 [8].

8

J.F. Peters

sample Shubert CC leaf and by 100x in Fig. 1.4 for the sample Native Pin CC leaf. Intuitively, if we compare image colours, luminance contrast or sub image shapes, the microscope leaf images are similar. By considering nearness relations in the context of a perceptual system, it is possible to classify sets of perceptual objects. A formal basis for the discovery of different forms of near sets is the focus of the remaining sections of this chapter.

3 Relations and Classes The basic idea in the near set approach to object recognition is to compare object descriptions. Sample perceptual objects x, y ∈ O, x = y are near each other if, and only if x and y have similar descriptions. Similarly, sets X ,Y are perceptually near each other in the case where there is at least one pair of objects x ∈ X, y ∈ Y that have similar descriptions. In this section, two kinds of indiscernibility relations and a tolerance relation are briefly introduced. These relations make it possible to define various nearness relations and make it possible to provide a formal foundation for near sets. Because of the importance of image analysis as an application area for near sets, this section illustrates the relations both with images and with perceptual information tables. This practise is continued in the sequel to this section. 3.1

Indiscernibility and Tolerance Relations

Recall that each φ defines the description of an object (see Table 1). To establish a nearness relation, we first consider the traditional indiscernibility relation. Let B ⊆ F denote a set of functions representing perceptual object features. The indiscernibility relation ∼B introduced by Zdzisław Pawlak [13] is distinguished from weak indiscernibility introduced introduced by Ewa Orłowska [10]. In keeping with the original indiscernibility relation symbol ∼F [13], the symbol is used to denote weak indiscernibility instead of the notation wind [10]. The pioneering work by Andrzej Skowron and Janislaw Stepaniuk on tolerance relations [32] plays an important role in the providing a foundation for a near set approach to tolerance spaces. Definition 2. Indiscernibility Relation Let O, F be a perceptual system. For every B ⊆ F the indiscernibility relation ∼B is defined as follows: ∼B = {(x, y) ∈ O × O | ∀ φ ∈ B, φ (x) − φ (y) = 0}, where · represents the l 2 norm. If B = {φ } for some φ ∈ F, for simplicity, instead of ∼{φ } , we write ∼φ . Example 1. Clustering Matching Leaf Pixels For the two microscopic views of leaf stomata shown in Fig. 1.2 (Shubert choke cherry leaf stomata) and Fig. 1.4 (Native Pin cherry leaf stomata), consider the grey scale view of fragments of these leaves shown in Fig. 2. There areas of both leaf


9

Fig. 2. Near Leaf Fragments

fragments that contain pixels with matching grey levels. This view of the two leaf fragments is expressed formally in the context of the indiscernibility relation. Let g denote a probe function that measures the grey level of a pixel and let x, y denote pixels in leaf fragments shown in Fig 2. Then the relation ∼g is defined in the following way: x ∼g y ⇐⇒ g(x) = g(y). Classes of pixels with the same grey level are denoted by x/∼g (Pin Cherry) and y/∼g (Shubert CC) in Fig. 2. In the case where two classes of pixels are in relation ∼g to each other, we obtain a cluster of pixels. For example, x ∼g y suggests the formation of a cluster of pixels denoted by x/∼g ∪ y/∼g . A sample cluster of pixels with matching grey levels is depicted in Fig 2. The individual classes containing pixels with matching descriptions and the cluster of the two classes in Fig. 2 are examples of perceptual granules. Let ImPin , F denote perceptual system Sys. L1 with ImPin = {ImPin/∼g | ImPin/∼g = class of pixels in Pin cherry leaf}, FImPin = {g}. Similarly, let ImShubert , F denote perceptual system Sys. L2 with ImShubert = {ImShubert /∼g | ImShubert /∼g = class of pixels in Shubert CC leaf}, FImShubert = {g}. An obvious extension (not shown here) is the partition of any leaf fragment into non-overlapping sets of pixels having matching descriptions, i.e., sets of pixels with

10

J.F. Peters

matching grey levels. It is important to notice that each particular set of matching pixels may be in regions of the image that are not contiguous, i.e., pixels with matching grey levels can be located anywhere in an image. Example 2. Let O1 , F1 denote perceptual system Sys. 1 with O1 = {x1 , ... , x9 }, F1 = {φ1 , φ2 , φ3 , φ4 }, where the values of probe functions from F1 are given in the lefthand side of table 2. Similarly, let O2 , F2 denote perceptual system Sys. 2 with O2 = {y1 , ... , x14 }, F2 = {φ1 , φ2 , φ3 , φ4 }, where the values of the probe functions from F1 are given in the righthand side of table 2. The perceptual systems O1 , F1 , O2 , F2 have partitions (1) and (2) of the space of percepts defined by relations ∼F1 and ∼F2 . O1/∼F = {{x1 , x2 }, {x3 }, {x4 }, {x5 }, {x6 }, {x7 }, {a8 }, {x9 }, {x10 }}, 1

(1)

O2/∼F = {{y1 }, {y2 , y3 , y13 }, {y4 }, {y5 , y6 }, {y7 }, {y8 , y9 }, {y10 }, {y11 }, {y12 }, {y14 }}. 2 (2)

If we consider only probe function φ3 relative to O1 , then we obtain, e.g., several equivalence classes such as (3), each containing a pair of objects. x1/∼φ = {x1 , x2 },

(3)

x7 /∼φ = {x7 , x8 },

(4)

x9/∼φ = {x9 , x10 }.

(5)

3 3 3

Again, for example, if we probe O2 with φ3 , we obtain, e.g., a number of multiobject classes such as the one in (6). y2/∼φ = {y2 , y3 , y13 },

(6)

y4/∼φ = {y4 , y8 , y9 },

(7)

y5 /∼φ = {y5 , y6 , y11 , y14 },

(8)

3 3 3

y10/∼φ = {y10 , y12 }. 3

(9)

Definition 3. Weak Indiscernibility Relation Let O, F be a perceptual system. For every B ⊆ F the weak indiscernibility relation B is defined as follows: B = {(x, y) ∈ O × O | ∃ φ ∈ B, φ (x) − φ (y) = 0}. If B = {φ } for some φ ∈ F, instead of {φ } we write φ . Example 3. Let O1 , F1 denote perceptual system Sys. 1 with O1 = {x1 , ... , x9 }, F1 = {φ1 , φ2 , φ3 , φ4 }, where the values of probe functions from F1 are given in the lefthand side of table 2. Similarly, let O2 , F denote perceptual system Sys. 2 with O2 = {y1 , ... , y14 }, F = {φ1 , φ2 , φ3 , φ4 }, where the values of the probe functions from F are given in the righthand side of table 2. Let X ⊂ O1 , X = {x1 , x9 , x10 } and


11

Y ⊂ O2 ,Y = {y1 , y8 , y10 , y11 , y12 }. Consider partitions X /φ and Y /φ given in (10) 3 3 and (11), respectively. X /φ = {{x1 } , {x9 , x10 }} ,

(10)

Y /φ = {{y1 } , {y8 } , {y10 } , {y11 } , {y12 }} ,

(11)

3

3

Remark 1. Notice that the class {x1 } ∈ X /φ contains only a single object, since 3 there is no other object in x ∈ X such that φ3 (x1 ) = φ3 (x). Similarly, each of the classes in Y /φ contains only a single object. 3

Definition 4. Weak Tolerance Relation Let O, F be a perceptual system and let ε ∈ ℜ (reals). For every B ⊆ F the weak tolerance relation ∼ =B,ε is defined as follows: ∼ =B,ε = {(x, y) ∈ O × O | ∃ φ ∈ B, φ (x) − φ (y) ≤ ε }. That is, in general, the relation ∼B,ε is reflexive and symmetric but not transitive. This relation is very important in discovering near sets, since it defines tolerance classes relative to a threshold ε , rather than require strict equality of probe function values in the case of the indiscernibility relations ∼B and B (see, e.g., [24]). Remark 2. Notice that Def. 4 represents a special case. That is, in general, the sets X and Y represent sample sets of observations from distinct perceptual systems. In effect, it is possible to state a Proposition to this effect. Definition 5. Let P1 = O1 , F denote perceptual system P1. Similarly, let P2 = O2 , F denote a second, distinct perceptual system. Also, let ε ∈ ℜ. P1 has a weak tolerance relation to P2 if, and only if O1 ∼F,ε O2 . Proposition 1. Let Sys1 = O1 , F denote perceptual system Sys1. Similarly, let Sys2 = O2 , F denote a second, distinct perceptual system with the same set of features F. Let B ⊆ F and choose ε . Then Sys1∼B,ε Sys1 ⇐⇒ O1 ∼B,ε O2 . Example 4. Clusters of Similar Leaf Pixels For the two microscopic view of leaf stomata shown in Fig. 1.2 (Shubert choke cherry leaf stomata) and Fig. 1.4 (Native Pin cherry leaf stomata), consider the greyscale view of fragments of these leaves shown in Fig. 2. Let g denote the grey level of a pixel and let x, y denote pixels in leaf fragments shown in Fig 3. Then the relation ∼ ={g},ε is defined in the following way: x ∼ ={g},ε y ⇐⇒ g(x) = g(y). Classes of pixels with the same grey level are denoted by x/∼ ={g},0.2 (Native Pin cherry) and y/∼ ={g},0.2 (Shubert choke cherry) in Fig. 3 for ε = 0.2. In the case where

12

J.F. Peters

Fig. 3. Near Leaf Pixels

two classes of pixels are in relation ∼ ={g},ε to each other, we obtain a cluster of similar pixels. For example, x ∼ ={g},0.2 y suggests the formation of a cluster of pixels denoted by x/∼ ={g},0.2 ∪ y/∼ ={g},0.2 . A sample cluster of similar pixels with the norm of grey level differences within ε = 0.2 of each other is depicted in Fig 3. Example 5. Let O1 , F denote perceptual system Sys. 1 with O1 = {x1 , ... , x9 }, F = {φ1 , φ2 , φ3 , φ4 }, where the values of probe functions from F are given in the lefthand side of table 2. Similarly, let O2 , F denote perceptual system Sys. 2 with O2 = {y1 , ... , y14 }, F = {φ1 , φ2 , φ3 , φ4 }, where the values of the probe functions from F are given in the righthand side of table 2. Let ε = 0.1 for both perceptual systems. For example, let φ3 ∈ F1 . The perceptual system O1 , {φ3 } has tolerance classes (12), (13), (14) defined by relation φ3,0.1 . x1/φ

= {x1 , x2 , x5 , x6 , x7 , x8 },

(12)

x3/φ

= {x3 , x4 },

(13)

x9 /φ

= {x9 , x10 }.

(14)

3 ,0.1 3 ,0.1 3 ,0.1

For example, in x3/φ

3 ,0.1

, we have

|φ3 (x3 ) − φ3 (x4 )| = |0.05 − 0.054| ≤ 0.1 Similarly, the perceptual system O2 , {φ3 } has tolerance classes (15), (16), (17), (18) defined by relation φ3,0.1 .


y1 /φ

= {y1 , y2 , y3 , y13 },

(15)

y4 /φ

= {y4 , y5 , y6 , y8 , y9 , y11 , y14 },

(16)

y7 /φ

= {y7 , y4 , y8 , y9 },

(17)

= {y5 , y6 , y10 , y11 , y12 , y14 },

(18)

3 ,0.1 3 ,0.1 3 ,0.1

y10 /φ

3 ,0.1

For example, in y7 /φ

13

3 ,0.1

, we have

|φ3 (y7 ) − φ3 (y4 )| = |0.4 − 0.5| ≤ 0.1, |φ3 (y7 ) − φ3 (y8 )| = |0.4 − 0.5| ≤ 0.1, |φ3 (y7 ) − φ3 (y9 )| = |0.4 − 0.5| ≤ 0.1, |φ3 (y8 ) − φ3 (y9 )| = |0.5 − 0.5| ≤ 0.1

4 Nearness Relations Nearness. The state, fact or quality of being near. 1. Close kinship or relationship. –Oxford English Dictionary, 1933.

Three basic nearness relations are briefly presented and illustrated in terms of visual as well as numerical perceptual systems in this section. In keeping with the intuitive notion of nearness enunciated in the Oxford English Dictionary, one establishes the nearness of perceptual granules (e.g., perceptual objects such as image pixels, classes of pixels with matching descriptions, and clusters of classes of pixels with similar descriptions) by discovering relations that underlying perceptions of ‘closeness’. Nearness relations themselves are defined relative to the fundamental relations coming from rough set theory [13] and near set theory [20, 30, 24]. Table 3. Relation and Partition Symbols Symbol B φ ε ∼B B ∼B,ε x/∼B O/∼B φ (x) − φ (y)

Interpretation see Table 1, probe function in B, ε ∈ [0, 1], {(x, y) | f (x) = f (y) ∀ f ∈ B}, indiscernibility relation [13], weak indiscernibility relation [10], weak tolerance relation, x/∼B = {y ∈ X | y ∼B x}, elementary set (class), O/∼B = {x/∼B | x ∈ O}, quotient set, · = l 2 norm, nearness relation symbol, weak nearness relation symbol, weak tolerance nearness relation symbol.

14

J.F. Peters

Definition 6. Nearness Relation [30] Let O, F be a perceptual system and let X,Y ⊆ O. The set X is perceptually near to the set Y (X F Y ), if and only if there are x ∈ X and y ∈ Y such that x ∼F y (see Table 3). Example 6. Perceptually Near Leaf Fragments Consider the perceptual systems ImPin , F, ImShubert , F from ImPin = {ImPin/∼g | ImPin/∼g = class of pixels in Pin cherry leaf}, FImPin = {g}, ImShubert = {ImShubert /∼g | ImShubert /∼g = class of pixels in Shubert CC leaf}, FImShubert = {g}, ImPin g ImShubert . Example 7. Consider the perceptual systems O1 , F, O2 , F given in Table 2. From Example 3, we obtain B = {φ3 }, where φ3 ∈ F, Xnew = x9/∼φ , from Example 3, 3

= {x9 , x10 }, Ynew = y8 /∼φ

3

= {y4 , y8 , y9 }, Xnew φ3 Ynew , since φ3 (x9 ) = φ3 (y8 ) = 0.5 Definition 7. Weak Nearness Relation [30] Let O, F be a perceptual system and let X,Y ⊆ O. The set X is weakly near to the set Y within the perceptual system O, F (X F Y ) iff there are x ∈ X and y ∈ Y and there is B ⊆ F such that x B y. If a perceptual system is understood, then we say shortly that a set X is weakly near to set Y (see Table 3). Example 8. Weakly Near Leaf Fragments Let r, g, b, gr denote red, green, blue, grey in the RGB colour model2 . Consider the perceptual systems ImPin , FImPin , ImShubert , FImShubert in Example 1, where FImPin , FImShubert contain more than one probe function. ImPin = {ImPin/∼gr | ImPin/∼g = class of pixels in Pin cherry leaf}, FImPin = {r, g, b, gr}, 2

R G B e.g., r = R+G+B , g = R+G+B , b = R+G+B , gr = R+G+B , where R, G, B represent the 3 amounts of red, green, blue used to form a particular colour, also known as at the tristimulus values [8].


15

ImShubert = {ImShubert /∼g | ImShubert /∼g = class of pixels in Shubert CC leaf}, FImShubert = {r, g, b, gr}, ImPin gr ImShubert . Example 9. Consider the perceptual systems O1 , F, O2 , F given in Table 2. B = {φ3 }, where φ3 ∈ F, X = {x1 , x2 , x7 , x8 , x9 , x10 }, Y = {y4 , y5 , y6 , y8 , y9 , y11 }, X φ3 Y, since we can find x ∈ X , y ∈ Y where x φ3 y, e.g.,

φ3 (x9 ) = φ3 (y8 ) = 0.5. Definition 8. Weak Tolerance Nearness Relation [24] Let O, F be a perceptual system and let X ,Y ⊆ O, ε ∈ [0, 1]. The set X is perceptually near to the set Y within the perceptual system O, F (X F Y ) iff there exists x ∈ X , y ∈ Y and there is a φ ∈ F, ε ℜ such that xB,ε y (see Table 3). If a perceptual system is understood, then we say shortly that a set X is perceptually near to a set Y in a weak tolerance sense of nearness. Example 10. Weak Tolerance Near Leaf Fragments From Example 8, consider

ε = 0.2, ImPin = {ImPin/∼gr | ImPin/∼g = class of pixels in Pin cherry leaf}, FImPin = {r, g, b, gr}, ImShubert = {ImShubert /∼g | ImShubert /∼g = class of pixels in Shubert CC leaf}, FImShubert = {r, g, b, gr}, ImPin gr, 0.2 ImShubert . An example of a cluster of weak tolerance near sets is shown in Fig. 3. Example 11. Let O1 , F denote perceptual system Sys. 1 with O1 = {x1 , ... , x9 }, F = {φ1 , φ2 , φ3 , φ4 }, where the values of probe functions from F are given in the lefthand side of table 2. Similarly, let O2 , F denote perceptual system Sys. 2 with O2 = {y1 , ... , y14 }, F = {φ1 , φ2 , φ3 , φ4 }, where the values of the probe functions from F are given in the righthand side of table 2. Now choose ε and arbitrary samples X1 and Y1 so that they are also weak tolerance near sets.

ε = 0.1, B = {φ3 }, where φ3 ∈ F, X1 ∈ O1 ,Y1 ∈ O2 ,

16

J.F. Peters

X1 = {x1 , x2 , x7 , x8 , x9 , x10 }, Y1 = {y4 , y5 , y6 , y8 , y9 , y11 }, X1 φ Y1 , since we can find x ∈ X, y ∈ Y where xφ3 ,ε y, e.g., 3

|φ3 (x9 ) − φ3 (y8 )| = |0.5 − 0.5| = 0 ≤ 0.1; again, e.g., |φ3 (x10 ) − φ3 (y11 )| = |0.1 − 0.2| = 0.1 Remark 3. In Example 11, we know that X F Y , since there exists an x ∈ X, y ∈ Y (namely, x9 , y8 ) such that |φ3 (x) − φ3 (y)| ≤ ε We can generalize the result from Example 11 in Prop 2 by extending the idea in Prop. 1. Proposition 2. Let Sys1 = O1 , F denote perceptual system Sys1. Similarly, let Sys2 = O2 , F denote a second, distinct perceptual system. Then Sys1 F Sys1 ⇐⇒ O1 F O2 .

5 Near Sets Object recognition problems, especially in images [4], and the problem of the nearness of objects have motivated the introduction of near sets (see, e.g., [21]). Since we are mainly interested in comparing perceptual granules for selected realvalued probe functions, only weakly near sets and weakly tolerant near sets are briefly considered in this section based on the weak near relation [30, 24] F in Def. 7 and weak tolerance nearness relation [24] F in Def. 8. These two forms of near sets are especially useful in discovering near fuzzy and fuzzy near sets as well as rough near and near rough sets. Other forms of near sets are introduced in [21, 20, 30]. 5.1

Weakly Near Sets

Definition 9. Weakly Near Sets Let O, F be a perceptual system and let X,Y ⊆ O, X = Y . Let F denote a non-empty set of probe functions representing features of objects in O. A set X is weakly near Y iff X F Y , i.e., there exists φ ∈ F, x ∈ X , y ∈ Y such that x ∼ =φ y. Example 12. Weakly Near Sub-Images Let Im, F be a perceptual information system for a sample space such that Im is an image, i.e., perceptual objects here mean pixels in Im. Let X ,Y ⊆ Im, X = 0,Y / = 0, / i.e., X and Y are subimages of image Im. Let F denote a set of probe functions representing image features. Let gr : Im → R denote a function that maps each pixel to the grey level of a pixel. Assume that


17

Fig. 4. Motic Stereo Zoom SMZ140/143 Microscope

there are x ∈ X , y ∈ Y such that x ∼gs y, i.e., gr(x) − gr(y) = 0. Therefore, X F Y , i.e., image X is weakly near to Y . An illustration of weakly near images is given in Example 8. 5.2

Tolerance Near Sets

Definition 10. Tolerance Near Sets Let O, F be a perceptual system and let X ⊆ O. A set X is a tolerance near set iff there is Y ⊆ O such that X F Y . The family of near sets of a perceptual system O, F is denoted by NearF (O). In effect, tolerance perceptual near sets are those sets that are defined by the nearness relation F . Example 13. Tolerance Near Images 1 For an illustration of tolerance near images, see Example 10. Example 14. Tolerance Near Images 2 This example illustrates the use of a Motic stereo zoom SMZ140/143 microscope in comparing images showing fossils contained in a sample piece of Dominican amber. This particular microscope has a working distance of 80 mm and a magnification range between 2.5x and 120x. The setup shown in Fig. 4 was used to capture amber images shown in Fig. 5. Amber is an ancient tree resin. At the time when the resin was sticky, insects and other small organisms became trapped on the resin surface and gradually engulfed by the flowing resin [1]. It has also been pointed out that amber preserves organisms in finer fidelity than perhaps any other kind of fossil [1]. The particular amber examined in this example comes from the Dominican Republic. Dominican amber belongs to the mid-Miocene period, approximately 17-20 million years ago [7]. Microscopic images of insects fossilized in Miocene Dominican amber are shown on a 100 µm (micrometer) scale in Fig 5.

18

J.F. Peters

5.1: Miocene moth

5.2: Miocene beetle

Fig. 5. Sample Miocene Dominican amber

6.1: Moth sets

6.2: Beetle sets

Fig. 6. Amber segmentations, 5x5, ε = 0.1

The fragment of Dominican amber in Fig. 5.1 shows a fossil, probably a Acrolophus moth (see, e.g., a similar fossil also preserved in Dominican amber [1]). In another part of the same piece of amber shown in Fig. 5.2, there is another tiny fossil, probably a predatory larva of the cucujoid family Discolomidae (for a similar fossil, see, e.g., [1]). The scale for the two fossils shown in [1] is millimeters, not micrometers. These amber images are compared by segmenting them into 5x5 subimages (see Fig 6) and 3x3 subimages (see Fig 7). Subimages containing pixels with the same average grey level are masked (each with a colour representing a particular grey level). Let gr ¯ denote a function to compute the average grey level of a subimage. And let x, y denote n × n subimages. Then tolerance classes are identified using ∼ =gr, ¯ ε , where ¯ − gr(y) ¯ ≤ ε. x∼ =gr, ¯ ε y ⇐⇒ gr(x) This leads to a collection of tolerance classes for each image. For example, if we consider 5 × 5 subimages and let ε = 0.1, we obtain the tolerance classes shown in Fig. 6. In Fig. 6.1, the moth shows up as a collection tolerance classes represented by a single colour, except for the boundary of the moth that is surrounded by other tolerance classes representing varying average grey levels in the subimages along the border of the fossil. A similar result is obtained in Fig. 6.2 for the fossilized


7.1: Moth sets

19

7.2: Beetle sets

Fig. 7. Amber segmentations, 3x3, ε = 0.01

beetle larva. We get a quite different result, if we consider 3 × 3 subimages and let ε = 0.1. The smaller subimages leads to a more accurate representation of the perceptual granules in the original microscopic images. The goal in this work is to measure the degree of nearness of pairs of images as a means of discovering similar images. A tolerance relation-based nearness measure (19) has recently been introduced [6]. Let F denote a set of probe functions in a perceptual system O1 , F and assume B ⊆ F. Then NM∼ =B =

∑

∑

x /∼ =B ∈X/∼ =B y/∼ =B ∈Y/∼ =B

ξ (x/∼ =B , y/∼ =B ) , max(|x/∼ |, |y /∼B |) =B

(19)

where

ξ (x/∼ =B , y/∼ =B ) =

min(|x/∼ =B |, |y/∼ =B |) 0

, if φ (x) − φ (y) ≤ ε , , otherwise.

For simplicity, we write NM∼ =B instead of NM∼ =B , ε . For this example, assume O consists of n × n subimages. Then NM∼ =B = 0.0817 for 5 × 5 subimages with ε = 0.01. For the same ε , NM∼ =B = 0.460517 for 3 × 3 subimages shown in Fig. 7. This matches our intuition, since the fossils are similar in size but not in shape. In addition, notice that the penumbra region surrounding the border of the two fossils is more pronounced (evident) in the finer-grained tolerance classes in Fig. 7. NM∼ =B in (19) is an example of a characteristic function that defines a fuzzy set (we consider this in the sequel to this section). Example 15. Let O1 , F denote perceptual system Sys. 1 with O1 = {x1 , ... , x9 }, F = {φ1 , φ2 , φ3 , φ4 }, where the values of probe functions from F are given in the lefthand side of table 2. Similarly, let O2 , F denote perceptual system Sys. 2 with O2 = {y1 , ... , y14 }, F = {φ1 , φ2 , φ3 , φ4 }, where the values of the probe functions from F are given in the righthand side of table 2. Now choose samples X and Y

20

J.F. Peters

that are also weak tolerance near sets. Sets X,Y in Example 11 are near sets, since X φ Y . Again, for example, consider the following near sets extracted from 3 Table 2.

ε = 0.3, B = {φ3 }, X1 ∈ O1 ,Y1 ∈ O2 , X1 = {x1 , x2 , x5 , x6 , x7 , x8 , x9 , x10 }, Y1 = {y4 , y5 , y6 , y8 , y9 , y10 , y11 , y12 }, X1 φ Y1 , since we can find x ∈ X1 , y ∈ Y1 , where 3

xφ3,0.3 y, e.g.,x9 φ3,0.3 y10 , since |φ3 (x9 ) − φ3 (y10 )| = |0.5 − 0.7| = 0.2 ≤ 0.3 The basic idea here is to look for sets of objects containing at least one pair of objects that satisfy the weak tolerance relation. Consider, for example, sets X2 ∈ O2 ,Y1 ∈ O2 extracted from Table 2 in (23) and (24).

ε = 0.3 B = {φ4 },

(20) (21)

X2 ∈ O2 ,Y1 ∈ O2 , X2 = {x1 , x2 , x5 , x6 , x7 , x8 , x9 },

(22) (23)

Y2 = {y5 , y6 , y8 , y9 , y10 , y11 , y12 , y14 }, X2 φ Y2 , since we can find x ∈ X2 , y ∈ Y2 , where

(24) (25)

3

xφ4,0.3 y, e.g., x1 φ4,0.3 y8 , since |φ4 (x1 ) − φ4 (y8 )| = |0.75 − 0.6| = 0.15 ≤ 0.3; again, e.g., x7 φ4,0.3 y11 , since |φ4 (x7 ) − φ4 (y11 )| = |0.9 − 0.8| = 0.1 ≤ 0.3

6 Fuzzy Sets Fuzzy sets were introduced by Lotfi A. Zadeh in 1965 [33] viewed as a generalization of traditional set. Definition 11. Fuzzy Set Let X denote a set of objects and let A ⊆ X . A fuzzy set is a pair (A, μA ) such that μ : A → [0, 1]. The membership function μ generalizes the usual characteristic function ε : X → {0, 1}.


21

Fig. 8. Sample Fuzzy Near Sets

6.1

Near Fuzzy Sets

Fuzzy sets A1 and A2 shown in Fig. 8 are also near sets inasmuch as each fuzzy set has a non-empty core. Let X be a problem domain for a fuzzy set A. By definition [18], the core of A is core(A) = {x ∈ X | A(x) = 1} . The core of A is an example of a probe function that defines the class xcore(A) = {y ∈ X | core(A)(x) = core(A)(y) = 1} . It can also be argued that X, core(A) is a perceptual system. In the case where a pair of fuzzy sets has non-empty cores, then the fuzzy sets satisfy the condition for the weak nearness relation, i.e., we can find x ∈ X, y ∈ Y for (X, A1), (Y, A2) where x core y. In effect, A1 core A2. Proposition 3. Fuzzy sets with non-empty cores are near sets. 6.2

Fuzzy Near Sets

Fuzzy sets that are near each other can be partitioned into classes containing objects that have matching descriptions. For example, in Fig. 9, two fuzzy sets are shown: (X, f 1), where X = [0, 1.5], f 1(x) = |sin(x)sin(10x)| , (Y, f 2)where Y = [0.5, 1.5], f 2(y) = |sin(y)sin(10y)|.

22

J.F. Peters

Fig. 9. Sample Fuzzy Sets

Fig. 10. Sample Fuzzy Near Sets

Let x, x ∈ X and y, y ∈ Y . Then define tolerance classes for each fuzzy set. x/∼= = x | f 1(x) − f 1(x ) ≤ ε , f 1,ε y/∼= = y | f 2(y) − f 2(y ) ≤ ε . f 2,ε

Example 16. Fuzzy Weakly Near Sets Let X, { f 1}, Y, { f 2} denote perceptual systems. Let ε = 0.1. Then ∼ = f 1,0.1 defines a partition of the fuzzy set (X, f 1) and ∼ defines a partition of the fuzzy = f 2,0.1 set (Y, f 1). For example, the partition of X is defined by x/∼=

f 1,ε

= {x | f 1(x) − f 1(x ) ≤ 0.1}, x, x ∈ A in Fig. 10.


23

In addition, the two fuzzy sets are weakly near each other, since the objects in x/∼= and y/∼= have matching descriptions for each subinterval of the domains f 1,0.1

f 2,0.1

of the two functions, starting with A, A shown in Fig. 10. In effect, X B,ε Y.

Proposition 4. Fuzzy sets (X, φ ) , (Y, φ ) are weakly near sets if, and only if there exists at least one tolerance class x/∼= in (X , φ ) and y/∼= in (Y, φ ) such that φ ,ε φ ,ε x/∼= φ ,ε y/∼= . φ ,ε

φ ,ε

7 Rough Near Sets The germ of the idea for near sets first appeared within a poem by Zdisław Pawlak and this author in a poem entitled Near To written in 2002 and later published in English and Polish [27, 14]. In later years, the foundations for near sets grew out of a rough set approach to classifying images [28, 21, 4, 22]. It is fairly easy to show that every rough set is also a near set. This section briefly presents some fundamental notions in rough set theory resulting from the seminal work by Zdisław Pawlak during the early 1980s [13] and elaborated in [17, 16, 15]. An overview of the mathematical foundations of rough sets is given by Lech Polkowski in [31]. Let O, F denote a perceptual system containing a set of perceptual objects O and a set of functions F representing features of the objects in O. Further, let O∼B denote the set of all classes in the partition of O defined by ∼B for B ⊆ F. Recall that x/∼B denotes an equivalence class relative x ∈ O. For X ⊆ O, B ⊆ F, a sample perceptual granule X can be approximated with a B-lower B∗ X and B-upper approximation B ∗ X defined by B∗ X =

[x]B ,

x:[x]B ⊆X

B∗ X =

[x]B .

x:[x]B ∩X=0/

Let BNDB (X ) = B ∗ X − B∗ X denote the approximation boundary. A set X is a rough set in the case where the boundary BNDB (X ) is not empty, i.e., B ∗ X −B∗ X = 0. / That is, whenever B∗ X is a proper subset of B∗ X , i.e., the sample X has been classified imperfectly and X is considered a rough set. Notice, from Def. 6, B∗ X B X , and

B ∗ X B X ,

since the classes in an approximation of X contain objects with descriptions that match the description of at least one object in X. Hence, the pairs B∗ X , X and B ∗ X, X are examples of near sets. In general, Proposition 5. (Peters [20]) The pairs B∗ X , X and B∗ X , X are near sets. Proposition 6. (Peters [20]) Any equivalence class x/∼ , x/∼ > 2 is a near set. B

B

24

J.F. Peters

Conclusion This chapter includes a tutorial on near sets as well as a consideration of hybrid found by considering combination of near and fuzzy sets as well as near and rough sets. If nothing else, near sets provide a unifying influence in the study of pairs of sets that are either fuzzy or rough. Near sets have their origins in image analysis, especially if there is interest in comparing images. Fuzzy sets and rough sets are perhaps the most commonly used mathematical tools in a typical computational intelligence framework for applications such as control and classification. For this reason, there is considerable interest in finding connections between fuzzy sets and near sets as well as between rough sets and near sets. It has been shown that near sets are a generalization of rough sets. In fact, the formalization of near sets began in 2006 with a consideration of approximation spaces and the importance of the boundary of an approximation of a set in distinguishing between near sets and rough sets. It is a straightforward task to show that every rough set is a near set but not every near set is rough set. This is an important observation, since the population of non-rough sets appears to be considerably larger than the population of rough sets, if one considers the fact that every class with 2 or more objects is a near set. Near sets have proven to be useful machine learning (especially, biologically-inspired adaptive learning) and in image analysis.

References 1. Grimaldi, D., Engel, M.: Evolution of the Insects. Cambridge University Press, Cambridge (2005) 2. Gupta, S., Patnaik, K.: Enhancing performance of face recognition system by using near set approach for selecting facial features. Journal of Theoretical and Applied Information Technology 4(5), 433–441 (2008), http://www.jatit.org/volumes/fourth_volume_5_2008.php 3. Hassanien, A., Abraham, A., Peters, J., Schaefer, G.: Rough sets and near sets in medical imaging: A review. IEEE Trans. on Information Technology in Biomedicine (submitted) (2008) 4. Henry, C., Peters, J.: Image pattern recognition using approximation spaces and near sets. In: An, A., Stefanowski, J., Ramanna, S., Butz, C.J., Pedrycz, W., Wang, G. (eds.) RSFDGrC 2007. LNCS (LNAI), vol. 4482, pp. 475–482. Springer, Heidelberg (2007) 5. Henry, C., Peters, J.: Near set image segmentation quality index. In: GEOBIA 2008 Pixels, Objects, Intelligence. GEOgraphic Object Based Image Analysis for the 21st Century, pp. 284–289. University of Calgary, Alberta (2008), http://www.ucalgary.ca/geobia/Publishing 6. Henry, C., Peters, J.: Perception based image classification. IEEE Transactions on Systems, Man, and Cybernetics–Part C: Applications and Reviews (submitted) (2008) 7. Iturralde-Vinent, M., MacPhee, R.: Age and paleogeographical origin of dominican amber. Science 273, 1850–1852 (1996) 8. Jähne, B.: Digital Image Processing, 6th edn. Springer, Heidelberg (2005) 9. Murray, J., Bradley, H., Craigie, W., Onions, C.: The Oxford English Dictionary. Oxford University Press, Oxford (1933) 10. Orłowska, E. (ed.): Incomplete Information: Rough Set Analysis. Studies in Fuzziness and Soft Computing, vol. 13. Physica-Verlag, Heidelberg (1998)


25

11. Orłowska, E., Pawlak, Z.: Representation of nondeterministic information. Theoretical Computer Science 29, 27–39 (1984) 12. Pavel, M.: Fundamentals of Pattern Recognition, 2nd edn. Marcel Dekker, Inc., N.Y. (1993) 13. Pawlak, Z.: Classification of objects by means of attributes. Polish Academy of Sciences 429 (1981) 14. Pawlak, Z., Peters, J.: Jak blisko. Systemy Wspomagania Decyzji I, 57 (2007) 15. Pawlak, Z., Skowron, A.: Rough sets and boolean reasoning. Information Sciences 177, 41–73 (2007) 16. Pawlak, Z., Skowron, A.: Rough sets: Some extensions. Information Sciences 177, 28– 40 (2007) 17. Pawlak, Z., Skowron, A.: Rudiments of rough sets. Information Sciences 177, 3–27 (2007) 18. Pedrycz, W., Gomide, F.: An Introduction to Fuzzy Sets. Analysis and Design. MIT Press, Cambridge (1998) 19. Peters, J.: Classification of objects by means of features. In: Proc. IEEE Symposium Series on Foundations of Computational Intelligence (IEEE SCCI 2007), Honolulu, Hawaii, pp. 1–8 (2007) 20. Peters, J.: Near sets. General theory about nearness of objects. Applied Mathematical Sciences 1(53), 2029–2609 (2007) 21. Peters, J.: Near sets, special theory about nearness of objects. Fundamenta Informaticae 75(1-4), 407–433 (2007) 22. Peters, J.: Near sets. toward approximation space-based object recognition. In: Yao, J., ´ ¸ zak, D. (eds.) RSKT 2007. Lingras, P., Wu, W.-Z., Szczuka, M.S., Cercone, N.J., Sle LNCS (LNAI), vol. 4481, pp. 22–33. Springer, Heidelberg (2007) 23. Peters, J.: Classification of perceptual objects by means of features. Int. J. of Info. Technology & Intelligent Computing 3(2), 1–35 (2008) 24. Peters, J.: Discovery of perceputally near information granules. In: Yao, J. (ed.) Novel Developments in Granular Computing: Applications of Advanced Human Reasoning and Soft Computation. Information Science Reference, Hersey, N.Y., U.S.A. (to appear) (2008) 25. Peters, J., Ramanna, S.S.: Feature selection: A near set approach. In: ECML & PKDD Workshop on Mining Complex Data, Warsaw (2007) 26. Peters, J., Shahfar, S., Ramanna, S., Szturm, T.: Biologically-inspired adaptive learning: A near set approach. In: Frontiers in the Convergence of Bioscience and Information Technologies, Korea (2007) 27. Peters, J., Skowron, A.: Zdzisław pawlak: Life and work. Transactions on Rough Sets V, 1–24 (2006) 28. Peters, J., Skowron, A., Stepaniuk, J.: Nearness in approximation spaces. In: Proc. Concurrency, Specification and Programming (CS&P 2006), Humboldt Universität, pp. 435– 445 (2006) 29. Peters, J., Skowron, A., Stepaniuk, J.: Nearness of objects: Extension of approximation space model. Fundamenta Informaticae 79(3-4), 497–512 (2007) 30. Peters, J., Wasilewski, P.: Foundations of near sets. Information Sciences (submitted) (2008) 31. Polkowski, L.: Rough Sets. Mathematical Foundations. Springer, Heidelberg (2002) 32. Skowron, A., Stepaniuk, J.: Tolerance approximation spaces. Fundamenta Informaticae 27, 245–253 (1996) 33. Zadeh, L.: Fuzzy sets. Information and Control 8, 338–353 (1965)

Fuzzy without Fuzzy: Why Fuzzy-Related Aggregation Techniques Are Often Better Even in Situations without True Fuzziness Hung T. Nguyen1 , Vladik Kreinovich2 , and Fran¸cois Modave2 , and Martine Ceberio2 1

2

Department of Mathematical Sciences, New Mexico State University, Las Cruces, NM 88003, USA [email protected] Department of Computer Science, University of Texas at El Paso, El Paso, TX 79968, USA [email protected], [email protected], [email protected]

Summary. Fuzzy techniques have been originally invented as a methodology that transforms the knowledge of experts formulated in terms of natural language into a precise computer-implementable form. There are many successful applications of this methodology to situations in which expert knowledge exist, the most well known is an application to fuzzy control. In some cases, fuzzy methodology is applied even when no expert knowledge exists: instead of trying to approximate the unknown control function by splines, polynomials, or by any other traditional approximation technique, researchers try to approximate it by guessing and tuning the expert rules. Surprisingly, this approximation often works fine, especially in such application areas as control and multi-criteria decision making. In this chapter, we give a mathematical explanation for this phenomenon.

1 Introduction Fuzzy techniques: a brief reminder. Fuzzy techniques have been originally invented as a methodology that transforms the knowledge of experts formulated in terms of natural language into a precise computer-implementable form. There are many successful applications of this methodology to situations in which expert knowledge exist, the most well known is an application to fuzzy control; see, e.g., [6, 8, 18]. Universal approximation results. A guarantee of success comes from the fact that fuzzy systems are universal approximators in the sense that for every continuous function f (x1 , . . . , xn ) and for every ε > 0, there exists a set of rules for which the corresponding input-output function is ε-close to f ; see, e.g., [1, 8, 9, 11, 12, 16, 18, 19, 22, 23, 25] and references therein. A.-E. Hassanien et al. (Eds.): Foundations of Comput. Intel. Vol. 2, SCI 202, pp. 27–51. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com

28

H.T. Nguyen et al.

Fuzzy methodology is sometimes successful without any fuzzy expert knowledge. In some cases, fuzzy methodology is applied even when no expert knowledge exists: instead of trying to approximate the unknown control function by splines, polynomials, or by any other traditional approximation technique, researchers try to approximate it by guessing and tuning the expert rules. Surprisingly, this approximation often works fine. Similarly, fuzzy-type aggregation functions like OWA or Choquet integrals often work better than quadratic functions in multi-criteria decision making. What we plan to do. In this chapter, we give a mathematical explanation for these phenomena, and we show that approximation by using fuzzy methodology is indeed (in some reasonable sense) the best. Comment. In this chapter, we build upon our preliminary results published in [13, 15, 17].

2 Use of Fuzzy Techniques in Non-fuzzy Control: A Justification In many practical applications, data processing speed is important. We have mentioned that one of the main applications of fuzzy methodology is to intelligent control. In applications to automatic control, the computer must constantly compute the current values of control. The value of the control depends on the state of the controlled object (called plant in control theory). So, to get a high quality control, we must measure as many characteristics x1 , . . . , xn of the current state as we can. The more characteristics we measure, the more numbers we have to process, so, the more computation steps we must perform. The results of these computations must be ready in no time, before we start the next round of measurements. So, automatic control, especially high-quality automatic control, is a real-time computation problem with a serious time pressure. Parallel computing is an answer. A natural way to increase the speed of the computations is to perform computations in parallel on several processors. To make the computations really fast, we must divide the algorithm into parallelizable steps, each of which requires a small amount of time. What are these steps? The fewer variables, the faster. As we have already mentioned, the main reason why control algorithms are computationally complicated is that we must process many inputs. For example, controlling a car is easier than controlling a plane, because the plane (as a 3-D object) has more characteristics to take care of, more characteristics to measure and hence, more characteristics to process. Controlling a space shuttle, especially during the lift-off and landing, is even a more complicated task, usually performed by several groups of people who control the trajectory, temperature, rotation, etc. In short, the more

Fuzzy without Fuzzy: An Explanation

29

numbers we need to process, the more complicated the algorithm. Therefore, if we want to decompose our algorithm into fastest possible modules, we must make each module to process as few numbers as possible.

Functions of one variable are not sufficient Ideally, we should only use the modules that compute functions of one variable. However, if we only have functions of one variables (i.e., procedures with one input and one output), then, no matter how we combine them, we will always end up with functions of one variable. Since our ultimate goal is to compute the control function u = f(x1,...,xn) that depends on many variables x1,...,xn, we must therefore enable our processors to compute at least one function of two or more variables. What functions of two variables should we choose? Choosing functions of two or more variables. Inside the computer, each function is represented as a sequence of hardware implemented operations. The fastest functions are those that are computed by a single hardware operation. The basic hardware supported operations are: arithmetic operations a + b, a − b, a · b, a/b, and min(a, b)and max(a, b). Thetimerequired for each op eration, crudely speaking, corresponds to the number of bits operations that have to be performed: • • • •

•

Division is done by successive multiplication, comparison and subtraction (basically, in the same way as we do it manually), so, it is a much slower operation than −. Multiplication is implemented as a sequence of additions (again, basically in the same manner as we do it manually), so it is much slower than +. − and + are usually implemented in the same way. To add two n-bit binary numbers, we need n bit additions, and also potentially, n bit additions for carries. Totally, we need about 2n bit operations. min of two n-bit binary numbers can be done in n binary operations: we compare the bits from the highest to the lowest, and as soon as they differ, the number that has 0 as opposed to 1 is the desired minimum: e.g., the minimum of 0.10101 and 0.10011 is 0.10011, because in the third bit, this number has 0 as opposed to 1. Similarly, max is an n-bit operation.

So, the fastest possible functions of two variables are min and max. Similarly fast is computing the minimum and maximum of several (more than two) real numbers. Therefore, we will choose these functions for our control-oriented computer. Summarizing the above-given analysis, we can conclude that our computer will contain modules of two type: • •

modules that compute functions of one variable; modules that compute min and max of two or several numbers.

How to combine these modules? We want to combine these modules in such a way that the resulting computations are as fast as possible. The time that is

30

H.T. Nguyen et al.

required for an algorithm is crudely proportional to the number of sequential steps that it takes. We can describe this number of steps in clear geometric terms: • • • •

at the beginning, the input numbers are processed by some processors; these processors form the first layer of computations; the results of this processing may then go into different processors, that form the second layer; the results of the second layer of processing go into the third layer, etc.

In these terms, the fewer layers the computer has, the faster it is. So, we would like to combine the processors into the smallest possible number of layers. Now, we are ready for the formal definitions. Definition and main result. Let us first give an inductive definition of what it means for a function to be computable by a k-layer computer. Definition 1 • •

We say that a function f (x1 , . . . , xn ) is computable by a 1-layer computer if either n = 1, or the function f coincides with min or with max. Let k ≥ 1 be an integer. We say that a function f (x1 , . . . , xn ) is com putable by a (k+1)-layer computer if one of the following three statements is true: • f = g(h(x1 , . . . , xn )), where g is a function of one variable, and h(x1 , . . . , xn ) is computable by a k-layer computer; • f = min(g1 (x1 , . . . , xn ), . . . , gm (x1 , . . . , xn )), where all functions gi are computed by a k-layer computer; • f = max(g1 (x1 , . . . , xn ), . . . , gm (x1 , . . . , xn )), where all functions gi are computed by a k-layer computer.

Comment. A computer is a finite-precision machine, so, the results of the computations are never absolutely precise. Also, a computer is limited in the size of its numbers. So, we can only compute a function approximately, and only on a limited range. Therefore, when we say that we can compute an arbitrary function, we simply mean that for an arbitrary range T ,for an arbitrary continuous function f :[−T, T ]n → R, and for an arbitrary accuracy ε> 0, we can compute a function that is ε-close to f on the given range. In this sense, we will show that not every function can be computed on a 2-layer computer, but that 3 layers are already sufficient. Proposition 1. There exist real numbers T and ε > 0, and a continuous function f : [−T, T ]n → R such that no function ε-close to f on [−T, T ]n can be computed on a 2-layer computer.


31

Comment. To make the text more readable, we present both proofs in the last section. However, we will make one comment here. The function that will be proved to be not computable on a 2-layer computer is not exotic at all: it is f (x1 , x2 ) = x1 + x2 on the domain [−1, 1]2, and Proposition 1 holds for ε = 0.4. Theorem 1. For every real numbers T and ε > 0, and for every continuous function f : [−T, T ]n → R, there exists a function f that is ε-close to f on [−T, T ]n and that is computable on a 3-layer computer. Comment. In other words, functions computed by a 3-layer computer are universal approximators. Relation to fuzzy control. As we will see from the proof, the approximating function f is of the type max(A1 , . . . , Am ), where Aj = min(fj1 (x1 ), . . . , fjn (xn ). These functions correspond the so-called fuzzy control [6, 8, 18]: Indeed, let us define U= max |fji (xi )|, i,j,xi ∈[−T,T ]

and μji (xi ) =

fji (xi ) − (−U ) . U − (−U )

Let us now assume that the rules base that describes the expert recommendations for control consists of exactly two rules: • •

“if one of the conditions Cj is true, then u = U ”; “else, u = −U ”,

where each condition Cj means that the following n conditions are satisfied: • • • •

x1 satisfies the property Cj1 (described by a membership function μj1 (x1 )); x2 satisfies the property Cj2 (described by a membership function μj2 (x2 )); ... xn satisfies the property Cjn (described by a membership function μjn (xn )). In logical terms, the condition C for u = U has the form (C11 & . . . &C1n ) ∨ . . . ∨ (Ck1 & . . . &Ckn ).

If we use min for &, and max for ∨ (these are the simplest choices in fuzzy control methodology), then the degree μC with which we believe in a condition C = C1 ∨ . . . ∨ Ck can be expressed as: μC = max[min(μ11 (x1 ), . . . , μ1n ), . . . , min(μk1 , . . . , μkn )]. Correspondingly, the degree of belief in a condition for u = −U is 1 − μC . According to fuzzy control methodology, we must use a defuzzification to determine the actual control, which in this case leads to the choice of

32

H.T. Nguyen et al.

u=

U · μC + (−U ) · (1 − μC ) . μC + (1 − μC )

Because of our choice of μji , one can easily see that this expression coincides exactly with the function max(A1 , . . . , Am ), where Aj = min(fj1 (x1 ), . . . , fjn (xn )). So, we get exactly the expressions that stem from the fuzzy control methodology. Conclusion. Since our 3-layer expression describes the fastest possible computation tool, we can conclude that for control problems, the fastest possible universal computation scheme corresponds to using fuzzy methodology. This result explains why fuzzy methodology is sometimes used (and used successfully) without any expert knowledge being present, as an extrapolation tool for the (unknown) function. Comment. We have considered digital parallel computers. If we use analog processors instead, then min and max stop being the simplest functions. Instead, the sum is the simplest: if we just join the two wires together, then the resulting current is equal to the sum of the two input currents. In this case, if we use a sum (and more general, linear combination) instead of min and max, 3-layer computers are also universal approximators; the corresponding computers correspond to neural networks [10]. Proof of Proposition 1 0◦ . Let us proof (by reduction to a contradiction) that if a function f(x1 , x2 ) is 0.4−close to f (x1 , x2 ) = x1 + x2 on [−1, 1]2 , then f cannot be computed on a 2-layer computer. Indeed, suppose that it is. Then, according to the Definition, the function f(x1 , x2 ) is of one of the following three forms: • • •

g(h(x1 , x2 )), where h is computable on a 1-layer computer; min(g1 (x1 , x2 ), . . . , gm (x1 , x2 )), where all the functions gi are computable on a 1-layer computer; max(g1 (x1 , x2 ), . . . , gm (x1 , x2 )), where all the functions gi are computable on a 1-layer computer.

Let us show case-by-case that all these three cases are impossible. 1◦ . In the first case, f(x1 , x2 ) = g(h(x1 , x2 )), where h is computable on a 1-layer computer. Be definition, this means that h is either a function of one variable, or min, or max. Let us consider all these three sub-cases. 1.1◦ . If f(x1 , x2 ) = g(h(x1 )), then the function f depends only on x1 . In particular, f(0, −1) = f(0, 1). (1)


33

But since f is ε-close to f (x1 + x2 ) = x1 + x2 , we get f(0, −1) ≤ f (0, −1) + ε = −1 + 0.4 = −0.6, and

f(0, 1) ≥ f (0, 1) − ε = 1 − 0.4 > 0.6 > −0.6.

−1) ≤ −0.6 < f(0, 1), hence, f(0, −1) = f(0, 1), which contradicts So, f(0, to (1). So, this sub-case is impossible. Similarly, it is impossible to have h depending only on x2 . 1.2◦ . Let us consider the sub-case when f(x1 , x2 ) = g(min(x1 , x2 )). In this sub-case, f(−1, −1) = g(min(−1, −1)) = g(−1) = g(min(−1, 1)) = f(−1, 1), and But and

f(−1, −1) = f(−1, 1).

(2)

f(−1, −1) ≤ f (−1, −1) + ε = −2 + 0.4 = −1.6, f(−1, 1) ≥ f (−1, 1) − ε = 0 − 0.4 = −0.4 > −1.6,

so, the equality (2) is also impossible. 1.3◦ . Let us now consider the sub-case f(x1 , x2 ) = g(max(x1 , x2 )). In this sub-case, f(−1, 1) = g(max(−1, 1)) = g(1) = g(max(1, 1)) = f(1, 1), and But and

f(−1, 1) = f(1, 1).

(3)

f(−1, 1) ≤ f (−1, 1) + ε = 0 + 0.4 = 0.4, f(1, 1) ≥ f (1, 1) − ε = 2 − 0.4 = 1.6 > 0.4,

so, the equality (3) is also impossible. 2◦ . In the second case, f(x1 , x2 ) = min(g1 (x1 , x2 ), . . . , gm (x1 , x2 )), where all the functions gi are computable on a 1-layer computer. For this case, the impossibility follows from the following sequence of steps: 2.1◦ . If one of the functions gi is of the type min(x1 , x2 ), then we can rewrite min(g1 , . . . , gi−1 , min(x1 , x2 ), gi+1 , . . . , gm )

34

as

H.T. Nguyen et al.

(1)

(2)

min(g1 , . . . , gi−1 , gi , gi , gi+1 , . . . , gm ),

where g (i) (x1 , x2 ) = xi is a function that is clearly computable on a 1-layer computer. After we make such transformations, we get an expression for f that only contains max and functions of one variable. 2.2◦ . Let us show that this expression cannot contain max. Indeed, if it does, then f(x1 , x2 ) = min(. . . , max(x1 , x2 )) ≤ max(x1 , x2 ). In particular, f(1, 1) ≤ max(1, 1) = 1. But we must have f(1, 1) ≥ f (1, 1) − ε = 2 − 0.4 = 1.6 > 1. The contradiction shows that max cannot be one of the functions gi . 2.3◦ . So, each function gi depends only on one variable. If all of them depend on one and the same variable, say, x1 , then the entire function f depends only on one variable, and we have already proved (in the proof of the first case) that it is impossible. So, some functions gi depend on x1 , and some of the functions gi depend on x2 . Let us denote by h1 (x1 ) the minimum of all functions gi that depend on x1 , and by h2 (x2 ), the minimum of all the functions gi that depend on x2 . Then, we can represent f as f(x1 , x2 ) = min(h1 (x1 ), h2 (x2 )). 2.4◦ . To get a contradiction, let us first take x1 = 1 and x2 = 1. Then, f(1, 1) = min(h1 (1), h2 (1)) ≥ f (1, 1) − ε = 2 − 0.4 = 1.6. Since the minimum of the two numbers is ≥ 1.6, we can conclude that each of them is ≥ 1.6, i.e., that h1 (1) ≥ 1.6 and h2 (1) ≥ 1.6. For x1 = 1 and x2 = −1, we have f(1, −1) = min(h1 (1), h2 (−1)) ≤ f (1, −1) + ε = 0.4. Since h1 (1) ≥ 1.6, we conclude that f(1, −1) = h2 (−1). From f(1, −1) ≥ f (1, −1) − ε = −0.4,

(4)

we can now conclude that h2 (−1) ≥ −0.4. Similarly, one can prove that h1 (−1) ≥ −0.4. Hence, f(−1, −1) = min(h1 (−1), h2 (−1)) ≥ −0.4. But

f(−1, −1) ≤ f (−1, −1) + ε = −2 + 0.4 = −1.6 < −0.4 :

a contradiction with (4). The contradiction shows that the second case is also impossible.


35

3◦ . In the third case, f(x1 , x2 ) = max(g1 (x1 , x2 ), . . . , gm (x1 , x2 )), where all the functions gi are computable on a 1-layer computer. For this case, the impossibility (similarly to the second case) follows from the following sequence of steps: 3.1◦ . If one of the functions gi is of the type max(x1 , x2 ), then we can rewrite max(g1 , . . . , gi−1 , max(x1 , x2 ), gi+1 , . . . , gm ) as

(1)

(2)

max(g1 , . . . , gi−1 , gi , gi , gi+1 , . . . , gm ),

where g (i) (x1 , x2 ) = xi is a function that is clearly computable on a 1-layer computer. After we make such transformations, we get an expression for f that only contains min and functions of one variable. 3.2◦ . Let us show that this expression cannot contain min. Indeed, if it does, then f(x1 , x2 ) = max(. . . , min(x1 , x2 )) ≥ min(x1 , x2 ). In particular,

f(−1, −1) ≥ min(−1, −1) = −1.

But we must have f(−1, −1) ≤ f (−1, −1) + ε = −2 + 0.4 = −1.6 < −1. The contradiction shows that min cannot be one of the functions gi . 3.3◦ . So, each function gi depends only on one variable. If all of them depend on one and the same variable, say, x1 , then the entire function f depends only on one variable, and we have already proved (in the proof of the first case) that it is impossible. So, some functions gi depend on x1 , and some of the functions gi depend on x2 . Let us denote by h1 (x1 ) the maximum of all functions gi that depend on x1 , and by h2 (x2 ), the maximum of all the functions gi that depend on x2 . Then, we can represent f as f(x1 , x2 ) = max(h1 (x1 ), h2 (x2 )). 3.4◦ . To get a contradiction, let us first take x1 = −1 and x2 = −1. Then, f(−1, −1) = max(h1 (−1), h2 (−1)) ≤ f (−1, −1) + ε = −2 + 0.4 = −1.6. Since the maximum of the two numbers is ≤ −1.6, we can conclude that each of them is ≤ −1.6, i.e., that h1 (−1) ≤ −1.6 and h2 (−1) ≤ −1.6. For x1 = 1 and x2 = −1, we have f(1, −1) = max(h1 (1), h2 (−1)) ≥ f (1, −1) − ε = −0.4.

36

H.T. Nguyen et al.

Since h2 (−1) ≤ −1.6, we conclude that f(1, −1) = h1 (1). From −1) ≤ f (1, −1) + ε = 0.4, f(1, we can now conclude that h1 (1) ≤ 0.4. Similarly, one can prove that h2 (1) ≤ 0.4. Hence, f(1, 1) = max(h1 (1), h2 (1)) ≥ 0.4. (5) But

f(1, 1) ≥ f (1, 1) − ε = 2 − 0.4 = 1.6 > 0.4,

which contradicts to (5). The contradiction shows that the third case is also impossible. 4◦ . In all there cases, we have shown that the assumption that f can be computed on a 2-layer computer leads to a contradiction. So, f cannot be thus computed. Q.E.D. Proof of Theorem 1. Since the function f is continuous, there exists a δ > 0 such that if |xi − yi | ≤ δ, then |f (x1 , . . . , xn ) − f (y1 , . . . , yn )| ≤ ε. Let us mark the grid points on the grid of size δ, i.e., all the points for which each coordinate x1 , . . . , xn has the form qi · δ for integer qi (i.e., we mark the points with coordinates 0, ±δ, ±2δ, . . . , ±T ). On each coordinate, we thus mark ≈ 2T /δ points. So, totally, we mark ≈ (2T /δ)n grid points. Let us denote the total number of grid points by k, and the points themselves by Pj = (xj1 , . . . , xjn ), 1 ≤ j ≤ k. By mf , let us denote the minimum of f : mf =

min

x1 ∈[−T,T ],...,xn ∈[−T,T ]

f (x1 , . . . , xn ).

For each grid point Pj , we will form piece-wise linear functions fji (xi ) as follows: • • •

if |xi − xji | ≤ 0.6 · δ, then fji (xi ) = f (Pj )(≥ mf ); if |xi − xji | ≥ 0.7 · δ, then fji (xi ) = mf ; if 0.6 · δ ≤ |xi − xji | ≤ 0.7 · δ, then fji (xi ) = mf + (f (Pj ) − mf ) ·

0.7 · δ − |xi − xji | . 0.7 · δ − 0.6 · δ

Let us show that for these functions fji , the function 1 , . . . , xn ) = max(A1 , . . . , Am ), f(x where Aj = min(fj1 (x1 ), . . . , fjn (xn )), is ε-close to f .


37

To prove that, we will prove the following two inequalities: • •

For all x1 , . . . , xn , we have f(x1 , . . . , xn ) ≥ f (x1 , . . . , xn ) − ε. For all x1 , . . . , xn , we have f(x1 , . . . , xn ) ≤ f (x1 , . . . , xn ) + ε.

Let us first prove the first inequality. Assume that we have a point (x1 , . . . , xn ). For every i = 1, . . . , n, by qi , we will denote the integer that is the closest to xi /δ. Then, |xi − qi · δ| ≤ 0.5 · δ. These values qi determine a grid point Pj = (xj1 , . . . , xjn ) with coordinates xji = qi · δ. For this j, and for every i, |xi − xji | ≤ 0.5 · δ < 0.6 · δ, therefore, by definition of fji , we have fji (xi ) = f (Pj ). Hence, Aj = min(fj1 (x1 ), . . . , fjn (xn )) = min(f (Pj ), . . . , f (Pj )) = f (Pj ). Therefore, f(x1 , .., xn ) = max(A1 , . . . , Am ) ≥ Aj = f (Pj ). But since |xji − xi | ≤ 0.5 · δ < δ, by the choice of δ, we have |f (x1 , . . . , xn ) − f (Pj )| ≤ ε. Therefore, f (Pj ) ≥ f (x1 , . . . , xn ) − ε, and hence, f(x1 , . . . , xn ) ≥ f (Pj ) ≥ f (x1 , . . . , xn ) − ε. Let us now prove the second inequality. According to our definition of fji , the value of fji (xi ) is always in between mf and Pj , and this value is different from mf only for the grid points Pj for which |xji − xi | ≤ 0.7 · δ. The value Aj = min(fj1 (x1 ), . . . , fjn (xn )) is thus different from m only if all the values fji (xi ) are different from m, i.e., when |xji − xi | ≤ 0.7 · δ for all i. For this grid point, |xji − xi | ≤ 0.7 · δ < δ; therefore, |f (Pj ) − f (x1 , . . . , xn )| ≤ ε and hence, f (Pj ) ≤ f (x1 , . . . , xn ) + ε. By definition of fji , we have fji (xi ) ≤ f (Pj ). Since this is true for all i, we have Aj = min(fj1 (x1 ), . . . , fjn (xn )) ≤ f (Pj ) ≤ f (x1 , . . . , xn ) + ε. For all other grid points Pj , we have Aj (x1 , . . . , xn ) = mf

38

H.T. Nguyen et al.

for a given (x1 , . . . , xn ). Since mf has been defined as the minimum of f , we have Aj = mf ≤ f (x1 , . . . , xn ) < f (x1 , . . . , xn ) + ε. So, for all grid points, we have Aj ≤ f (x1 , . . . , xn ) + ε, and therefore, f(x1 , . . . , xn ) = max(A1 , . . . , Am ) ≤ f (x1 , . . . , xn ) + ε. The second inequality is also proven. So, both inequalities are true, and hence, f is ε-close to f . The theorem is proven.

3 Fuzzy-Type Aggregation in Multi-criteria Decision Making: A Problem A similar situation occurs in multi-criterion decision making. To describe the problem, let us briefly explain what multi-criteria decision making is about. One of the main purposes of Artificial Intelligence in general is to incorporate a large part of human intelligent reasoning and decision-making into a computer-based systems, so that the resulting intelligent computerbased systems help users in making rational decisions. In particular, to help a user make a decision among a large number of alternatives, an intelligent decision-making systems should select a small number of these alternatives – alternatives which are of the most potential interest to the user. For example, with so many possible houses on the market, it is not realistically possible to have a potential buyer inspect all the house sold in a given city. Instead, a good realtor tries to find out the buyer’s preferences and only show him or her houses that more or less fit these preferences. It would be great to have an automated system for making similar pre-selections. To be able to make this selection, we must elicit the information about the user preferences. In principle, we can get a full picture of the user preferences by asking the user to compare and/or rank all possible alternatives. Such a complete description of user preferences may be sometimes useful, but in decision making applications, such an extensive question-asking defeats the whole purpose of intelligent decision-making systems – to avoid requiring that the the user make a large number of comparisons. The existing approach to this problem is called multi-criteria decision making (MCDM). The main idea behind this approach is that each alternative is characterized by the values of different parameters. For example, the buyer’s selection of a house depends on the house’s size, on its age, on its geographical location, on the number of bedrooms and bathrooms, etc. The idea is to elicit


39

preferences corresponding to each of these parameters, and then to combine these single-parameter preferences into a reasonable model for describing the user’s choice. In the standard decision making theory, preferences are characterized by assigning, to each alternative, a numerical value called its utility. In these terms, the multi-criteria decision making approach means that we try to combine single-variable utility values u1 (x1 ), . . . , un (xn ) characterizing the user’s preferences over individual parameters x1 , . . . , xn into a utility value u(x1 , . . . , xn ) that characterizes the utility of an alternative described by the values (x1 , . . . , xn ). In the first approximation, it makes sense simply to add the individual utility values with appropriate weights, i.e., to consider linear aggregation u(x1 , . . . , xn ) = w1 · u1 (x1 ) + . . . + wn · un (xn ). In many practical situations, linear aggregation works well, but in some cases, it leads to counterintuitive conclusions. For example, when selecting a house, a user can assign certain weights to all the parameters characterizing different houses, but the user may also has absolute limitations: e.g., a user with kids may want a house with at least two bedrooms, and no advantages in location and price would entice her to buy a one-bedroom house. To describe such reasonable preferences, we must therefore go beyond linear aggregation functions. From the purely mathematical viewpoint, the inadequacy of a linear model is a particular example of a very typical situation. Often, when we describe the actual dependence between the quantities in physics, chemistry, engineering, etc., a linear expressions y = c0 + c1 · x1 + . . . + cn · xn is a very good first approximation (at least locally), but to get a more accurate approximations, we must take non-linearity into account. In mathematical applications to physics, engineering, etc., there is a standard way to take non-linearity into account: if a linear approximation is not accurate enough, a natural idea is n n n to use a quadratic approximation y ≈ a0 + ci · xi + cij · xi · xj ; if i=1

i=1 j=1

the quadratic approximation is not sufficient accurate, we can use a cubic approximation, etc.; see, e.g., [4]. At first glance, it seems reasonable to apply a similar idea to multi-criteria decision making and consider quadratic aggregation functions def

u = u(x1 , . . . , xn ) = u0 +

n i=1

wi · ui (xi ) +

n n

wij · ui (xi ) · uj (xj ).

i=1 j=1

Surprisingly, in contrast to physics and engineering applications, quadratic approximation do not work as well as approximations based on the use of piece-wise linear functions, such as the OWA operation u = w1 ·u(1) +. . .+wn · u(n) , where u(1) = max(u1 (x1 ), . . . , un (xn )) is the largest of n utility values ui (xi ), u(2) is the second largest, . . . , and u(n) = min(u1 (x1 ), . . . , un (xn )) is the smallest of n utility values; see, e.g., [24].

40

H.T. Nguyen et al.

In our own research, we have applied OWA and we have also applied similar piece-wise linear operations (based on the so-called Choquet integral [5]), and we also got good results – better than quadratic approximations; see, e.g., [2] and references therein. Similar results have been obtained by others. For quite some time, why piece-wise approximations are better than quadratic ones remains a mystery to us – and to many other researchers whom we asked this question. Now, we finally have an answer to this question – and this answer is presented in the current chapter. Thus, the chapter provides a new justification of the use of piece-wise aggregation operations in multi-criteria decision making – a justification that explains why these aggregation operations are better than the (seemingly more natural) quadratic ones.

4 Standard Decision Making Theory: A Brief Reminder To explain our answer to the long-standing puzzle, we need to recall the properties of the utility functions. The needed properties of utility functions are described in this section. Readers who are already well familiar with the standard decision making theory (and with the corresponding properties of utility functions) can skip this section and proceed directly to the next one. To be able to describe decisions, we must have a numerical scale for describing preferences. The traditional decision making theory (see, e.g., [7, 14, 20]) starts with an observation that such a scale can be naturally obtained by using probabilities. Specifically, to design this scale, we select two alternatives: • •

a very negative alternative A0 ; e.g., an alternative in which the decision maker loses all his money (and/or loses his health as well), and a very positive alternative A1 ; e.g., an alternative in which the decision maker wins several million dollars.

Based on these two alternatives, we can, for every value p ∈ [0, 1], consider a randomized alternative L(p) in which we get A1 with probability p and A0 with probability 1 − p. (It should be mentioned that in the standard decision making theory, randomized alternatives like L(p) are also (somewhat misleadingly) called lotteries. This name comes from the fact that a lottery is one of the few real-life examples of randomized outcomes with known probabilities.) In the two extreme cases p = 0 and p = 1, the randomized alternative L(p) turns into one of the original alternatives: when p = 1, we get the favorable alternative A1 (with probability 1), and when p = 0, we get the unfavorable alternative A0 . In general, the larger the probability p of the favorable alternative A1 , the more preferable is the corresponding randomized alternative L(p). Thus, the corresponding randomized alternatives (“lotteries”) L(p) form a continuous 1-D scale ranging from the very negative alternative A0 to the very positive alternative A1 .


41

So, it is reasonable to gauge the preference of an arbitrary alternative A by comparing it to different alternatives L(p) from this scale until we find A’s place on this scale, i.e., the value p ∈ [0, 1] for which, to this decision maker, the alternative A is equivalent to L(p): L(p) ∼ A. This value is called the utility u(A) of the alternative A in the standard decision making theory. In our definition, the numerical value of the utility depends on the selection of the alternatives A0 and A1 : e.g., A0 is the alternative whose utility is 0 and A1 is the alternative whose utility is 1. What if we use a different set of alternatives, e.g., A0 < A0 and A1 > A1 ? Let A be an arbitrary alternative between A0 and A1 , and let u(A) be its utility with respect to A0 and A1 . In other words, we assume that A is equivalent to the randomized alternative in which: • •

we have A1 with probability u(A), and we have A0 with probability 1 − p.

In the scale defined by the new alternatives A0 and A1 , let u (A0 ), u (A1 ), and u (A) denote the utilities of A0 , A1 , and A. This means, in particular: • •

that A0 is equivalent to the randomized alternative in which we get A1 with probability u (A0 ) and A0 with probability 1 − u (A0 ); and that A1 is equivalent to the randomized alternative in which we get A1 with probability u (A1 ) and A0 with probability 1 − u (A1 ).

Thus, the alternative A is equivalent to the compound randomized alternative, in which • •

first, we select A1 or A0 with probabilities u(A) and 1 − u(A), and then depending on the first selection, we select A1 with probability u (A1 ) or u (A0 ) – and A0 with the remaining probability.

As the result of this two-stage process, we get either A0 or A1 . The probability p of getting A1 in this two-stage process can be computed by using the formula of full probability p = u(A) · u (A1 ) + (1 − u(A)) · u (A0 ) = u(A) · (u (A1 ) − u (A0 )) + u (A0 ). So, the alternative A is equivalent to a randomized alternative in which we get A1 with probability p and A0 with the remaining probability 1 − p. By definition of utility, this means that the utility u (A) of the alternative A in the scale defined by A0 and A1 is equal to this value p: u (A) = u(A) · (u (A1 ) − u (A0 )) + u (A0 ). So, changing the scale means a linear re-scaling of the utility values: u(A) → u (A) = λ · u(A) + b for λ = u (A1 ) − u (A0 ) > 0 and b = u (A0 ).

42

H.T. Nguyen et al.

Vice versa, for every λ > 0 and b, one can find appropriate events A0 and for which the re-scaling has exactly these values λ and b. In other words, utility is defined modulo an arbitrary (increasing) linear transformation. The last important aspect of the standard decision making theory is its description of the results of different actions. Suppose that an action leads to alternatives a1 , . . . , am with probabilities p1 , . . . , pm . We can assume that we have already determined the utility ui = u(ai ) of each of the alternatives a1 , . . . , am . By definition of the utility, this means that for each i, the alternative ai is equivalent to the randomized alternative L(ui ) in which we get A1 with probability ui and A0 with probability 1 − ui . Thus, the results of the action are equivalent to the two-stage process in which, with the probability pi , we select a randomized alternative L(ui ). In this two-stage process, the results are either A1 or A0 . The probability p of getting A1 in this two-stage process can be computed by using the formula for full probability: p = p1 ·u1 +. . .+pm ·um . Thus, the action is equivalent to a randomized alternative in which we get A1 with probability p and A0 with the remaining probability 1 − p. By definition of utility, this means that the utility u of the action in question is equal to A1

u = p1 · u1 + . . . + pm · um . In statistics, the right-hand of this formula is known as the expected value. Thus, we can conclude that the utility of each action with different possible alternatives is equal to the expected value of the utility.

5 Why Quadratic Aggregation Operations Are Less Adequate Than OWA and Choquet Operations: An Explanation To adequately describe the decision maker’s preferences, we must be able, given an alternative characterized by n parameters x1 , . . . , xn , to describe the utility u(x1 , . . . , xn ) of this alternative. To get a perfect description of the user’s preference, we must elicit such a utility value for all possible combinations of parameters. As we have mentioned in the Introduction, for practical values n, it is not realistic to elicit that many utility values from a user. So, instead, we elicit the user’s preference over each of the parameters xi , and then aggregate the resulting utility values ui (xi ) into an approximation for def

u(x1 , . . . , xn ): u(x1 , . . . , xn ) ≈ f (u1 , . . . , un ), where ui = ui (xi ). We have also mentioned that in the first approximation, linear aggregan tion operations f (u1 , . . . , un ) = a0 + wi · ui work well, but to get a more i=1

adequate representation of the user’s preferences, we must go beyond linear functions. From the purely mathematical viewpoint, it may seem that quadratic functions f (u1 , . . . , un ) should provide a reasonable next approximation, but in practice, piece-wise linear aggregation operations such as OWA


43

(or Choquet integral) provide a much more adequate description of expert preferences. For example, for two parameters, the general OWA combination of two utility values has the form f (u1 , u2 ) = w1 · min(u1 , u2 ) + w2 · max(u1 , u2 ). Similarly, the general OWA combination of three utility values has the form f (u1 , u2 , u3 ) = w1 · min(u1 , u2 , u3 )+ w2 · max(min(u1 , u2 ), min(u1 , u3 ), min(u2 , u3 )) + w3 · max(u1 , u2 , u3 ). Let us show that this seemingly mysterious advantage of non-quadratic aggregation operations can be explained based on the main properties of the utility functions. Indeed, as we have mentioned in Section 2, the utility is defined modulo two types of transformations: changing a starting point u → u + b and changing a scale u → λ · u for some λ > 0. It is therefore reasonable to require that the aggregation operation should not depend on which “unit” (i.e., which extreme event A1 ) we use to describe utility. Let us describe this requirement in precise terms. In the original scale, • •

we start with utility values u1 , . . . , un ; to these values, we apply the aggregation operation f (u1 , . . . , un ) and get the resulting overall utility u = f (u1 , . . . , un ).

On the other hand, • •

we can express the same utility values in a new scale, as u1 = λ · u1 , . . . , un = λ · un ; then, we use the same aggregation function to combine the new utility values; as a result, we get the resulting overall utility u = f (u1 , . . . , un ).

Substituting the expressions ui = λ · ui into this formula, we conclude that u = f (λ · u1 , . . . , λ · un ). We require that the utility u = f (u1 , . . . , un ) = f (λ · u1 , . . . , λ · un ) reflect the same degree of preference as the utility u = f (u1 , . . . , un ) but in a different scale: u = λ · u, i.e., f (λ · u1 , . . . , λ · un ) = λ · f (u1 , . . . , un). It is worth mentioning that in mathematics, such functions are called homogeneous (of first degree). So, we arrive at the conclusion that an adequate aggregation operation should be homogeneous. This conclusion about the above mysterious fact. On the other hand, one can show that linear aggregation operations and piece-wise linear aggregation operations like OWA are scale-invariant.

44

H.T. Nguyen et al.

Let us start with a linear aggregation operation f (u1 , . . . , un ) = w1 · u1 + . . . + wn · un . For this operation, we get f (λ · u1 , . . . , λ · un ) = w1 · (λ · u1 ) + . . . + wn · (λ · un ) = λ · (w1 · u1 + . . . + wn · un ) = λ · f (u1 , . . . , un ). Let us now consider the OWA aggregation operation f (u1 , . . . , un ) = w1 · u(1) + . . . + wn · u(n) , where u(1) is the largest of n values u1 , . . . , un , u(2) is the second largest, etc. If we multiply all the utility values ui by the same constant λ > 0, their order does not change. In particular, this means that the same value u(1) which was the largest in the original scale is the largest in the new scale as well. Thus, its numerical value u(1) can be obtained by re-scaling u(1) : u(1) = λ · u(1) . Similarly, the same value u(2) which was the second largest in the original scale is the second largest in the new scale as well. Thus, its numerical value u(2) can be obtained by re-scaling u(2) : u(2) = λ · u(2) , etc. So, we have u(i) = λ · u(i) for all i. Thus, for the OWA aggregation operation, we have f (λ·u1 , . . . , λ·un ) = w1 ·u(1) +. . .+wn ·u(n) = w1 ·(λ·u(1) )+. . .+wn ·(λ·u(n) ) = λ · (w1 · u(1) + . . . + wn · u(n) ) = λ · f (u1 , . . . , un ). On the other hand, a generic quadratic operation is not homogeneous. Indeed, a general quadratic operation has the form f (u1 , . . . , un ) =

n

wi · ui +

i=1

n n

wij · ui · uj .

i=1 j=1

Here, f (λu1 , . . . , λun ) =

n

wi · (λ · ui ) +

i=1

λ·

n

n n

wij · (λ · ui ) · (λ · uj ) =

i=1 j=1

wi · ui + λ2 ·

i=1

n n

wij · ui · uj .

i=1 j=1

On the other hand, λ · f (u1 , . . . , un ) = λ ·

n i=1

wi · ui + λ ·

n n i=1 j=1

wij · ui · uj .


45

The linear terms in the expressions f (λu1 , . . . , λun ) and λ · f (u1 , . . . , un ) coincide, but the quadratic terms differ: the quadratic term in f (λu1 , . . . , λun ) differs from the quadratic term in λ · f (u1 , . . . , un ) by a factor of λ. Thus, the only possibility to satisfy the scale-invariance (homogeneity) requirement for all λ is to have these differing quadratic terms equal to 0, i.e., to have wij = 0 – but in this case the aggregation operation is linear. So, quadratic operations are indeed not homogeneous – which explains whey they are less adequate in describing user’s preferences than homogeneous operations like OWA or Choquet integral.

6 OWA and Choquet Operations Are, in Some Reasonable Sense, the Most General Ones: A New Result In the previous section, we explained the empirical fact that in multi-criteria decision making, OWA and Choquet operations lead to more adequate results than seemingly natural quadratic aggregation operations. The explanation is that, due to the known properties of the utility, it is reasonable to require that aggregation operation be scale-invariant (homogeneous); OWA and Choquet operations are scale-invariant but quadratic operations are not. However, in principle, OWA and Choquet operations are just a few examples of scale-invariant operations, so by itself, the above result does not explain why OWA and Choquet operations are so successful and not any other scale-invariant operation. In this section, we give such an explanation. This explanation is based on the fact that OWA and Choquet operations are compositions of linear functions, min, and max. In this section, we prove that, crudely speaking, every scale-invariant operation can be composed of linear functions and min and max operations. Definition 2. A function f (x1 , . . . , xn ) is called homogeneous if for every x1 , . . . , xn and for every λ > 0, we have f (λ·x1 , . . . , λ·xn ) = λ·f (x1 , . . . , xn ). Definition 3. By a basic function, we mean one of the following functions: • • •

a linear function f (x1 , . . . , xn ) = w1 · x1 + . . . + wn · xn ; a minimum function f (x1 , . . . , xn ) = min(xi1 , . . . , xim ); and a maximum function f (x1 , . . . , xn ) = max(xi1 , . . . , xim ).

We also say that basic functions are 1-level compositions of basic functions. We say that a function f (x1 , . . . , xn ) is a k-level composition of basic functions if f (x1 , . . . , xn ) = g(h1 (x1 , . . . , xn ), . . . , hm (x1 , . . . , xn )), where g is a basic function, and the functions h1 (x1 , . . . , xn ), . . . , hm (x1 , . . . , xn ) are (k − 1)-level compositions of basic functions. By induction over k, one can easily prove that all compositions of basic functions are homogeneous. For example:

46

• • •

H.T. Nguyen et al.

a linear combination is a basic function; an OWA combination of two values is a 2-level composition of basic functions; a general OWA operation is a 3-level composition of basic functions.

It turns out that an arbitrary homogeneous function can be approximated by appropriate 3-level compositions. Definition 4. Let k > 0 be a positive integer. We say that k-level compositions have a universal approximation property for homogeneous functions if for every continuous homogeneous function f (x1 , . . . , xn ), and for every two numbers ε > 0 and Δ > 0, there exists a function f(x1 , . . . , xn ) which is a k-level composition of basic functions and for which 1 , . . . , xn )| ≤ ε |f (x1 , . . . , xn ) − f(x for all x1 , . . . , xn for which |xi | ≤ Δ for all i. Theorem 2. 3-level compositions have a universal approximation property for homogeneous functions. A natural question is: do we need that many levels of composition? What is we only use 1- or 2-level compositions? It turns out that in this case, we will not get the universal approximation property – and thus, the 3 levels of OWA operations is the smallest possible number. Theorem 3 • •

1-layer computations do not have a universal approximation property for homogeneous functions; 2-layer computations do not have a universal approximation property for homogeneous functions.

Comment. A natural question is: why should we select linear functions, min, and max as basic functions? One possible answer is that these operations are the fastest to compute, i.e., they require the smallest possible number of computational steps. Indeed, the fastest computer operations are the ones which are hardware supported, i.e., the ones for which the hardware has been optimized. In modern computers, the hardware supported operations with numbers include elementary arithmetic operations (+, −, ·, /, etc.), and operations min and max. In the standard (digital) computer (see, e.g., [3]): • • • •

addition of two n-bit numbers requires, in the worst case, 2n bit operations: n to add corresponding digits, and n to add carries; multiplication, in the worst case, means n additions – by each bit of the second factor; so, we need O(n2 ) bit operations; division is usually performed by trying several multiplications, so it takes even longer than multiplication; finally, min and max can be performed bit-wise and thus, require only n bit operations.


47

Thus, the fastest elementary operations are indeed addition (or, more generally, linear combination), min, and max. Proof of Theorems 2 and 3 1◦ . Before we start proving, let us notice that the values of the functions min(xi1 , . . . , xim ) and max(xi1 , . . . , xim ) depend on the order between the values x1 , . . . , xn . There are n! possible orders, so we can divide the whole n-dimensional space of all possible tuples (x1 , . . . , xn ) into n! zones corresponding to these different orders. 2◦ . In each zone, a basic function is linear: • • •

a linear function is, of course, linear; a minimizing function min(xi1 , . . . , xim ) is simply equal to the variable xik which is the smallest in this zone and is, thus, linear; a maximizing function max(xi1 , . . . , xim ) is simply equal to the variable xik which is the largest in this zone and is, thus, also linear.

3◦ . If a function f (x1 , . . . , xn ) can be approximated, with arbitrary accuracy, by functions from a certain class, this means that f (x1 , . . . , xn ) is a limit of functions from this class. 4◦ . Basic functions are linear in each zone; thus, their limits are also linear in each zone. Since some homogeneous functions are non-linear, we can thus conclude that basic functions do not have a universal approximation property for homogeneous functions. 5◦ . Let us now consider 2-level compositions of basic functions, i.e., functions of the type f (x1 , . . . , xn ) = g(h1 (x1 , . . . , xn ), . . . , hm (x1 , . . . , xn )), where g and hi are basic functions. Since there are three types of basic functions, we have three options: • • •

it is possible that g(x1 , . . . , xm ) is a linear function; it is possible that g(x1 , . . . , xm ) is a minimizing function; and it is possible that g(x1 , . . . , xm ) is a maximizing function.

Let us consider these three options one by one. 5.1◦ . Let us start with the first option, when g(x1 , . . . , xm ) is a linear function. Since on each zone, each basic function hi is also linear, the composition f (x1 , . . . , xn ) is linear on each zone. 5.2◦ . If g(x1 , . . . , xm ) is a minimizing function, then on each zone, each hi is linear and thus, the composition f (x1 , . . . , xn ) is a minimum of linear functions. It is known that minima of linear functions are concave; see, e.g., [21]. So, within this option, the function f (x1 , . . . , xn ) is concave. 5.3◦ . If g(x1 , . . . , xm ) is a maximizing function, then on each zone, each hi is linear and thus, the composition f (x1 , . . . , xn ) is a maximum of linear functions. It is known that maxima of linear functions are convex; see, e.g., [21]. So, within this option, the function f (x1 , . . . , xn ) is convex.

48

H.T. Nguyen et al.

6◦ . In each zone, 2-level compositions of basic functions are linear, concave, or convex. The class of all functions approximable by such 2-level compositions is the class of limits (closure) of the union of the corresponding three classes: of linear, concave, and convex sets. It is known that the closure of the finite union is the union of the corresponding closures. A limit of linear functions is always linear, a limit of concave functions is concave, and a limit of convex functions is convex. Thus, by using 2-level compositions, we can only approximate linear, concave, or convex functions. Since there exist homogeneous functions which are neither linear nor concave or convex, we can thus conclude that 2-level compositions are not universal approximators for homogeneous functions. 7◦ . To complete the proof, we must show that 3-level compositions are universal approximators for homogeneous functions. There are two ways to prove it. 7.1◦ . First, we can use the known facts about concave and convex functions [21]: • •

that every continuous function on a bounded area can be represented as as a difference between two convex functions, and that every convex function can be represented as a maximum of linear functions – namely, all the linear functions which are smaller than this function.

These facts are true for general (not necessarily homogeneous) functions. For homogeneous functions f (x1 , . . . , xn ), one can easily modify the existing proofs and show: • •

that every homogeneous continuous function on a bounded area can be represented as as a difference between two convex homogeneous functions, and that every homogeneous convex function can be represented as a maximum of homogeneous linear functions – namely, all the homogeneous linear functions which are smaller than this function.

Thus, we can represent the desired function f (x1 , . . . , xn ) as the difference between two convex homogeneous functions f (x1 , . . . , xn ) = f1 (x1 , . . . , xn ) − f2 (x1 , . . . , xn ). Each of these convex functions can be approximated by maxima of linear functions and thus, by 2-level compositions. Substraction f1 −f2 adds the third level, so f (x1 , . . . , xn ) can indeed be approximated by 3-level compositions. To prove that a function f (x1 , . . . , xn ) can be represented as a different between two convex functions, we can, e.g., first approximate it by a homogeneous function which is smooth on a unit sphere {(x1 , . . . , xn ) : x21 + . . . + x2n = 1},


49

and then take f1 (x1 , . . . , xn ) = k · x21 + . . . + x2n for a large k. For smooth functions, convexity means that the Hessian matrix – consisting of its second ∂2f derivatives – is positive definite. ∂xi ∂xj For sufficiently large k, the difference f2 (x1 , . . . , xn ) = f1 (x1 , . . . , xn ) − f (x1 , . . . , xn ) is also convex – since its second derivatives matrix is dominated by positive definite terms coming from f1 . Thus, the difference f1 − f2 = f is indeed the desired difference. 7.2◦ . Another, more constructive proof, is, for some δ > 0, to select a finite δ dense set of points e = (e1 , . . . , en ) on a unit square. For each such point, we build a 2-level composition which coincides with f on the corresponding ray {λ · (e1 , . . . , en ) : λ > 0}. This function can be obtained, e.g., as a minimum of several linear functions which have the right value on this ray but change drastically immediately outside this ray. For example, let f0 (x) be an arbitrary homogeneous linear function which coincides with f (x) at the point e – and thus, on the whole ray. To construct the corresponding linear functions, we can expand the vector e to an orthonormal basis e, e , e , etc., and take linear functions f0 (x)+k·(e ·x) and f0 (x) − k · (e · x) for all such e (and for a large k > 0). Then, the minimum of all these functions is very small outside the ray. We then take the maximum of all these minima – a 3-level composition. The function f (x1 , . . . , xn ) is continuous on a unit sphere and thus, uniformly continuous on it, i.e., for every ε > 0, there is a δ such that δ-close value on the unit sphere lead to ε-close values of f . By selecting appropriate δ and k (depending on δ), we can show that the resulting maximum is indeed ε-close to f . The theorem is proven.

7 Conclusions Fuzzy techniques have been originally invented as a methodology that transforms the knowledge of experts (formulated in terms of natural language) into a precise computer-implementable form. There are many successful applications of this methodology to situations in which expert knowledge exist; the most well known (and most successful) are applications to fuzzy control. In some cases, fuzzy methodology is applied even when no expert knowledge exists. In such cases, instead of trying to approximate the unknown control function by splines, polynomials, or by any other traditional approximation technique, researchers try to approximate it by guessing and tuning the expert rules. Surprisingly, this approximation often works fine. Similarly, in multi-criteria decision making, it is necessary to aggregate (combine) utility values corresponding to several criteria (parameters). The

50

H.T. Nguyen et al.

simplest way to combine these values is to use linear aggregation. In many practical situations, however, linear aggregation does not fully adequately describe the actual decision making process, so non-linear aggregation is needed. From the purely mathematical viewpoint, the next natural step after linear functions is the use of quadratic functions. However, in decision making, a different type of non-linearities are usually more adequate than quadratic ones: fuzzy-type non-linearities like OWA or Choquet integral that use min and max in addition to linear combinations. In this chapter, we give a mathematical explanation for this empirical phenomenon. Specifically, we show that approximation by using fuzzy methodology is indeed the best (in some reasonable sense).

Acknowledgments This work was supported in part by NSF grants HRD-0734825, EAR-0225670, and EIA-0080940, by Texas Department of Transportation grant No. 0-5453, by the Japan Advanced Institute of Science and Technology (JAIST) International Joint Research Grant 2006-08, and by the Max Planck Institut f¨ ur Mathematik.

References 1. Buckley, J.J.: Sugeno type controllers are universal controllers. Fuzzy Sets and Systems 53, 299–303 (1993) 2. Ceberio, M., Modave, F.: An interval-valued, 2-additive Choquet integral for multi-cruteria decision making. In: Proceedings of the 10th Conf. on Information Processing and Management of Uncertainty in Knowledge-based Systems IPMU 2004, Perugia, Italy (July 2004) 3. Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to Algorithms. MIT Press, Cambridge (2001) 4. Feynman, R., Leighton, R., Sands, M.: Feynman Lectures on Physics. Addison Wesley, Reading (2005) 5. Grabisch, M., Murofushi, T., Sugeno, M. (eds.): Fuzzy Measures and Integrals. Physica-Verlag, Heidelberg (2000) 6. Kandel, A., Langholtz, G. (eds.): Fuzzy Control Systems. CRC Press, Boca Raton (1994) 7. Keeney, R.L., Raiffa, H.: Decisions with Multiple Objectives. John Wiley and Sons, New York (1976) 8. Klir, G., Yuan, B.: Fuzzy sets and fuzzy logic: theory and applications. Prentice Hall, Upper Saddle River (1995) 9. Kosko, B.: Fuzzy systems as universal approximators. In: Proceedings of the 1st IEEE International Conference on Fuzzy Systems, San Diego, CA, pp. 1153– 1162 (1992) 10. Kreinovich, V., Bernat, A.: Parallel algorithms for interval computations: an introduction. Interval Computations 3, 6–62 (1994)


51

11. Kreinovich, V., Mouzouris, G.C., Nguyen, H.T.: Fuzzy rule based modeling as a universal approximation tool. In: Nguyen, H.T., Sugeno, M. (eds.) Fuzzy Systems: Modeling and Control, pp. 135–195. Kluwer, Boston (1998) 12. Kreinovich, V., Nguyen, H.T., Yam, Y.: Fuzzy systems are universal approximators for a smooth function and its derivatives. International Journal of Intelligent Systems 15(6), 565–574 (2000) 13. Lea, R.N., Kreinovich, V.: Intelligent control makes sense even without expert knowledge: an explanation. In: Reliable Computing. Supplement, Extended Abstracts of APIC 1995: International Workshop on Applications of Interval Computations, El Paso, TX, February 23–25, pp. 140–145 (1995) 14. Luce, R.D., Raiffa, H.: Games and Decisions: Introduction and Critical Survey. Dover, New York (1989) 15. Modave, F., Ceberio, M., Kreinovich, V.: Choquet integrals and OWA criteria as a natural (and optimal) next step after linear aggregation: a new general justification. In: Proceedings of the 7th Mexican International Conference on Artificial Intelligence MICAI 2008, Mexico City, Mexico, October 27–31 (to appear) (2008) 16. Nguyen, H.T., Kreinovich, V.: On approximation of controls by fuzzy systems. In: Proceedings of the Fifth International Fuzzy Systems Association World Congress, Seoul, Korea, pp. 1414–1417 (July 1993) 17. Nguyen, H.T., Kreinovich, V.: Fuzzy aggregation techniques in situations without experts: towards a new justification. In: Proceedings of the IEEE Conference on Foundations of Computational Intelligence FOCI 2007, Hawaii, April 1–5, pp. 440–446 (2007) 18. Nguyen, H.T., Walker, E.A.: A first course in fuzzy logic. CRC Press, Boca Raton (2005) 19. Perfilieva, I., Kreinovich, V.: A new universal approximation result for fuzzy systems, which reflects CNF-DNF duality. International Journal of Intelligent Systems 17(12), 1121–1130 (2002) 20. Raiffa, H.: Decision Analysis. Addison-Wesley, Reading (1970) 21. Rockafeller, R.T.: Convex Analysis. Princeton University Press, Princeton (1970) 22. Wang, L.-X.: Fuzzy systems are universal approximators. In: Proceedings of the IEEE International Conference on Fuzzy Systems, San Diego, CA, pp. 1163– 1169 (1992) 23. Wang, L.-X., Mendel, J.: Generating Fuzzy Rules from Numerical Data, with Applications, University of Southern California, Signal and Image Processing Institute, Technical Report USC-SIPI # 169 (1991) 24. Yager, R.R., Kacprzyk, J. (eds.): The Ordered Weighted Averaging Operators: Theory and Applications. Kluwer, Norwell (1997) 25. Yager, R.R., Kreinovich, V.: Universal approximation theorem for uninormbased fuzzy systems modeling. Fuzzy Sets and Systems 140(2), 331–339 (2003)

Intermediate Degrees Are Needed for the World to Be Cognizable: Towards a New Justification for Fuzzy Logic Ideas Hung T. Nguyen1 , Vladik Kreinovich2 , J. Esteban Gamez2 , Fran¸cois Modave2 , and Olga Kosheleva3 1

2

3

Department of Mathematical Sciences, New Mexico State University, Las Cruces, NM 88003, USA [email protected] Department of Computer Science, University of Texas at El Paso, El Paso, TX 79968, USA [email protected], [email protected], [email protected] Department of Teacher Education, University of Texas at El Paso, El Paso, TX 79968, USA [email protected]

Summary. Most traditional examples of fuzziness come from the analysis of commonsense reasoning. When we reason, we use words from natural language like “young”, “well”. In many practical situations, these words do not have a precise true-or-false meaning, they are fuzzy. One may therefore be left with an impression that fuzziness is a subjective characteristic, it is caused by the specific way our brains work. However, the fact that that we are the result of billions of years of successful adjusting-to-the-environment evolution makes us conclude that everything about us humans is not accidental. In particular, the way we reason is not accidental, this way must reflect some real-life phenomena – otherwise, this feature of our reasoning would have been useless and would not have been abandoned long ago. In other words, the fuzziness in our reasoning must have an objective explanation – in fuzziness of the real world. In this chapter, we first give examples of objective real-world fuzziness. After these example, we provide an explanation of this fuzziness – in terms of cognizability of the world.

1 Introduction One of the main ideas behind Zadeh’s fuzzy logic and its applications is that everything is a matter of degree. We are often accustomed to think that every statement about a physical world is true or false: • • •

that an object is either a particle or a wave, that a person is either young or not, that a person is either well or ill,

A.-E. Hassanien et al. (Eds.): Foundations of Comput. Intel. Vol. 2, SCI 202, pp. 53–74. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com

54

H.T. Nguyen et al.

but in reality, we sometimes encounter intermediate situations. That everything is a matter of degree is a convincing empirical fact, but a natural question is: why? How can we explain this fact? This is what we will try to do in this chapter: come up with a theoretical explanation of this empirical fact. Most traditional examples of fuzziness come from the analysis of commonsense reasoning. When we reason, we use words from natural language like “young”, “well”. In many practical situations, these words do not have a precise true-or-false meaning, they are fuzzy. One may therefore be left with an impression that fuzziness is a subjective characteristic, it is caused by the specific way our brains work. However, the fact that that we are the result of billions of years of successful adjustingto-the-environment evolution makes us conclude that everything about us humans is not accidental. In particular, the way we reason is not accidental, this way must reflect some real-life phenomena – otherwise, this feature of our reasoning would have been useless and would not have been abandoned long ago. In other words, the fuzziness in our reasoning must have an objective explanation – in fuzziness of the real world. In this chapter, we first give examples of objective real-world fuzziness. After these example, we provide an explanation of this fuzziness – in terms of cognizability of the world. Some of our results first appeared in the conference papers [4, 16].

2 Examples of Objective “Fuzziness” Fractals The notion of dimension has existed for centuries. Already the ancient researchers made a clear distinction between 0-dimensional objects (points), 1-dimensional objects (lines), 2-dimensional objects (surfaces), 3-dimensional objects (bodies), etc. In all these examples, dimension is a natural number: 0, 1, 2, 3, . . . Since the 19th century, mathematicians have provided a mathematical extension of the notion of dimension that allowed them to classify some weird mathematical sets as being of fractional (non-integer) dimension, but for a long time, these weird sets remained anomalies. In the 1970s, B. Mandlebrot noticed that actually, many real-life objects have fractional dimension, ranging from the shoreline of England to the shape of the clouds and mountains to noises in electric circuits (to social phenomena such as stock prices). He called such sets of fractional (non-integer) dimension fractals; see, e.g., [11, 12, 13]. It is now clear that fractals play an important role in nature. So, what we originally thought of as an integer-valued variable turned out to be real-valued. Quantum physics Until the 19th century, physical phenomena were described by classical physics. In classical physics, some variables are continuous, some are discrete. For

Intermediate Degrees Are Needed for the World to Be Cognizable

55

example, the coordinates and velocities of particles usually take continuous values. However, if we are interested in stable states or periodic trajectories, we often end up with a discrete set of stable states. This discreteness underlies most engineering implementations of computers: to represent 0 or 1, we select an object with 2 possible states, and use one of these states to represent 0 and another to represent 1. In the 20th century, however, it turned out that a more adequate description of the physical world comes from quantum physics. One of the peculiar features of quantum physics is the so-called superposition principle (see, e.g., [2]) according to which with every two states 0| and 1|, it is also possible to have “intermediate” states (superpositions) c0 · 0| + c1 · 1| for all complex values c0 and c1 for which |c0 |2 + |c1 |2 = 1. So, what we originally thought of as an integer-valued variable turned out to be real-valued. It is worth mentioning that these quantum combinations of 0 and 1 states are not only happening in real life, but, as it was discovered in the 1990s, their use can drastically speed up computations. For example: √ • we can search in an unsorted list of n elements in time n – which is much faster than the time n which is needed on non-quantum computers [6, 7, 18]; • we can factor a large integer in time which does not exceed a polynomial of the length of this integer – and thus, we can break most existing cryptographic codes like widely used RSA codes which are based on the difficulty of such a factorization on non-quantum computers [18, 21, 22]. These techniques form the basis of quantum computing; see, e.g., [18]. Fractional charges of quarks In the late 19th century and early 20th century, it was experimentally confirmed that seemingly continuous matter is actually discrete: it consists of molecules, molecules consist of atoms, and atoms consist of elementary particles. A part of this confirmation came from an experimental discovery that all electric charges are proportional to a single charge – which was later revealed to be equal to the charge of an electron. Based on this proportionality, physicists concluded that many observed elementary particles ranging from (relatively) stables particles such as protons and neutrons to numerous unstable ones – like many mesons and baryons discovered in super-collides and in cosmic rays – cannot be further decomposed into “more elementary” objects. In the 1960s, M. Gell-Mann [2, 5, 20] discovered that if we allow particles with fractional electronic charge, then we can describe protons, neutrons, mesons, and baryons as composed of 3 (now more) even more elementary particles called quarks. At first, quarks were often viewed as purely mathematical constructions, but experiments with particle-particle collisions revealed that, within a proton, there are three areas (partons) where the reflection seems to be the largest – in perfect accordance with the fact that in the quark model, a proton consists of exactly three quarks.

56

H.T. Nguyen et al.

So, what we originally thought of as an integer-valued variable turned out to be real-valued. There exist other examples of objective “fuzziness” In physics, there are many other examples when what we originally thought of as an integer-valued variable turned out to be real-valued. In this chapter, we just described the most well known ones.

3 Our Explanation of Why Physical Quantities Originally Thought to Be Integer-Valued Turned out to Be Real-Valued: Main Idea In philosophical terms, what we are doing is “cognizing” the world, i.e., understanding how it works and trying to predict consequences of different actions – so that we will be able to select an action which is the most beneficial for us. Of course, our knowledge is far from complete, there are many real-world phenomena which we have not cognized yet – and many philosophers believe that some of these phenomena are not cognizable at all. If a phenomenon is not cognizable, there is nothing we can do about it. What we are interested in is phenomena which are cognizable. This is what we will base our explanation on – that in such phenomena, it is reasonable to expect continuous-valued variables, i.e., to expect that properties originally thought to be discrete are actually matters of degree.

4 First Explanation: G¨ odel’s Theorem vs. Tarski’s Algorithm G¨ odel’s theorem: a brief reminder. Our first explanation of “objective fuzziness” is based on the historically first result in which something was actually proven to be not cognizable – the well-known 1931 G¨ odel’s theorem; see, e.g., [3]. This theorem can be formulated in terms of arithmetic. Specifically, we have variables which run over natural numbers 0, 1, 2, . . . A term is anything that can be obtained from these variables and natural-valued constants by using addition and multiplication, e.g., 2 · x · y + 3 · z (subtraction is also allowed). Elementary formulas are defined as expressions of the type t = t , t < t , t > t , t ≤ t , t ≥ t , and t = t for some terms t and t . Examples are 2 · x · y + 3 · z = 0 or x < y + z. Finally, a formula is anything which is obtained from elementary formulas by using logical connectives “and” (&), “or” (∨), “implies” (→), “not” (¬), and quantifiers “for all x” (∀x) and “there exists x” (∃x). Example:


57

∀x ∀y(x < y → ∃z(y = x + y)). Many statements about the physical world can be formulated in terms of such formulas. Our objective is therefore to find out whether a given formula is true or false. G¨ odel’s theorem states that no algorithm is possible that would, given a formula, check whether this formula is true or false. In other words, if we allow variables with discrete values, then it is not possible to have an algorithm which would solve all the problems. Tarksi’s result In the 1940s, another well-known logician, Alfred Tarski, raised an interesting question: what if we only allow continuous variables? In other words, what if we consider the same formulas as Gödel considered, but we change their interpretation: now every variable can take arbitrary real values. It turns out that in this case, it is possible to have an algorithm that, given a formula, checks whether this formula is true or false. [23]. Thus, in a cognizable situations, we cannot have variables which only take discrete values – these variables must be able to take arbitrary real values [4, 16]. we have to note that it is worth mentioning that the original Tarski’s algorithm required an unrealistically large amount of computation time; however, later, faster, practically useful algorithms have been invented; see, e.g., [1, 14].

5 Second Explanation: Efficient Algorithms vs. NP-Hardness Not all algorithms are practical Our first explanation of continuity (and “fuzziness”) was that with the discrete variables, we cannot have a deciding algorithm, but with continuous variables, we can. The existence of an algorithm is necessary for cognition, but not sufficient. It is well known that some theoretical algorithms are not practical at all. For example, if an algorithm requires an exponential number of computational steps 2n on an input of size n, this means that for inputs of a reasonable size n ≈ 300 − 400, the required computation time exceeds the lifetime of the Universe. Feasible vs. non-feasible algorithms There is still no perfect formalization of this difference between “practical” (feasible) and impractical (non-feasible) algorithms. Usually:

58

• •

H.T. Nguyen et al.

algorithms for which the computation time tA (x) is bounded by some polynomial P (n) of the length n = len(x) of the input (e.g., linear-time, quadratic-time, etc.) are practically useful, while for practically useless algorithms, the computation time grows with the size of the input much faster than a polynomial.

In view of this empirical fact, in theoretical computer science, algorithms are usually considered feasible if their running time is bounded by a polynomial of n. The class of problems which can be solved in polynomial time is usually denoted by P; see, e.g., [19]. Notion of NP-hardness Not all practically useful problems can be solved in polynomial time. To describe such problems, researchers have defined several more general classes of problems. One of the most well known classes is the class NP. By definition, this class consists of all the problems which can be solved in non-deterministic polynomial time – meaning that if we have a guess, we can check, in polynomial time, whether this guess is a solution to our problem. Most computer scientists believe that NP=P, i.e., that some problems from the class NP cannot be solved in polynomial time. However, this inequality has not been proven, it is still an open problem. What is known is that some problems are NP-hard, i.e., any problem from the class NP can be reduced to each of these problems in polynomial time. One of such NP-hard problems is the problem SAT of propositional satisfiability: given a propositional formula F , i.e., a formula obtained from Boolean (yes-no) variables x1 , . . . , xn by using &, ∨, and ¬, check whether there exist values x1 , . . . , xn which make this formula true. NP-hardness of SAT means that if NP=P (i.e., if at least one problem from the class NP cannot be solved in polynomial time), then SAT also cannot be solved in polynomial time. In other words, SAT is the hardest of the problems from this class. It is known that all the problems from the class NP can be solved in exponential time. Indeed, for a problem of size n, there are ≤ an possible guesses, where a is the size of the corresponding alphabet, so we can simply try all these guesses one by one. Example: systems of linear equations One of the simplest-to-solve numerical problems is the solution to a system of linear equations a11 · x1 + . . . + a1n · xn = b1 ; ... am1 · x1 + . . . + amn · xn = bm . In the situation when all the unknowns xi can take arbitrary real values, there exist efficient algorithms for solving such systems of equations – even the


59

well-known Gauss elimination method, while not the fastest, it still feasible. However, as soon as we restrict ourselves to discrete (e.g., integer-valued) variables xi , the solution of such a system becomes an NP-hard problem [19]. So, we end up with the same conclusion: that in a cognizable situations, we cannot have variables which only take discrete values – these variables must be able to take arbitrary real values.

6 Case Study: Selecting the Most Representative Sample Introduction to the problem. In many practical situations, it is desirable to find the statistical analysis of a certain population, but this population is so large that it is not practically possible to analyze every individual element from this population. In this case, we select a sample (subset) of the population, perform a statistical analysis on this sample, and use these results as an approximation to the desired statistical characteristics of the population as a whole. For example, this is how polls work: instead of asking the opinion of all the people, pollsters ask a representative sample, and use the opinion of this sample as an approximation to the opinion of the whole population. The more “representative” the sample, the larger our confidence that the statistical results obtained by using this sample are indeed a good approximation to the desired population statistics. Typically, we gauge the representativeness of a sample by how well its statistical characteristics reflect the statistical characteristics of the entire population. For example, in the sample of human voters, it is reasonable to require that in the selected sample, the average age, the average income, and other characteristics are the same as in the population in a whole. Of course, the representativeness of averages is not enough: e.g., the voting patterns of people whose salary is exactly the national average are not necessarily a good representation of how people will work on average. For that, we need the sample to include both poorer and reacher people – i.e., in general, to be representative not only in terms of averages but also in terms of, e.g., standard deviations (i.e., equivalently, in terms of variances). In practice, many techniques are used to design a representative sample; see, e.g., [10]. In this section, we show that the corresponding exact optimization problem is computationally difficult (NP-hard). How is this result related to fuzzy techniques? The main idea behind fuzzy techniques is that they formalize expert knowledge expressed by words from natural language. In this section, we show that if we do not use this knowledge, i.e., if we only use the data, then selecting the most representative sample becomes a computationally difficult (NP-hard) problem. Thus, the need to find such samples in reasonable time justifies the use of fuzzy techniques.

60

H.T. Nguyen et al.

We have to note that similar results are known: for example, it is known that a similar problem of maximizing diversity is NP-hard; see, e.g., [9]. Towards formulation of the problem in exact terms. Let us assume that we have a population consisting of N objects. For each of N objects, we know the values of k characteristics x1 , x2 , . . . , xk . The value of the first characteristic x1 for i-th object will be denoted by x1,i , the value of the second characteristic x2 for the i-th object will be denoted by x2,i , . . . , and finally, the value of the characteristic xk for the i-th object will be denoted by xk,i . As a result, we arrive at the following formal definition: Definition 1. By a population, we mean a tuple def

p = N, k, {xj,i } , where: • • •

N is an integer; this integer will be called the population size; k is an integer; this integer is called the number of characteristics; xj,i (1 ≤ j ≤ k, 1 ≤ i ≤ N ) are real numbers; the real number xj,i will be called the value of the j-th characteristic for the i-th object.

Based on these known values, we can compute the population means E1 =

N N 1 1 · x1,i , E2 = · x2,i , . . . , N i=1 N i=1

and the population variances V1 =

N N 1 1 · (x1,i − E1 )2 , V2 = · (x2,i − E2 )2 , . . . N i=1 N i=1

We can also compute higher order central moments. Definition 2. Let p = N, k, {xj,i } be a population, and let j be an integer from 1 to k. •

By the population mean Ej of the j-th characteristic, we mean the value N 1 Ej = · xj,i . N i=1

•

By the population variance Vj of the j-th characteristic, we mean the value N 1 Vj = · (xj,i − Ej )2 . N i=1


•

61

For every integer d ≥ 1, by the even order population central moment (2d) Mj of order 2d of the j-th characteristic, we mean the value (2d)

Mj

=

N 1 · (xj,i − Ej )2d . N i=1

Comment. In particular, the population central moment Mj of order 2 (corresponding to d = 1) is simply the population variance. In addition to the values x1,i , x2,i , . . . , we are given a size n < N of the desirable sample. For each sample I = {i1 , . . . , in } ⊆ {1, 2, . . . , N } of size n, we can compute the sample means (2)

E1 (I) =

1 1 x1,i , E2 (I) = x2,i , . . . n n i∈I

i∈I

and the sample variances V1 (I) =

1 1 (x1,i − E1 (I))2 , V2 (I) = (x2,i − E2 (I))2 , . . . n n i∈I

i∈I

Definition 3. Let N be a population size. •

By a sample, we mean a non-empty subset I ⊆ {1, 2, . . . , N }.

•

For every sample I, by its size, we mean the number of elements in I.

Definition 4. Let p = N, k, {xj,i } be a population, let I be a sample of size n, and let j be an integer from 1 to k. •

By the sample mean Ej (I) of the j-th characteristic, we mean the value Ej (I) =

1 · xj,i . n i∈I

•

By the sample variance Vj (I) of the j-th characteristic, we mean the value Vj (I) =

1 · (xj,i − Ej (I))2 . n i∈I

•

For every d ≥ 1, by the sample central moment Mj the j-th characteristic, we mean the value

(2d)

(2d)

Mj

(I) =

1 · (xj,i − Ej (I))2d . n i∈I

(I) of order 2d of

62

H.T. Nguyen et al.

Comment. Similarly to the population case, the sample central moment Mj of order 2 (corresponding to d = 1) is simply the sample variance. We want to select the most representative sample, i.e., the sample I for which the sample statistics E1 (I), E2 (I), . . . , V1 (I), V2 (I), . . . are the closest to the population statistics E1 , E2 , . . . , V1 , V2 , . . . (2)

Definition 5. Let p = N, k, {xj,i } be a population. •

By an E-statistics tuple corresponding to p, we mean a tuple def

t(1) = (E1 , . . . , Ek ). •

By an (E, V )-statistics tuple corresponding to p, we mean a tuple def

t(2) = (E1 , . . . , Ek , V1 , . . . , Vk ). •

For every integer d ≥ 1, by a statistics tuple of order 2d corresponding to p, we mean a tuple def

(2)

(2)

(4)

(4)

(2d)

t(2d) = (E1 , . . . , Ek , M1 , . . . , Mk , M1 , . . . , Mk , . . . , M1

(2d)

, . . . , Mk

).

Comment. In particular, the statistics tuple of order 2 is simply the (E, V )statistics tuple. Definition 6. Let p = N, k, {xj,i } be a population, and let I be a sample. •

By an E-statistics tuple corresponding to I, we mean a tuple def

t(1) (I) = (E1 (I), . . . , Ek (I)). •

By an (E, V )-statistics tuple corresponding to I, we mean a tuple def

t(2) (I) = (E1 (I), . . . , Ek (I), V1 (I), . . . , Vk (I)). •

For every integer d ≥ 2, by a statistics tuple of order 2d corresponding to I, we mean a tuple def

(2)

(2)

t(2d) (I) = (E1 (I), . . . , Ek (I), M1 (I), . . . , Mk (I), (4)

(4)

(2d)

M1 (I), . . . , Mk (I) . . . , M1

(2d)

(I), . . . , Mk

(I)).

Comment. In particular, the statistics tuple of order 2 corresponding to a sample I is simply the (E, V )-statistics tuple corresponding to this same tuple. We will show that no matter how we define closeness, this problem is NP-hard (computationally difficult).


63

Let us describe the problem in precise terms. To describe which tuple def

t(I) = (E1 (I), E2 (I), . . . , V1 (I), V2 (I), . . .) is the closest to the original statistics tuple def

t = (E1 , E2 , . . . , V1 , V2 , . . .), we need to fix a distance function ρ(t(I), t) describing how distant are the two given tuples. Similarly to the usual distance, we would like this distance function to be equal to 0 when the tuples coincide and to be positive if when the tuples are different. So, we arrive at the following definitions. Definition 7. By a distance function, we mean a mapping ρ that maps every two real-valued tuples t and t of the same size into a real value ρ(t, t ) in such a way that ρ(t, t) = 0 for all tuples t and ρ(t, t ) > 0 for all t = t . As an example, we can take Euclidean metric between the tuples t = (t1 , t2 , . . .) and t = (t1 , t2 , . . .): (tj − tj )2 . ρ(t, t ) = j

Now, we are ready to formulate the problem. Definition 8. Let ρ be a distance function. By a E-sample selection problem corresponding to ρ, we mean the following problem. We are given: • •

a population p = N, k, {xj,i } , and an integer n < N .

Among all samples I ⊆ {1, . . .e, NI} of size n, we must find the sample I for which the distance ρ(t(1) (I), t(1) ) between the corresponding E-statistical tuples is the smallest possible. Definition 9. Let ρ be a distance function. By a (E, V )-sample selection problem corresponding to ρ, we mean the following problem. We are given: • •


Among all samples I ⊆ {1, . . . , N } of size n w e must find th e sample I for which the distance ρ(t(2) (I), t(2) ) between the corresponding (E, V )-statistical tuples is the smallest possible. Definition 10. Let ρ be a distance function, and let d ≥ 1 be an integer. By a 2d-th order sample selection problem corresponding to ρ, we mean the following problem. We are given:

64

• •

H.T. Nguyen et al.


Among all samples I ⊆ {1, . . . , N } of size n, we must find the sample I for which the distance ρ(t(2d) (I), t(2d) ) between the corresponding (2d)-th order statistical tuples is the smallest possible. Proposition 1. For every distance function ρ, the corresponding E-sample selection problem is NP-hard. Proposition 2. For every distance function ρ, the corresponding (E, V )sample selection problem is NP-hard. Proposition 3. For every distance function ρ and for every integer d ≥ 1, the corresponding (2d)-th order sample selection problem is NP-hard. What is NP-hardness: a brief informal reminder. In order to prove these results, let us recall what NP-hardness means. Informally, a problem P0 is called NP-hard if it is at least as hard as all other problems from the class NP (a natural class of problems). To be more precise, a problem P0 is NP-hard if every problem P from the class NP can be reduced to this problem P0 . A reduction means that to every instance p of the problem P, we must be able to assign (in a feasible, i.e., polynomial-time way) an instance p0 of our problem P0 in such a way that the solution to the new instance p0 will lead to the solution of the original instance p. For precise definitions, see, e.g., [19]. How NP-hardness is usually proved. The original proof of NP-hardness of certain problems P0 is rather complex, because it is based on explicitly proving that every problem from the class NP can be reduced to the problem P0 . However, once we have proven NP-hardness of a problem P0 , the proof of NP-hardness of other problems P1 is much easier. Indeed, from the above description of a reduction, one can easily see that reduction is a transitive relation: if a problem P can be reduced to a problem P0 , and the problem P0 can be reduced to a problem P1 , then, by combining these two reductions, we can prove that P can be reduced to P1 . Thus, to prove that a new problem P1 is NP-hard, it is sufficient to prove that one of the known NP-hard problems P0 can be reduced to this problem P1 . Indeed, since P0 is NP-hard, every other problem P from the class NP can be reduced to this problem P0 . Since P0 can be reduced to P1 , we can now conclude, by transitivity, that every problem P from the class NP can be reduced to this problem P1 – i.e., that the problem P1 is indeed NP-hard. Comment. As a consequence of the definition of NP-hardness, we can conclude that if a problem P0 is NP-hard, then every more general problem P1 is also NP-hard. Indeed, the fact that P0 is NP-hard means that every instance p of every problem P can be reduced to some instance p0 of the problem P0 . Since the


65

problem P1 is more general than the problem P0 , every instance p0 of the problem P0 is also an instance of the more general problem P1 . Thus, every instance p of every problem P can be reduced to some instance p0 of the problem P1 – i.e., that the more general problem P1 is indeed NPhard. Main idea of the proof: reduction to subset sum, a known NP-hard problem. We prove NP-hardness of our problem by reducing a known NP-hard problem to it: namely, a subset sum problem, in which we are given m positive integers s1 , . . . , sm , and we must find the signs εi ∈ {−1, 1} for which m

εi · si = 0;

i=1

see, e.g., [19]. A reduction means that to every instance s1 , . . . , sm of the subset sum problem, we must assign (in a feasible, i.e., polynomial-time way) an instance of our problem in such a way that the solution to the new instance will lead to the solution of the original instance. Reduction: explicit description. Let us describe this reduction: we take N = 2n, k = 2, n = m, and we select the values xj,i as follows: • •

x1,i = si and x1,m+i = −si for all i = 1, . . . , m; x2,i = x2,m+i = 2i for all i = 1, . . . , m.

We will show that for this new problem, the most representative sample I has ρ(t(I), t) = 0 if and only if the original instance of the subset sum problem has a solution. General analysis. Indeed, by definition of a distance function, the equality ρ(t(I), t) = 0 is equivalent to t(I) = t, i.e., to the requirement that for the sample I, means (and variances) within the sample are exactly the same as for the entire population. Consequences for the second component. Let us start by analyzing the consequences of this requirement for the mean of the second component. For the entire population of size N = 2m, for each i from 1 to m, we have two elements, i-th and (m + i)-th, with the value x2,i = x2,m+i = 2i . Thus, for the population as a whole, this mean is equal to E2 =

2 + 22 + . . . + 2m . m

For the selected subset I of size m, this mean should be exactly the same: E2 (I) = E2 . Thus, we must have E2 (I) =

2 + 22 + . . . + 2m . m

66

H.T. Nguyen et al.

By definition, E2 (I) =

1 · x2,i . m i∈I

Thus, we conclude that

def

S2 (I) =

x2,i = 2 + 22 + . . . + 2m .

i∈I

What can we now conclude about the set I? First of all, we can notice that in the sum 2+22 +. . .+2m , all the terms are divisible by 4 except for the first term 2. Thus, the sum itself is not divisible by 4. In our population, we have exactly two elements, element 1 and element m + 1, for which x2,1 = x2,m+1 = 2. For every other element, we have x2,i = x2,m+i = 2i for i ≥ 2 and therefore, the corresponding value is divisible by 4. In regards to a selection I, there are exactly three possibilities: • • •

the set I contains none of the two elements 1 and m + 1; the set I contains both elements 1 and m + 1; and the set I contains exactly one of the two elements 1 and m + 1.

In the first two cases, the contribution of these two elements to the sum S2 (I) is divisible by 4 (it is 0 or 4). Since all other elements in the sum S2 (I) are divisible by 4, we would thus conclude that the sum itself is divisible by 4 – which contradicts to our conclusion that this sum is equal to 2 + 22 + . . . + 2m and is, therefore, not divisible by 4. This contradiction shows that the set I must contain exactly one of the two elements 1 and m + 1. Let us denote this element by k1 . For this element, x2,k1 = 2. Subtracting x2,k1 and 2 from the two sides of the equality S2 (I) =

x2,i = 2 + 22 + . . . + 2m ,

i∈I

we conclude that S2 (I − {k1 }) =

x2,i = 22 + 23 + . . . + 2m .

i∈I−{k1 }

In the new sum 22 + 23 + . . . + 2m , all the terms are divisible by 23 = 8 except for the first term 22 . Thus, the sum itself is not divisible by 8. In our remaining population {2, . . . , m, m + 2, . . . , 2m}, we have exactly two elements, element 2 and element m + 2, for which x2,2 = x2,m+2 = 22 . For every other element, we have x2,i = x2,m+i = 2i for i ≥ 3 and therefore, the corresponding value is divisible by 3.


67

In regards to a selection I, there are exactly three possibilities: • • •

the set I contains none of the two elements 2 and m + 2; the set I contains both elements 2 and m + 2; and the set I contains exactly one of the two elements 2 and m + 2.

In the first two cases, the contribution of these two elements to the sum S2 (I − {k1 }) is divisible by 8 (it is 0 or 8). Since all other elements in the sum S2 (I − {k1 }) are divisible by 8, we would thus conclude that the sum itself is divisible by 8 – which contradicts to our conclusion that this sum is equal to 22 + 23 + . . . + 2m and is, therefore, not divisible by 8. This contradiction shows that the set I must contain exactly one of the two elements 2 and m + 2. Let us denote this element by k2 . For this element, x2,k2 = 22 . Subtracting x2,k2 and 22 from the two sides of the equality S2 (I − {k1 }) = x2,i = 22 + 23 + . . . + 2m , i∈I−{k1 }

we conclude that S2 (I − {k1 , k2 }) =

x2,i = 23 + 24 + . . . + 2m .

i∈I−{k1 ,k2 }

Now, we can similarly conclude that the set I contains exactly one element from the pair {3, m + 3}, and in general, for every i from 1 to m, we can conclude that the selection set I contains exactly one element ki from the pair {i, m + i}. Consequences for the first component. Let us now analyze the consequences of this requirement for the mean of the first component. For the entire population of size N = 2m, for each i from 1 to m, we have two elements, i-th and (m + i)-th, with the opposite values x1,i = si and x2,m+i = −si . Thus, for the population as a whole, this mean is equal to E1 = 0. For each i from 1 to m, the selection set contains exactly one element of these two: ki = i and ki = m + i. Thus, E1 (I) = 0 means that the m corresponding sum is equal to 0: x1,ki = 0. Here, x1,ki = εi · si , where: i=1

• •

εi = 1 if ki = i, and εi = −1 if ki = m + i. m Thus, we conclude that εi · si = 0 for some εi ∈ {−1, 1}, i.e., that the i=1

original instance of the subset problem has a solution. Equivalence. Vice versa, if the original instance of the subset problem has m a solution, i.e., if εi · si = 0 for some εi ∈ {−1, 1}, then we can select i=1

I = {k1 . . . , km }, where:

68

H.T. Nguyen et al.

• •

ki = i when εi = 1, and ki = m + i when εi = −1.

One can easily check that in this case, we have E1 (I) = E1 , E2 (I) = E2 , (2d) (2d) (2d) and M2 (I) = V1 (I) = V1 , V2 (I) = V2 , and, in general, M1 (I) = M1 (2d) M2 . Conclusion. The reduction is proven, so the problem of finding the most representative sample is indeed NP-hard. Discussion. In the definitions of sample selection problem P1 ( Definitions 8–10), the objective is to find the sample I of given size n (which is smaller than N , the size of the population) such that the distance ρ(t(I), t) is the smallest possible. In the above text, we have proved, in effect, that the selection of a sample I of a given size n (< N ), such that the distance ρ(t(I), t) = 0, is NP-hard. The distance is always non-negative. Thus, when the smallest possible distance is 0, finding the sample I for which the distance ρ(t(I), t) is the smallest possible is equivalent to finding the sample for which this distance is zero. In general, the smallest possible distance does not necessarily equal to 0. Thus, the sample selection problem P1 is more general that the auxiliary “zero-distance” problem P0 for which we have proven NP-hardness. We have already mentioned earlier that if a problem P0 is NP-hard, then a more general problem P1 is NP-hard as well. Thus, we have indeed proved that the (more general) sample selection problem is NP-hard. Towards auxiliary results. In our proofs, we considered the case when the desired sample contains half of the original population. In practice, however, samples form a much smaller portion of the population. A natural question is: what if we fix a large even number 2P 2, and look for samples which constitute the (2P )-th part of the original population? It turns out that the resulting problem of selecting the most representative sample is still NP-hard. Definition 11. Let ρ be a distance function, and let 2P be a positive even 1 integer. By a problem of selecting an E-sample of relative size , we mean 2P the following problem: • We are given a population p = N, k, {xj,i } . n • Among all samples I ⊆ {1, . . . , N } of size n = , we must find the 2P (1) (1) sample I for which the distance ρ(t (I), t ) between the corresponding E-statistical tuples is the smallest possible. Definition 12. Let ρ be a distance function, and let 2P be a positive even 1 , we integer. By a problem of selecting an (E, V )-sample of relative size 2P mean the following problem:


• •

69

We are given a population p = N, k, {xj,i } . n Among all samples I ⊆ {1, . . . , N } of size n = , we must find the 2P (2) (2) sample I for which the distance ρ(t (I), t ) between the corresponding (E, V )-statistical tuples is the smallest possible.

Definition 13. Let ρ be a distance function, let let 2P be a positive even integer. By a problem of 1 sample of relative size , we mean the following 2P • We are given a population p = N, k, {xj,i } . • Among all samples I ⊆ {1, . . . , N } of size n

d ≥ 1 be an integer, and selecting an (2d)-th order problem:

n , we must find the 2P sample I for which the distance ρ(t(2d) (I), t(2d) ) between the corresponding statistical tuples of order 2d is the smallest possible. =

Proposition 4. For every distance function ρ and for every even integer 1 2P , the corresponding problem of selecting an E-sample of relative size 2P is NP-hard. Proposition 5. For every distance function ρ and for every even integer 2P , the corresponding problem of selecting an (E, V )-sample of relative size 1 is NP-hard. 2P Proposition 6. For every distance function ρ, for every integer d ≥ 1, and for every even integer 2P , the corresponding problem of selecting a (2d)-th 1 order sample of relative size is NP-hard. 2P Proof of Propositions 4–6. The proof is similar to the proofs of Propositions 1–3. The main difference is that for each i from 1 to m, we now have not two but 2P different objects i, m + i, 2m + i, . . . , k · m + i, . . . , (2P − 1) · m + i with the same value x2,i = x2,m+i = . . . = x2,k·m+i = . . . = x2,(2P −1)·m+i = (2P )i . (And this common value is also different.) Among these 2P objects with the same value of the second characteristic x2,. , for the first half, we have x1,. = si and for the second half, we have x1,. = −si , i.e.: x1,i = x1,m+i = . . . = m1,(P −1)·m+i = si ; x1,P ·m+i = x1,(P +1)·m+i = . . . = m1,(2P −1)·m+i = −si .

70

H.T. Nguyen et al.

By using divisibility by (2P )2 (instead of divisibility by 22 ), we conclude that the best fitting sample is the one which has exactly one element of each group. Thus, from E1 (I) = E1 , we similarly conclude that the original instance of the subset problem has a solution – and hence that the new problems are indeed NP-hard.

7 Symmetry: Another Fundamental Reason for Continuity (“Fuzziness”) Case study: benzene. To explain why symmetry leads to continuity, let us start with a chemical example. In the traditional chemistry, a molecule is composed from atoms that exchange electrons with each other. If an atom borrows one electron from another atom, we say that they have a connection of valence 1, if two electrons, there is a connection of valence 2, etc. From the analysis of benzene, it has been clear that it consists of 6 carbon and six hydrogen atoms, i.e., that its chemical formula is C6 H6 . However, for a long time, it was not clear how exactly they are connected to each other. The solution came in the 19th century to a chemist August Kekule in a dream. He dreamed of six monkeys that form a circle in which each monkey holds to the previous monkey’s tail. According to this solution, the six C atoms form a circle. To each of these atoms, a H atom is attached. Each C atom has a 1 valence connection to H, 1 valence connection to one of its neighbors, and 2 to another neighbor. The resulting chemical structure is still routinely described in chemical textbooks – because a benzene loop is a basis of organic chemistry and life. However, now we understand that this formula is not fully adequate. Indeed, according to this formula, the connections between C atoms are of two different types: of valence 1 and of valence 2. In reality, the benzene molecule is completely symmetric, there is no difference between the strengths of different connections. It is not possible to have a symmetric configuration is we require that valencies are integers. To equally split the remaining valence of 3 (1 is taken for H) between the two neighbors, we need a valence of 3/2. This is not possible in classical chemistry – but this is possible, in some sense, in quantum chemistry where, as we have mentioned, we have a continuum of intermediate states; see, e.g., [2]. Fuzzy logic itself is such an example. Fuzzy logic itself can be viewed as an example where symmetries leads to values intermediate between the original discrete values. Indeed, in traditional logic, we have two possible truth values: 1 (“true”) and 0 (“false”). How can we use this logic to describe the absence of knowledge? If we do not know whether a given statement A is true or not, this means that we have the exact same degree of belief in the statement A as we have in its negation ¬A. In the traditional logic, none of the two truth


71

values are symmetric (invariant) under such transformation A → ¬A. Thus, to adequately describe this situation, we need to also consider additional (intermediate) truth values. And indeed, in fuzzy logic with the set of truth values [0, 1] and the negation operation f¬ (x) = 1 − x, there is a value which is invariant under the operation A → ¬A: the value 0.5.

8 Case Study: Territory Division Formulation of the problem. In many conflict situations, several participants want to divide a territory between themselves. It may be farmer’s children dividing his farm, it may be countries dividing a disputed territory. Traditional (non-fuzzy) formalization of the problem. Let us follow [15] and describe a traditional (non-fuzzy) formalization of this problem. Let us denote the disputed territory (i.e., to be more precise, the set of all the points in this territory) by T . Our objective is to divide this territory between n participants, i.e., to select a division of the set T into the sets T1 , T2 , . . . , Tn for which Ti ∩ Tj = ∅ for i = j and T1 ∪ T2 ∪ . . . ∪ Tn = T. It is reasonable to assume that the utility ui of the i-th participant in acquiring the territory Ti is linear in Ti , i.e., has the form ui (Ti ) = Ui (x) dx Ti

for some appropriate function Ui (x). As we mentioned in [15], it is reasonable to use Nash’s criterion to select the optimal division, i.e., to select the division for which the product def

u = u1 (T1 ) · u2 (T2 ) · . . . · un (Tn ) attains the largest possible value. According to [15], in the optimal solution, for every participants i, there is a weight ci such that each point x is assigned to the participant with the largest weighted utility ci · Ui (x). In particular, for two participants, there is a threshold c such that all the points x for which U1 (x)/U2 (x) > c go to the first participant, and all the points x for which U1 (x)/U2 (x) < c go to the second participant. Possibility of a “fuzzy” solution. From the commonsense viewpoint, why do we have to necessarily divide all the disputed territory? Why cannot we control some parts of it together? In other words, instead of dividing the set T into subsets Ti , why cannot we assign, to every point x ∈ T and to every i, the degree di (x) to which the i-th participant will control the neighborhood of this point – in such a way that for every point x,

72

H.T. Nguyen et al.

d1 (x) + . . . + dn (x) = 1. In other words, instead of a crisp partition we have a fuzzy partition. In this setting, the utility ui of the i-th participant has the form ui (di ) = Ui (x) · di (x) dx, and our objective is to find a fuzzy partition for which the product def

u = u1 (d1 ) · u2 (d2 ) · . . . · un(dn ) attains the largest possible value. Observation: the above “fuzzy” problem always has a crisp optimal solution. The derivation from [15] was based on the idea that if we attain a maximum, then a small change of assignment in the vicinity of each point will only decrease (or not change) the desired product. For the fuzzy problem, a similar argument shows that there are weights ci such that in the optimal solution, every point x for which the weighted utility each point x is assigned to the participant with the largest weighted utility ci · Ui (x) of the i-th participant is larger than the weighted utility of all other participants is assigned to this i-th participant. The only points about which we cannot make a definite assignment are the ones in which two or more participants have exactly the same weighted utility. How we divide these points between these participants does not matter – as long as the overall degree of all the points assigned to each of these participants remains the same. In particular, this means that it is always possible to have a crisp division with the optimal value of the desired product. So, we arrive at a somewhat paradoxical situation: even when we allow “fuzzy” divisions, the corresponding optimization problem always have a crisp solution. So, at first glance, it may seem that fuzzy solutions are not needed at all. As we will see, the situation changes if we consider symmetry. Symmetry leads to fuzziness. For the territory division problem, a symmetry means a transformation f : T → T that preserves the area of each (crisp) subset and that preserves the utility of each subarea to each participant. Preserving area means that f has to be a measure-preserving transformation. Preserving utility means that we must have Ui (x) = Ui (f (x)) for all x. It is reasonable to require that if the original situation allows a symmetry, then the desired division should be invariant with respect to this symmetry. Let us show that this requirement leads to a fuzzy solution. Indeed, let us consider the simplest situation in which we have only two participants, and both assign equal value to all the points U1 (x) = U2 (x) = 1. In this case, the utility of each set Ti is simply equal to its area Ai , so the optimization problem takes the form


73

A1 · A2 → max . Since the sum A1 + A2 is equal to the area A of the original territory T , this problem takes the form A1 · (A − A1 ) → max . One can easily check that the optimal crisp solution means that A1 = A/2, i.e., that we divide the area T into two equal halves. This solution is optimal but it is not symmetric. Indeed, in this case, symmetries are simply area-preserving transformations. Symmetry of the division means that f (T1 ) = T1 for all such transformations f . However, for every two points x, y ∈ T , we can have an area-preserving transformation f that maps x into y: f (x) = y. In particular, we can have sauch a transformation for x ∈ T1 and y ∈ T2 , in which case f (T1 ) = T1 . Thus, a crisp symmetric solution is impossible. In contrast, a fuzzy symmetric solution is quite possible – and uniquely determined: we simply assign to each point x equal degrees d1 (x) = d2 (x) = 1/2. Then, f (d1 ) = d1 and f (d2 ) = d2 for all area-preserving transformations f . In general, we always have an optimal symmetric solution: in this solution, equally desired points – for which ci · Ui (x) = cj · Uj (x) – are all assigned a joint control with the same degree of ownership depending only on i and j.

9 Conclusion In this chapter, we have proven that from the natural assumption that the world is cognizable, we can conclude that intermediate degrees are needed to describe real-world processes. This conclusion provides an additional explanation for the success of fuzzy techniques (and other techniques which use intermediate degrees) – success which often goes beyond situations in which the intermediate degrees are needed to describe the experts’ uncertainty.

Acknowledgments This work was supported in part by NSF grants HRD-0734825, EAR-0225670, and EIA-0080940, by Texas Department of Transportation grant No. 0-5453, by the Japan Advanced Institute of Science and Technology (JAIST) International Joint Research Grant 2006-08, and by the Max Planck Institut f¨ ur Mathematik.

References 1. Basu, S., Pollack, R., Roy, M.-F.: Algorithms in Real Algebraic Geometry. Springer, Berlin (2006) 2. Feynman, R.P., Leighton, R., Sands, M.: The Feynman Lectures on Physics. Addison Wesley, Reading (2005)

74

H.T. Nguyen et al.

3. Franzen, T.: G¨ odel’s Theorem: An Incomplete Guide to its Use and Abuse. A. K. Peters (2005) 4. Gamez, J.E., Modave, F., Kosheleva, O.: Selecting the Most Representative Sample is NP-Hard: Need for Expert (Fuzzy) Knowledge. In: Proceedings of the IEEE World Congress on Computational Intelligence WCCI 2008, Hong Kong, China, June 1–6, pp. 1069–1074 (2008) 5. Gell-Mann, M.: The Quark and the Jaguar. Owl Books (1995) 6. Grover, L.K.: A fast quantum mechanical algorithm for database search. In: Proceedings of the 28th Annual ACM Symposium on the Theory of Computing, p. 212 (May 1996) 7. Grover, L.K.: From Schr¨ odinger’s equation to quantum search algorithm. American Journal of Physics 69(7), 769–777 (2001) 8. Klir, G., Yuan, B.: Fuzzy sets and fuzzy logic: theory and applications. Prentice Hall, Upper Saddle River (1995) 9. Kuo, C.C., Glover, F., Dhir, K.S.: Analyzing and modeling the maximum diversity problem by zero-one programming. Decision Sciences 24(6), 1171–1185 (1993) 10. Lohr, H.: Sampling: Design and Analysis. Duxbury Press (1999) 11. Mandelbrot, B.: Fractals: Form, Chance and Dimension. W. H. Freeman and Co., New York (1977) 12. Mandelbrot, B.: The Fractal Geometry of Nature. W. H. Freeman & Co., New York (1982) 13. Mandelbrot, B., Hudson, R.L.: The (Mis)Behavior of Markets: A Fractal View of Risk, Ruin, and Reward. Basic Books (2004) 14. Mishra, B.: Computational real algebraic geometry. In: Handbook on Discreet and Computational Geometry. CRC Press, Boca Raton (1997) 15. Nguyen, H.T., Kreinovich, V.: How to Divide a Territory? A New Simple Differential Formalism for Optimization of Set Functions. International Journal of Intelligent Systems 14(3), 223–251 (1999) 16. Nguyen, H.T., Kreinovich, V.: Everything Is a Matter of Degree: A New Theoretical Justification of Zadeh’s Principle. In: Proceedings of the 27th International Conference of the North American Fuzzy Information Processing Society NAFIPS 2008, New York, May 19–22 (2008) 17. Nguyen, H.T., Walker, E.A.: A first course in fuzzy logic. CRC Press, Boca Raton (2005) 18. Nielsen, M., Chuang, I.: Quantum Computation and Quantum Information. Cambridge University Press, Cambridge (2000) 19. Papadimitriou, C.H.: Computational Complexity. Addison Wesley, San Diego (1994) 20. Povh, B.: Particles and Nuclei: An Introduction to the Physical Concepts. Springer, Heidelberg (1995) 21. Shor, P.: Polynomial-Time Algorithms for Prime Factorization and Discrete Logarithms on a Quantum Computer. In: Proceedings of the 35th Annual Symposium on Foundations of Computer Science, Santa Fe, NM, November 20–22 (1994) 22. Shor, P.: Polynomial-Time Algorithms for Prime Factorization and Discrete Logarithms on a Quantum Computer. SIAM J. Sci. Statist. Comput. 26, 1484 (1997) 23. Tarski, A.: A Decision Method for Elementary Algebra and Geometry, 2nd edn., 63 p. Berkeley and Los Angeles (1951)

Paraconsistent Annotated Logic Program Before-after EVALPSN and Its Application Kazumi Nakamatsu School of Human Science and Environment University of Hyogo, 1-1-12 Shinzaike, HIMEJI 670-0092 Japan [email protected]

Summary. We have already proposed a paraconsistent annotated logic program called EVALPSN. In EVALPSN, an annotation called an extended vector annotation is attached to each literal. In order to deal with before-after relation between two time intervals, we introduce a new interpretation for extended vector annotations in EVALPSN, which is named Before-after(bf) EVALPSN. In this chapter, we introduce the bf-EVALPSN and its application to real-time process order control and its safety verification with simple examples. First, the background and overview of EVALPSN are introduced, and paraconsistent annotated logic as the formal background of EVALPSN and EVALPSN itself are recapitulated with simple examples. Then, after bfEVALPSN is formally defined, how to implement and apply bf-EVALPSN to real-time intelligent process order control and its safety verification with simple practical examples. Last, unique and useful features of bf-EVALPSN are introduced, and conclusions and remarks are provided.

1 Introduction and Background One of the main purposes of paraconsistent logic is to deal with inconsistency in a framework of consistent logical systems. It has been almost six decades since the first paraconsistent logical system was proposed by S.Jaskowski[12]. It was four decades later that a family of paraconsistent logic called “annotated logic” was proposed by da Costa et al.[8, 49], which can deal with inconsistency by introducing many truth values called “annotations” into their syntax as attached information to formulas. The paraconsistent annotated logic by da Costa et al.was developed from the viewpoint of logic programming by Subrahmanian et al.[7, 13, 48]. Furthermore, in order to deal with inconsistency and non-monotonic reasoning in a framework of annotated logic programming, ALPSN (Annotated Logic Program with Strong Negation) and its stable model semantics was developed by Nakamatsu and Suzuki [17]. It has been shown that ALPSN can deal with some non-monotonic reasonings such as default logic [46], autoepistemic A.-E. Hassanien et al. (Eds.): Foundations of Comput. Intel. Vol. 2, SCI 202, pp. 75–108. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com

76

K. Nakamatsu

logic [15] and a non-monotonic Assumption Based Truth Maintenance System(ATMS) [9] in a framework of annotated logic programming [18, 36, 37]. Even though ALPSN can deal with non-monotonic reasoning such as default reasoning and conflicts can be represented as paraconsistent knowledge in it, it is difficult and complicated to deal with reasoning to resolve conflicts in ALPSN. On the other hands, it is known that defeasible logic can deal with conflict resolving in a logical way [5, 41, 42], although defeasible logic cannot deal with inconsistency in its syntax and its inference rules are too complicated to implement them easily. In order to deal with conflict resolving and inconsistency in a framework of annotated logic programming, a new version of ALPSN, VALPSN (Vector Annotated Logic Program with Strong Negation) that can deal with defeasible reasoning and inconsistency was also developed by Nakamatsu et al.[22]. Moreover, it has been shown that VALPSN can be applied to conflict resolving in various systems [19, 20, 21]. It also has been shown that VALPSN provides a computational model of defeasible logic[5, 6]. Later, VALPSN was extended to EVALPSN (Extended VALPSN) by Nakamatsu et al. [23, 24] to deal with deontic notions (obligation, permission, forbiddance, etc.) and defeasible deontic reasoning [43, 44]. Recently, EVALPSN has been applied to various kinds of safety verification and intelligent control, for example, railway interlocking safety verification [27], robot action control [25, 28, 29, 39], safety verification for air traffic control [26], traffic signal control [30], discrete event control [31, 32, 33] and pipeline valve control [34, 35]. Considering the safety verification for process control, there is an occasion in which the safety verification for process order control is significant. For example, suppose a pipeline network in which two kinds of liquids, nitric acid and caustic soda are used for cleaning the pipelines. If those liquids are processed continuously and mixed in the same pipeline by accident, explosion by neutralization would be caused. In order to avoid such a dangerous accident, the safety for process order control should be strictly verified in a formal way such as EVALPSN. However, it seems to be a little difficult to utilize EVALPSN for verifying process order control as well as the safety verification for each process in process control. We have already proposed a new EVALPSN, bf(before-after)-EVALPSN that can deal with before-after relations between two time intervals [40]. This chapter mainly focuses on introducing bf-EVALPSN and its application to real-time process order control and its safety verification with simple process order control examples. As far as we know there seems to be no other efficient computational tool that can deal with the real-time safety verification for process order control than bf-EVALPSN. This chapter is organized as follows : firstly, in Section 1, the background and overview of the paraconsistent annotated logic program EVALPSN are introduced ; in Section 2, paraconsistent annotated logic as the background knowledge of EVALPSN and EVALPSN itself are formally recapitulated with simple examples ; in Section 3, after bf-EVALPSN is formally defined, how to

Paraconsistent Annotated Logic Program Before-after EVALPSN

77

implement and apply bf-EVALPSN to real-time safety verification for process order control is described with simple practical examples ; in Section 4, unique and useful features of bf-EVALPSN are introduced with examples ; lastly, conclusions and remarks are provided.

2 Paraconsistent Annotated Logic Program This section is devoted to clarify the formal background of the paraconsistent annotated logic program EVALPSN. The more details of EVALPSN has been introduced in [40]. We assume that the reader is familiar with the basic knowledge of classical logic and logic programming [14]. In order to understand EVLPSN and its reasoning we introduce Paraconsistent Annotated Logics P T [8] in the following subsection. 2.1

Paraconsistent Annotated Logic P T

Here we briefly recapitulate the syntax and semantics for propositional paraconsistent annotated logics proposed by da Costa et al. [8]. Generally, a truth value called an annotation is attached to each atomic formula explicitly in paraconsistent annotated logic, and the set of annotations constitutes a complete lattice. We introduce a paraconsistent annotated logic P T with the four valued complete lattice T . Definition 2.1 The primitive symbols of P T are : 1. propositional symbols p, q, · · · , pi , qi , · · · ; 2. each member of T is an annotation constant (we may call it simply an annotation) ; 3. the connectives and parentheses ∧, ∨, →, ¬, (, ) . Formulas are defined recursively as follows : 1. if p is a propositional symbol and μ ∈ T is an annotation constant, then p : μ is an annotated atomic formula (atom) ; 2. if F, F1 , F2 are formulas, then ¬F, F1 ∧ F2 , F1 ∨ F2 , F1 → F2 are formulas. We suppose that the four-valued lattice in Fig.1 is the complete lattice T , where annotation t may be intuitively regarded as the truth value true and annotation f as the truth value false. It may be comprehensible that annotations ⊥, t, f and correspond to the truth values ∗, T, F and T F in Visser[50] and None, T, F, and Both in Belnap[4], respectively. Moreover, the complete lattice T can be viewed as a bi-lattice in which the vertical −→ direction ⊥ indicates knowledge amount ordering and the horizontal direc− → tion ft does truth ordering [10]. We use the symbol ≤ to denote the ordering −→ in terms of knowledge amount (the vertical direction ⊥) over the complete lattice T , and the symbols ⊥ and are used to denote the bottom and top

78

K. Nakamatsu (inconsistent) @ @ f (false) t (true) @ @ ⊥(unknown) Fig. 1. The 4-valued Complete Lattice T

elements, respectively. In the paraconsistent annotated logic P T , each annotated atomic formula can be interpreted epistemically, for example, p : t may be interpreted epistemically as “the proposition p is known to be true”. There are two kinds of negation in the paraconsistent annotated logic P T , one of them, represented by the symbol ¬ in Definition 2.1, is called epistemic negation, and the epistemic negation in P T followed by an annotated atomic formula is defined as a mapping between elements of the complete lattice T as follows : ¬(⊥) = ⊥,

¬(t) = f,

¬(f) = t,

¬() = .

This definition shows that the epistemic negation maps annotations to themselves without changing the knowledge amounts of the annotations, and the epistemic negation followed by an annotated atomic formula can be eliminated by syntactical operation. For example, the knowledge amount of annotation t is the same as that of annotation f as shown in the complete lattice T , and we have the epistemic negation ¬(p : t) = p : ¬(t) = p : f 1 . which shows that the knowledge amount in terms of the proposition p cannot be changed by the epistemic negation. There is another negation called ontological(strong) negation that is defined by the epistemic negation. Definition 2.2 (Strong Negation) Let F be any formula, ∼ F =def F → ((F → F ) ∧ ¬(F → F )). The epistemic negation in the above definition is not interpreted as a mapping between annotations since it is not followed by an annotated atomic formula. Therefore, the strongly negated formula ∼ F is intuitively interpreted so that if the formula F exists, the contradiction ((F → F ) ∧ ¬(F → F )) is implied. Usually, strong negation is used for denying the existence of the formula following it. The semantics for the paraconsistent annotated logics P T is defined as follows. 1

An expression ¬p : µ is conveniently used for expressing a negative annotated literal instead of ¬(p : µ) or p : ¬(µ).


79

Definition 2.3 Let ν be the set of all propositional symbols and F be the set of all formulas. An interpretation I is a function, I : ν −→ T . To each interpretation I, we can associate the valuation function such that vI : F −→ {0, 1}, which is defined as : 1. let p be a propositional symbol and μ an annotation, vI (p : μ) = 1 iff μ ≤ I(p), I(p) ; vI (p : μ) = 0 iff μ ≤ 2. let A and B be any formulas, and A not an annotated atom, vI (¬A) = 1 iff vI (A) = 0, vI (∼ B) = 1 iff vI (B) = 0 ; other formulas A → B, A ∧ B, A ∨ B are valuated as usual. We provide an intuitive interpretation for strongly negated annotated atom with the complete lattice T . For example, the strongly negated literal ∼ (p : t) implies the knowledge “p is false(f) or unknown(⊥)” since it denies the existence of the knowledge that “p is true(t)”. This intuitive interpretation is proofed by Definition 2.3 as follows : if vI (∼ (p : t)) = 1, we have vI (p : t) = 0 and for any annotation μ ∈ {⊥, f, t, } ≤ t, we have vI (p : μ) = 1, therefore, we obtain that μ = f or μ = ⊥. 2.2

EVALPSN (Extended Vector Annotated Logic Program with Strong Negation)

Generally, an annotation is explicitly attached to each literal in paraconsistent annotated logic programs as well as the paraconsistent annotated logic P |calT . For example, let p be a literal, μ an annotation, then p : μ is called an annotated literal. The set of annotations constitutes a complete lattice. An annotation in EVALPSN has a form of [(i, j), μ] called an extended vector annotation. The first component (i, j) is called a vector annotation and the set of vector annotations constitutes a complete lattice, Tv (n) = { (x, y)|0 ≤ x ≤ n, 0 ≤ y ≤ n, x, y and n are integers } shown by the Hasse’s diagram as n = 2 in Fig.2. The ordering( v ) of the complete lattice Tv (n) is defined as follows : let (x1 , y1 ), (x2 , y2 ) ∈ Tv (n),

80

K. Nakamatsu (2, 2) q P P @ (2, 1) PP ∗1 ∗3 P (1, 2) q P q @ PP α P @(1, 1) @ P @q ∗2 @q (2, 0) (0, 2) q @ @ @q γP @q β PP (0, 1) @ (1, 0) P P @q ⊥ (0, 0)

Fig. 2. Lattice Tv (2) and Lattice Td

(x1 , y1 ) v (x2 , y2 ) iff x1 ≤ x2 and y1 ≤ y2 . For each extended vector annotated literal p : [(i, j), μ], the integer i denotes the amount of positive information to support the literal p and the integer j denotes that of negative one. The second component μ is an index of fact and deontic notions such as obligation, and the set of the second components constitutes the following complete lattice, Td = {⊥, α, β, γ, ∗1 , ∗2 , ∗3 , }. The ordering( d ) of the complete lattice Td is described by the Hasse’s diagram in Fig.2. The intuitive meanings of all members in Td are ⊥(unknown), α(fact), β(obligation), ∗1 (fact and obligation),

γ(non-obligation),

∗2 (obligation and non-obligation), ∗3 (fact and non-obligation), and (inconsistency). −→ The complete lattice Td is a quatro-lattice in which the direction ⊥ mea− → −−→ sures knowledge amount, the direction γβ deontic truth, the direction ⊥∗2 −→ deontic knowledge amount and the direction ⊥α factuality. For example, the annotation β(obligation) can be intuitively interpreted to be more obligatory than the annotation γ(non-obligation), and the annotations ⊥(no knowledge) and ∗2 (obligation and non-obligation) are deontically neutral, that is to say, it cannot be said whether they represent obligation or non-obligation. A complete lattice Te (n) of extended vector annotations is defined as the product, Tv (n) × Td . The ordering( e ) of the complete lattice Te (n) is also defined as follows : let [(i1 , j1 ), μ1 ], [(i2 , j2 ), μ2 ] ∈ Te , [(i1 , j1 ), μ1 ] e [(i2 , j2 ), μ2 ] iff

(i1 , j1 ) v (i2 , j2 ) and μ1 d μ2 .


81

There are two kinds of epistemic negation (¬1 and ¬2 ) in EVALPSN, which are defined as mappings over the complete lattices Tv (n) and Td , respectively. Definition 2.4(epistemic negations ¬1 and ¬2 in EVALPSN) ¬1 ([(i, j), μ]) = [(j, i), μ], ∀μ ∈ Td ¬2 ([(i, j), ⊥]) = [(i, j), ⊥],

¬2 ([(i, j), α]) = [(i, j), α],

¬2 ([(i, j), β]) = [(i, j), γ], ¬2 ([(i, j), γ]) = [(i, j), β], ¬2 ([(i, j), ∗1 ]) = [(i, j), ∗3 ], ¬2 ([(i, j), ∗2 ]) = [(i, j), ∗2 ], ¬2 ([(i, j), ∗3 ]) = [(i, j), ∗1 ], ¬2 ([(i, j), ]) = [(i, j), ]. If we regard the epistemic negations in Definition 2.4 as syntactical operations, the epistemic negations followed by literals can be eliminated by the syntactical operations. For example, ¬1 p : [(2, 0), α] = p : [(0, 2), α] and ¬2 q : [(1, 0), β] = p : [(1, 0), γ]. The strong negation (∼) in EVALPSN is defined as well as the paraconsistent annotated logic P T . Definition 2.5 (well extended vector annotated literal) Let p be a literal. p : [(i, 0), μ] and p : [(0, j), μ] are called weva(well extended vector annotated)-literals, where i, j ∈ {1, 2, · · · , n}, and μ ∈ { α, β, γ }. Defintion 2.6 (EVALPSN) If L0 , · · · , Ln are weva-literals, L0 ← L1 ∧ · · · ∧ Li ∧ ∼ Li+1 ∧ · · · ∧ ∼ Ln is called an EVALPSN clause. An EVALPSN is a finite set of EVALPSN clauses. Fact and deontic notions, “obligation”, “forbiddance” and “permission” are represented by extended vector annotations, [(m, 0), α], [(m, 0), β], [(0, m), β], and [(0, m), γ], respectively, where m is a positive integer. For example, p : [(2, 0), α] is intuitively interpreted as “it is known to be true of strength 2 that p is a fact”; p : [(1, 0), β] is as “it is known to be true of strength 1 that p is obligatory”; p : [(0, 2), β] is as “it is known to be false of strength 2 that p is obligatory”, that is to say, “it is known to be true of strength 2 that p is forbidden”; p : [(0, 1), γ] is as “it is known to be false of strength 1 that p is not obligatory”, that is to say, “it is known to be true of strength 1 that p is permitted”. Generally, if an EVALPSN contains the strong negation ∼, it has stable model semantics [40] as well as other ordinary logic programs with strong negation. However, the stable model semantics may have a problem that some programs may have more than two stable models and others have no stable model.

82

K. Nakamatsu

Moreover, computing stable models takes a long time compared to usual logic programming such as PROLOG programming. Therefore, it does not seem to be so appropriate for practical application such as real time processing in general. However, we fortunately have cases to implement EVALPSN practically, if an EVALPSN is a stratified program, it has a tractable model called a perfect model [45] and the strong negation in the EVALPSN can be treated as the Negation as Failure in logic programming with no strong negation. The details of stratified program and some tractable models for normal logic programs can be found in [3, 11, 45, 47], furthermore the details of the stratified EVALPSN are described in [40]. Therefore, inefficient EVALPSN stable model computation does not have to be taken into account in practice since all EVALPSNs that will appear in the subsequent sections are stratified.

3 Before-after EVALPSN In this section, we define bf(before-after)-EVALPSN formally and introduce how to implement it aiming at its application to the real-time safety verification for process order control. 3.1

Before-after Relation in EVALPSN

First of all, we introduce a special literal R(pi, pj, t) whose vector annotation represents the before-after relation between processes P ri (pi) and P rj (pj) at time t, which may be regarded as time intervals in general, and the literal R(pi, pj, t) is called a bf-literal 2 . Definition 3.1(bf-EVALPSN) An extended vector annotated literal R(pi , pj , t) : [μ1 , μ2 ] is called a bf-EVALP literal, where μ1 is a vector annotation and μ2 ∈ {α, β, γ}. If an EVALPSN clause contains bf-EVALP literals, it is called a bf-EVALPSN clause or just a bf-EVALP clause if it contains no strong negation. A bf-EVALPSN is a finite set of bf-EVALPSN clauses. We provide some paraconsistent interpretations of vector annotations for representing bf-relations, which are called bf-annotations. Strictly speaking, bf-relations between time intervals are classified into 15 kinds according to bf-relations between start/finish times of two time intervals. We define the 15 kinds of bf-relations in bf-EVALPSN with regarding processes as time intervals. Suppose that there are two processes, P ri with its start/finish times xs and xf , and P rj with its start/finish times ys and yf . 2

Hereafter, the expression “before-after” is abbreviated as just “bf” in this chapter.

Paraconsistent Annotated Logic Program Before-after EVALPSN xs

83

P ri ys

P rj

Fig. 3. Bf-relations, Before/After

xs

P ri

xf -

ys

P rj

yf -

Fig. 4. Bf-relations, Disjoint Before/After

Before (be)/After (af) Firstly, we define the most basic bf-relations before/after according to the bf-relation between each start time of two processes, which are represented by bf-annotations be/af, respectively. If one process has started before/after another one started, then the bf-relations between those processes are defined as “before(be)/after(af)”, respectively. The bf-relations also are described in Fig.3 with the condition that process P ri has started before process P rj starts. The bf-relation between their start/finish times is denoted by the inequality {xs < ys } 3 . For example, a fact at time t “process P ri has started before process P rj started” can be represented by a bf-EVALP clause R(pi, pj, t) : [be, α]. The bf-relations before/after do not care when the two processes finish. Disjoint Before (db)/After (da) Bf-relations disjoint before/after between processes P ri and P rj are represented by bf-annotations db/da, respectively. The expressions “disjoint before/after” imply that there is a timelag between the earlier process finish and the later one start. They also are described in Fig.4 with the condition that process P ri has finished before process P rj starts. The bf-relation between their start/finish times is denoted by the inequality {xf < ys }. For example, an obligation at time t “process P ri must start after process P rj finishes” can be represented by a bf-EVALP clause, R(pi, pj, t) : [da, β]. Immediate Before (mb)/After (ma) Bf-relations immediate before/after between the processes P ri and P rj are represented by bf-annotations mb/ma, respectively. The expressions “immediate before/after” imply that there is no timelag between the earlier process 3

If time t1 is earlier than time t2 , we conveniently denote the relation by the inequality t1 < t2 .

84

K. Nakamatsu P ri xs

ys xf

P rj -yf

Fig. 5. Bf-relations, Immediate Before/After xs

P ri ys

xf -

P rj

yf -

Fig. 6. Bf-relations, Joint Before/After

finish time and the later one start time. The bf-relations also are described in Fig.5 with the condition that process P ri has finished immediately before process P rj starts. The bf-relation between their start/finish times is denoted by the equality {xf = ys }. For example, a fact at time t “process P ri has finished immediately before process P rj starts” can be represented by a bf-EVALP clause, R(pi, pj, t) : [mb, α]. Joint Before (jb)/After (ja) Bf-relations joint before/after between processes P ri and P rj are represented by bf-annotations jb/ja, respectively. The expressions “joint before/after” imply that the two processes overlap and the earlier process has finished before the later one finishes. The bf-relations also are described in Fig.6 with the condition that process P ri has started before process P rj starts and process P ri has finished before process P rj finishes. The bf-relation between their start/finish times is denoted by the inequalities {xs < ys < xf < yf }. For example, a fact at time t “process P ri has started before process P rj starts and finished before process P rj finishes” can be represented by a bf-EVALP clause, R(pi, pj, t) : [jb, α]. S-included Before (sb), S-included After (sa) Bf-relations s-included before/after between processes P ri and P rj are represented by bf-annotations sb/sa, respectively. The expressions “s-included before/after” imply that one process has started before another one starts and they have finished at the same time. The bf-relations also are described in Fig.7 with the condition that process P ri has started before process P rj starts and they have finished at the same time. The bf-relation between their start/finish times is denoted by the equality and inequalities {xs < ys < xf = yf }. For example, a fact at time t “process P ri has started before process P rj starts and they have finished at the same time” can be represented by a bf-EVALP clause, R(pi, pj, t) : [sb, α].

Paraconsistent Annotated Logic Program Before-after EVALPSN xs

xf -

P ri ys

85

P rj

-

yf

Fig. 7. Bf-relations, S-included Before/After xs

xf -

P ri ys

P rj - yf

Fig. 8. Bf-relations, Included Before/After

Included Before (ib)/After (ia) Bf-relations included before/after between processes P ri and P rj are represented by bf-annotations ib/ia, respectively. The expressions “included before/after” imply that one process has started/finished before/after another one starts/finished, respectively. The bf-relations also are described in Fig.8 with the condition that process P ri has started before process P rj starts and finished after process P rj finished. The bf-relation between their start/finish times is denoted by the inequailies {xs < ys , yf < xf }. For example, an obligation at the time t “process P ri must start before process P rj starts and finish after process P rj finishes” can be represented by a bf-EVALP clause, R(pi, pj, t) : [ib, β]. F-included Before (fb)/After (fa) The bf-relations f-include before/after between processes P ri and P rj are represented by bf-annotations fb/fa, respectively. The expressions “f-included before/after” imply that the two processes have started at the same time and one process has finished before another one finishes. The bf-relations also are described in Fig.9 with the condition that processes P ri and P rj have started at the same time and process P ri has finished after process P rj finished. The bf-relation between their start/finish times is denoted by the equality and inequality {xs = ys , yf < xf }. For example, a fact at time t “processes P ri and P rj have started at the same time and process P ri has finished after process P rj finished” can be represented by a bf-EVALP clause, R(pi, pj, t) : [fa, α]. xs ys

P ri P rj

xf -

yf

Fig. 9. Bf-relations, F-included Before/After

86

K. Nakamatsu xs ys

P ri

xf -

P rj

-

yf

Fig. 10. Bf-relation, Paraconsistent Before-after

Paraconsistent Before-after (pba) Bf-relation paraconsistent before-after between processes P ri and P rj is represented by bf-annotation pba. The expression “paraconsistent before-after” implies that the two processes have started at the same time and also finished at the same time. The bf-relation is also described in Fig.10 with the condition that processes P ri and P rj have not only started but also finished at the same time. The bf-relation between their start/finish times is denoted by the equalities {xs = ys , yf = xf }. For example, an obligation at time t “processes P ri and P rj must not only start but also finish at the same time” can be represented by a bf-EVALP clause, R(pi, pj, t) : [pba, β]. Here we define the epistemic negation ¬1 that maps bf-annotations to themselves in bf-EVALPSN. Definition 3.2 (Epistemic Negation ¬1 for Bf-annotations) The epistemic negation ¬1 over the set of bf-annotations, {be, af, da, db, ma, mb, ja, jb, sa, sb, ia, ib, fa, fb, pba} is obviously defined as the following mapping : ¬1 (af) = be, ¬1 (da) = db, ¬1 (ma) = mb, ¬1 (ja) = jb,

¬1 (be) = af, ¬1 (db) = da,

¬1 (sa) = sb, ¬1 (ia) = ib,

¬1 (sb) = sa, ¬1 (ib) = ia,

¬1 (fa) = fb,

¬1 (fb) = fa,

¬1 (mb) = ma, ¬1 (jb) = ja,

¬1 (pba) = pba. If we consider the before-after measure over the 15 bf-annotations, obviously there exists a partial order( p+ ), we can solve p+ (a) and q+/+ (a) using p+ (4a) and q+/+ (4a), the prediction for figure 2B is p+ (a) = 0.4129 and q+/+ (a) = 0.7418. Because fine-scale binary (presence-absence) data are more information rich

A Bayesian Solution to the Modifiable Areal Unit Problem

185

than coarse-scale data [69], scaling-up predictions of occupancy are bound to be more accurate than those obtained by scaling-down. Other tests using Drosophilidae species across the mesocosm arena also confirm the prediction capacity of equations (4) and (5) [6]. Thus, the BYE provides a preliminary solution to the scale problem in the MAUP. The Aggregation Problem For the transect shape grain as in figure 1C and figure 2D, we can calculate its probability for absence p0 (4a) and the conditional probability q0/0 (4a) according to the BYE [6], p0 (4a) = p0 (a) × q0/0 (a)3

(12)

q0/0 (4a) = q0/0 (a) × b0 (a)3

(13)

q0/0 (4a) = q0/0 (a)

(14)

4

where equation (7) is the probability adjacent to the long edge and equation (8) is to the short edge. Using the similar procedure, we can have,

Aggregation

0

Segregation Random

Unfeasible region

0

Fig. 4. The contour plot of the difference between the occupancy with the transect grain (as in figure 1C) and the one with the square grain (as in figure 1B), p+ (4a)transect − p+ (4a)square . The unfeasible region indicates 2 − 1/p+ > q+/+ ; the aggregation region indicates p+ < q+/+ ; the segregation region indicates p+ > q+/+ , random indicates the line that p+ = q+/+ .

186

C. Hui

p+ (4a) = 1 −

(1 − 2p+ (a) + p+ (a) × q+/+ (a))3 (1 − p+ (a))2

(15)

The formula for q+/+ (4a) were not shown here for conciseness but can easily be obtained using equations (7) and (8). According to equation (9), we have the occupancy for figure 2D, p+ (4a) = 0.6823, which is comparable to the observation (p+ (4a) = 0.65). A quick test of the spatial distribution pattern can be executed, by comparing the occupancy in a chessboard (as in figure 1B) with the one in a transect grain map (as in figure 1C). We find that the occupancy will be larger in a longer-perimeter sample if the spatial distribution is aggregated but will be smaller if it is segregated in samples with similar sample area or grain (see figure 4; [6]). In a real case, De Grave and Casey [70] reported the variability in density estimates of intertidal, infaunal benthos due to the shape of sampling devices. They found that the density for most intertidal macrofauna is lower in rectangular shaped samples compared to the density observed in square shaped samples with similar area, but the situation was inversed for Bathyporeia guilliamsoniana Bate. According to the above analysis, the reason might be that most species are aggregated in space while B. guilliamsoniana has a segregated distribution due to its high mobility. Therefore, the BYE further provides the solution for the aggregation (zoning) problem in the MAUP. 2.5

Scaling Patterns

As demonstrated above, this iterative formula of the Bayesian solution calculates the species occupancy and spatial correlation after combining four neighboring cells into one new larger grain, and thus fits the understanding of species scaling pattern as a percolation process [71, 72]. Criticism of the formula mainly concerns the difficulty in calculation [73], as well as its restriction of linking species distributions in a grain of a only to those in a grain of 4a, i.e. its discrete essence. This section is intended to advance the understanding of the scaling pattern of species distribution based on the same rationale behind this Bayesian estimate. This will be achieved by generating, through induction, new formulae of species occupancy and spatial correction that rely on scale (grain size) as the only independent variable. Intriguingly, this formulae not only simplifies the Bayesian estimate, but also generates the shapes of species scaling patterns that are consistent with those from the intraspecific occupancy-abundance relationship (OAR) and the area-ofoccupancy (AOO) models. OAR describes the positive correlation between the abundance and occupancy of a species over time or across regions, is one of the most widely used descriptions of species distribution patterns [74, 72]. AOO has been found to obey a log-log linear relationship across sample scales (grain sizes) [72]. Such a log-log linear AOO provides an instant link to the box-counting fractal dimension [72], and therefore has been argued to reveal the scale-invariant nature of the species distribution.


ª Ui, j « « p « U i 1, j ¬

o

U i , j 1 º o ª U i , j 2

» » U i 1, j 1 »¼

« « p « U i 1, j 2 ¬

o

187

U i , j 3 º

» » U i 1, j 3 »¼

Fig. 5. The procedure of calculating the probability of absence in a (2× 2)-cell (left block), p0 (2 × 2), and the conditional probability of a randomly chosen neighbor of this (2 × 2)-cell being also absent, q0/0 (2 × 2). Black arrows indicate conditional probabilities; white arrows indicate Bayesian estimates.

A spatial (presence-absence) map can be expressed by a binary matrix, M =< ρi,j >m×m , with the element ρi,j being either + or 0, indicating the state of the cell (i, j) as either occupied or empty respectively. The dimensionality of the matrix, m × m, can normally be considered as infinite. Now if we combine n×n cells together to form a new cell, the binary matrix will become M =< ρi,j >m/n×m/n , with ρi,j = 1 indicating that at least one previous cell (or a sub-cell) is occupied and ρi,j = 0 indicating that all n × n previous cells are empty. A solution to the spatial scaling of species distribution is to calculate the global density (occupancy) and the local density (spatial correlation) for various grain sizes. it can be achieved by first calculating the probability of absence p0 (n × n) and q0/0 (n × n), and then using the relationships between those probabilities, p+ = 1 − p0 and q+/+ = 1 − (1 − q0/0 )p0 /(1 − p0 ) to calculate p+ (n × n) and q+/+ (n × n). For n = 2, given that the probability of a randomly selected cell being empty (ρi,j = 0) is p0 , the probability of finding two empty neighbors 2 (ρi,j+1 = 0 and ρi+1,j = 0) is q0/0 and the probability that a shared neighbor of these two absent cells is also empty (ρi+1,j+1 = 0) be denoted as b0 , we can 2 2 easily obtain p0 (2 × 2) = p0 × q0/0 × b0 and q0/0 (2 × 2) = q0/0 × b20 according to the diagram in figure 5. For n ≥ 3, probabilities p0 (n × n) and q0/0 (n × n) can be calculated according to different procedures of combining n × n cells in forming a new larger grain. This is essentially similar to the mathematical problem of the seven bridges of Konigsberg. For example when n = 3, it is possible to calculate p0 (3 × 3) and q0/0 (3 × 3) using a spiral method of adding neighbors together to form a larger grain (figure 6A). Accordingly, we have 6 6 p0 (3 × 3) = p0 × q0/0 × b0 × g0 and q0/0 (3 × 3) = q0/0 × b0 × k0 × g0 , where g0 is the probability that a shared neighbor of four absent cells is empty, and k0 is the probability that a shared neighbor of three absent cells is empty. These two extra probabilities, i.e. g0 and k0 , also need to be estimated in this spiral procedure of scaling-up. Notably, this is not the only procedure to combine nine cells together in forming larger grains. In figure 6B, another procedure is presented for calculating the probability of p0 (3 × 3) and q0/0 (3 × 3), which 4 4 gives p0 (3 × 3) = p0 × q0/0 × b40 and q0/0 (3 × 3) = q0/0 × b40 × k0 . The number of prcedures of calculating p0 (n × n) and q0/0 (n × n) becomes extremely large for n >> 3 (similar to the number of Euler walks). To solve this problem

188

C. Hui A

Ui , j

U i , j1

p

p

n

U i1, j

Ui1, j 1

U i1, j 2

m

Ui , j 2

p

U i2, j

n o Ui 2, j 1 o U i 2, j 2

B

Ui , j U i1, j p

U i2, j

o m

U i , j 1

Ui , j 2

p

n

Ui1, j 1

U i1, j 2

n o Ui 2, j 1 o U i 2, j 2

C

Ui , j

U i , j 1

p

n

p

U i1, j

Ui1, j 1

U i1, j 2

p

U i2, j

o

Ui , j 2

n

p

o Ui 2, j 1

U i 2 , j 2

Fig. 6. The procedures of calculating the probability of absence in a (3 × 3)-cell. Black arrows indicate the order of adding sub-cells using, A a spiral procedure, B an alternative procedure, and C a zigzag procedure. see text for detail.

and also to avoid calculating extra probabilities, a procedure is used here for calculating p0 (n × n) and q0/0 (n × n) according to a zigzag method of adding cells together. For example, following this zigzag procedure (figure 6C), we 4 3 have p0 (3 × 3) = p0 × q0/0 × b40 and q0/0 (3 × 3) = q0/0 × b60 . After examining several more cases (e.g. for n = 4 and 5), formulae can easily be induced for the general case: 2(n−1)

p0 (n × n) = p0 × q0/0

(n−1)2

× b0

n(n−1)

n q0/0 (n × n) = q0/0 × b0

.

,

(16) (17)

Since n × n indicates the size of the new grain (denoted as a), we have the scaling pattern of absence, p0 (a) = ba0 (

p0 × b0 q0/0 2a1/2 )( ) , 2 q0/0 b0

(18)


189

q0/0 a1/2 ) . (19) b0 Accordingly, we can simply get the following scaling patterns of occupancy and spatial correlation, q0/0 (a) = ba0 (

1/2

p+ (a) = 1 − θ × β 2a

1/2

(θ−1 × β −a

× δa,

(20)

− 1)(1 − p+ (a))2 , (21) p+ (a) 2 where θ = p0 × b0 /q0/0 , β = q0/0 /b0 and δ = b0 are model parameters. This scaling pattern describes species occupancy and spatial correlation as a function of the spatial scale a, and elucidates the percolation process of non-random structure when scaling up. The scaling patterns of species distribution were illustrated in figure 7. First, if we classify the spatial structure into three categories: aggregation (q+/+ > p+ ), randomness (q+/+ = p+ ) and segregation (q+/+ < p+) [6, 20], the category of spatial structure will not change across categories with change of scales (figure 7A), i.e. it remains in the same category. With the increase of grain a, occupancy p+ (a) increases monotonously, whereas the spatial correlation q+/+ (a) decreases at first but then increases ultimately at the same rate of p+ (a), i.e. it converges to randomness. The watershed threshold for q+/+ (a) changing from declining to increasing [the dashed line in figure 7A] can not be solved analytically, but an approximate numerical solution can 2 be obtained as an ellipse, (p+ − 1)2 /1.142 + q+/+ /0.892 = 1, after a rotation of π/4. This threshold is also insensitive to the initial values of p0 and q0/0 . It is worth noting that the spatial correlation is not perfectly synchronized with the changing rate of occupancy dp+ (a)/da; however generally speaking, a lower spatial correlation indeed corresponds with a higher changing rate of occupancy, i.e. the scale-dependence of occupancy becomes strong when the spatial correlation is weak [6]. Second, the overall saturated-curve-shape and the S-shape (especially obvious for those highly aggregated patterns, q+/+ >> p+ ) of occupancy scaling are consistent with the results from OAR models [72]. Furthermore, if the spatial structure is random, it is easy to obtain that p+ (a) = 1 − e−λ×a (where λ = −ln(δ)), which is actually the Poisson OAR for randomness (e.g. [74, 72]). Finally, a quasi-power law holds for the scaling pattern of occupancy over about two orders of scale magnitude (figure 7B), which is consistent with a log-log linear AOO as reported in literature [75]. However, such a quasi-power law does not reflect a strict self-similar nature of species distributions. A further test on the spatial structure reveals the scale-dependence of spatial correlation (figure 7C). A power law form of AOO does not explain how those occupied cells distribute, and, therefore, does not necessarily lead to the scale-invariance (self-similarity) of the spatial distribution, i.e. the fractal objects can always lead to a power law scaling, but not vice verse. The approach presented here facilitates accurate, cost-efficient estimation of occupancy, and provides a comprehensive approach towards modelling species distributions. q+/+ (a) = p+ (a) +

190

C. Hui

A

B

C

Fig. 7. Scaling patterns of species occupancy and spatial correlation. A, Several trajectories of p+ (a) and q+/+ (a) in a parametric plot, which all lead to the topright with the increase of grain a. The dashed line indicates the watershed threshold of q+/+ (a) where it changes from declining to increasing with the increase of grain a. B, Scaling patterns of occupancy, with p+ = 0.01 and q+/+ = 0, 0.1, 0.2, ..., 0.9 (from top to bottom curves). C , Scaling patterns of spatial correlation, with the same initial values as in B.

3 Conclusion From the above analysis, the following propositions are brought forward. (1: For the scale problem) With the increase of grain size, the occupancy p+ and the spatial correlation q+/+ both increase. The BYE can largely explain the trajectories of these two variables across scales. (2: For the scale problem) The accuracy of prediction of the BYE for scaling-up is higher than for scalingdown; the accuracy of prediction of the BYE for occupancy, p+ (first-order description of the spatial distribution), is higher than for the spatial correlation, q+/+ (second-order description of the spatial distribution). (3: For the aggregation/zoning problem) Under the same grain size, the occupancy prediction is sensitive to irregularity (or length of the perimeter) as well as the


191

spatial sturcture (the intensity of aggregation, randomness and segregation; e.g. those described by the join-count statistics). If the spatial structure is aggregated (p+ < q+/+ ), occupancy estimates with longer perimeter grain will have higher values. If the spatial structure is random (p+ = q+/+ ), occupancy estimates will not be affected by the shape of the grain. If the spatial structure is segregated (p+ > q+/+ ), occupancy estimates with longer perimeter grains will have higher values of occupancy than those with shorter perimeter grain. Thus, the BYE provide a preliminary solution to the presence-absence format MAUP. The above analysis shows that it will be possible in the near future to provide a comprehensive solution to the MAUP. Studies on the scaling pattern of species distribution present a framework to the scale problem in MAUP. Hui and McGeoch [72] have reviewed different occupancy-abundance models as well as the scaling pattern of occupancy. Because such essential links between presence-absence data and the abundance (or the intensity) of the spatial patterns, those more general models, slotting them into the BYE framework, will eventually resolve the scale problem of the MAUP. Moreover, the solution to the scale problem will also benefit conservation science. Kunin [75] gave a log-log linear area-of-occupancy (i.e. the relationship between occupancy and grain size), which has been used to predict the biodiversity trend [76]. Hartley and Kunin [77] suggested that the solution to the scale problem can also be used to estimate the abundance of focal species based only on presence-absence data, or even on the occupancy alone. Such techniques will surely improve the efficiency of conservation management. Furthermore, He and Hubbell [71] have explored the effect of perimeter length of grain on the occupancy and abundance estimates. Their findings are consistent with the analysis of the aggregation problem here in the MAUP. The solution to the aggregation problem will help us to understand ecological fallacy and sampling artifacts. In conclusion, the BYE together with those advances will contribute to building a comprehensive solution to the Modifiable Areal Unit Problem.

4 Future Directions Research should foucs on the following with regard to solving the MAUP in the spatial analysis. First, a scale-free index for the description of spatial patterns is urgently needed. Up to now, as outlined in this chapter, spatial statistics and indices are largely scale-dependent. Even though a few indices, such as Shannon’s information and entropy index, have proclaimed to be scale-free, this is questionable under stingent testing. Furthermore, although specific scale-dependent statistics and indices might be especially valuable for case studies, they contribute little to solving the scale problem as a whole. Second, more effort should be put into finding proper measurements of irregular shapes (e.g. the width-to-length ratio here). Studies have revealed the strong correlation between the grain irregularity and the intensity of the

192

C. Hui

zoning effect [6, 71]. Fractality might shed light on this issue as its initial intention is to describe the irrigularity of fractal, self-similar objects. Finally, the MAUP is a push forward for unveiling the spatial character of natural systems rather than simply a problem. Fotheringham [78] has suggested the shifting of spatial analysis towards relationships that focus on rates of change (see also [24]): “Can we acquire information on the rate of change in variables and relationships of interest with respect to scale?” This is exactly the focus of this study, i.e. seeking the amount of change when the grain shifts from a to 4a. Furthermore, Hui and McGeoch [72] indicated that the scale problem is eventually a percolation process. Linking with knowledge from other nonlinear sciences will surely help to overcome the MAUP in spatial analysis. Suprisingly, the analysis in this chapter can also be used in the analysis of co-variance and association, which is not part of the MAUP. Species association belongs to the compositional variance, measuring the degree of co-occurrence and co-distribution of two species in samples [3, 5], the scaling pattern of which is closely linked with the mechnaism behind beta diversity and species-area relationship [3, 4, 5, 79]. For instance, considering two species, there exist four scenarios for a randomly chosen cell: species A and B coexist, PA∩B (a) (also called the joint occupancy); only species A occurs, PA∩B (a); only species B occurs, PA∩B (a); neither exists, PA∩B (a). Note that PA (a) and PB (a) have the same meanings as p+ (a)A and p+ (a)B , respectively. Similarly, we can define a postive association of species A and B as PA∩B (a) > PA (a) × PB (a), a negative association as PA∩B (a) < PA (a) × PB (a), and the independence of the distribution of these two species as PA∩B (a) = PA (a) × PB (a). This definition is also well consisitent with the study of null models for species co-occurrence [80, 81]. Therefore, the same framework developed in this chapter can also be used to calculate the scaling pattern of association and co-variance. Further exploration of the inclusion of spatial autocorrelation structure into spatial models should be productive. Indeed, the fields of spatial analysis [12, 43] and modelling [58] have developed largely independently to date. Integration of these two approaches, as demonstrated in this chapter, are likely to result in significant advances towards the development of a general spatial framework for understanding the non-randomness phenomena in nature, and surely warrants further attention.

References 1. Levin, S.A.: The problem of pattern and scale in ecology. Ecology 73, 1943–1967 (1992) 2. Plimak, L.I., Walls, D.F.: Nonclassical spatial and momentum distributions in a Bose-condensed gas. Phys. Rev. A 54, 652–655 (1996) 3. Whittaker, R.H.: Evolution and measurement of species diversity. Taxon 21, 213–251 (1972) 4. Hui, C.: On species-area and species accumulation curves: a comment on Chong and Stohlgren’s index. Biol. Indic. 8, 327–329 (2008)


193

5. Hui, C., McGeoch, M.A.: Does the self-similar species distribution model lead to unrealistic predictions? Ecology 89, 2946–2952 (2008) 6. Hui, C., McGeoch, M.A., Warren, M.: A spatially explicit approach to estimating species occupancy and spatial correlation. J. Anim. Ecol. 75, 140–147 (2006) 7. Openshaw, S.: The modifiable areal unit problem. Geo Books, Norwick (1984) 8. Unwin, D.J.: GIS, spatial analysis and spatial statistics. Prog. Human Geogr. 20, 540–551 (1996) 9. Burger, O., Todd, L.: Grain, extent, and intensity: the components of scale in archaeological survey. In: Lock, G., Molyneaux, B.L. (eds.) Confronting scale in archaeological: issues of theory and practice, pp. 235–255. Springer, New York (2006) 10. Taylor, L.R.: Aggregation, variance and the mean. Nature 189, 732–735 (1961) 11. Ratcliffe, J.H., McCullagh, M.J.: Hotbeds of crime and the search for spatial accuracy. Geogr. Sys. 1, 385–395 (1999) 12. Wiens, J.A.: Ecological heterogeneity: ontogeny of concepts and approaches. In: Hutchings, M.J., Jones, E.A., Stewart, A.J.A. (eds.) The ecological consequences of environmental heterogeneity, pp. 9–31. Blackwell Science, Oxford (2000) 13. Li, H., Reynolds, J.F.: On definition and quantification of heterogeneity. Oikos 73, 280–284 (1995) 14. Burrough, P.A., McDonnell, R.A.: Principles of geographical information systems. Oxford Univ. Press, Oxford (1998) 15. Perry, J.N., Liebhold, A.M., Rosenberg, M.S., Dungan, J.L., Miriti, M., Jakomulska, A., Citron-Pousty, S.: Illustrations and guidelins for selecting statistical methods for quantifying spatial pattern in ecological data. Ecography 25, 578– 600 (2002) 16. Dungan, J.L., Perry, J.N., Dale, M.R.T., Legendre, P., Citron-Pousty, S., Fortin, M.J., Jakomulska, A., Miriti, M., Rosenberg, M.S.: A balanced view of scale in spatial statistical analysis. Ecography 25, 626–640 (2002) 17. Anselin, L.: Local indicators of spatial association. Geogr Analysis 27, 93–116 (1995) 18. Hui, C.: Crossing the borders of spatial analysis and modelling: a rethink. In: Kelly, J.T. (ed.) Progress in Mathematical Biology Research, pp. 170–197. Nova Science, Hauppauge (2008) 19. Sato, K., Iwasa, Y.: Pair approximation for lattice-based ecological models. In: Dieckmann, U., Law, R., Metz, J.A.J. (eds.) The geometry of ecological interactions: simplifying spatial complexity, pp. 341–359. Cambridge Univ Press, Cambridge (2000) 20. Hui, C., Li, Z.: Distribution patterns of metapopulation determined by Allee effects. Popul. Ecol. 46, 55–63 (2004) 21. Fortin, M.J., Dale, M.R.T., ver Hoef, J.: Spatial analysis in ecology. In: ElShaarawi, A.H., Piegorsch, W.W. (eds.) Encyclopedia of environmentrics, pp. 2051–2058. Wiley and Sons, New York (2002) 22. Hui, C., McGeoch, M.A.: Spatial patterns of prisoner’s dillema game in metapopulations. Bull. Math. Biol. 69, 659–676 (2007) 23. Gehlke, C., Biehl, K.: Certain effects of grouping upon the size of the correlation coefficient in census tract material. J. Am. Stat. Assoc. 29, 169–170 (1934) 24. Jelinski, D.E., Wu, J.: The modifiable areal unit problem and implications for landscape ecology. Land Ecol. 11, 129–140 (1996)

194

C. Hui

25. Fotheringham, A.S., Wong, D.W.S.: The modifiable areal unit problem in multivariate statistical-analysis. Environ. Plan A 23, 1025–1044 (1991) 26. Dorling, D.: The visualization of local urban change across Britain. Environ. Plan B 22, 269–290 (1995) 27. Amrhein, C.G.: Searching for the elusive aggregation effect - Evidence from statistical simulations. Environ. Plan A 27, 105–119 (1995) 28. Dark, S.J., Bram, D.: The modifiable areal unit problem (MAUP) in physical geography. Prog. Phys. Geogr. 31, 471–479 (2007) 29. Downey, L.: Using geographic information systems to reconceptualize spatial relationships and ecological context. Am. J. Soc. 112, 567–612 (2006) 30. Flowerdew, R., Manley, D., Steel, D.: Scales, levels and processes: Studying spatial patterns of British census variables. Comp. Environ. Urban. Sys. 30, 2143–2160 (2006) 31. Lery, B.: A comparison of foster care entry risk at three spatial scales. Subs Use Misuse 43, 223–237 (2008) 32. Sexton, K., Waller, L.A., McMaster, R.B., Maldonado, G., Adgate, J.L.: The importance of spatial effects for environmental health policy and research. Human Ecol. Risk Ass. 8, 109–125 (2002) 33. Lembo, A.J., Lew, M.Y., Laba, M., Baveye, P.: Use of spatial SQL to assess the practical significance of the modifiable areal unit problem. Comp. Geosci. 32, 270–274 (2006) 34. Wong, D.W.S.: Spatial decomposition of segregation indices: A framework toward measuring segregation at multiple levels. Geogra. Anal. 35, 179–194 (2003) 35. Harrison, J.A., Allan, D.G., Underhill, L.G., Herremans, M., Tree, A.J., Parker, V., Brown, C.J.: The atlas of Southern African birds, BirdLife South Africa, Johannesburg (1997) 36. Fielding, A.H., Bell, J.F.: A review of methods for the assessment of prediction errors in conservation presence/absence models. Environ. Cons. 24, 38–49 (1997) 37. Kadmon, R., Farber, O., Danin, A.: A systematic analysis of factors affecting the performance of climate envelope models. Ecol. Appl. 13, 853–867 (2003) 38. Hui, C., Li, Z.: Dynamical complexity and metapopulation persistence. Ecol. Model 164, 201–209 (2003) 39. Hui, C., Yue, D.: Niche construction and polymorphism maintenance in metapopulations. Ecol. Res. 20, 115–119 (2005) 40. Levin, S.A., Grenfell, B., Hastings, A., Perelson, A.S.: Mathematical and computational challenges in population biology and ecosystem science. Science 275, 334–343 (1997) 41. Hui, C., McGeoch, M.A.: Evolution of body size, range size, and food composition in a predator-prey metapopulation. Ecol. Complex 3, 148–159 (2006) 42. Hui, C., Li, Z., Yue, D.X.: Metapopulation dynamics and distribution, and environmental heterogeneity induced by niche construction. Ecol. Model 177, 107–118 (2005) 43. Fortin, M.J., Dale, M.R.T.: Spatial analysis: a guide for ecologists. Cambridge Univ. Press, Cambridge (2005) 44. Fahrig, L., Nuttle, W.K.: Population ecology in spatial heterogeneous environments. In: Lovett, G.M., Jones, C.G., Turner, M.G., Weathers, K.C. (eds.) Ecosystem function in heterogeneous landscapes, pp. 95–118. Springer, Berlin (2005)


195

45. Pacala, S.W., Levin, S.A.: Biologically generated spatial pattern and the coexistence of competing species. In: Tilman, D., Kareiva, P. (eds.) Spatial ecology: the role of space in population dynamics and interspecific interactions, pp. 204–232. Princeton Univ. Press, Princeton (1997) 46. Downing, J.A.: Biological heterogeneity in aquatic ecosystems. In: Kolasa, J., Pickett, S.T.A. (eds.) Ecological heterogeneity, pp. 160–180. Springer, Berlin (1991) 47. Morisita, M.: Id −index, a measure of dispersion of individuals. Res. Popul. Ecol. 4, 1–7 (1962) 48. Lloyd, M.: Mean crowding. J. Anim. Ecol. 36, 1–30 (1967) 49. Bliss, C.I., Fisher, R.A.: Fitting the negative binomial distribution to biological data. Biometrics 9, 176–200 (1953) 50. Ripley, B.D.: Spatial statistics. Wiley, New York (1981) 51. Moran, P.A.P.: Notes on continuous stochastic phenomena. Biometrika 37, 17– 23 (1950) 52. Geary, R.C.: The contiguity ratio and statistical mapping. Incorp. Stat. 5, 115– 145 (1954) 53. Perry, J.N.: Spatial analysis by distance indices. J. Anim. Ecol. 64, 303–314 (1995) 54. Perry, J.N.: Measures of spatial pattern for counts. Ecology 79, 1008–1017 (1998) 55. Levins, R.: Some demographic and genetic consequences of environmental heterogeneity for biological control. Bull. Entomol. Soc. Am. 15, 237–240 (1969) 56. Hanski, I.: Metapopulation dynamics. Nature 396, 41–49 (1998) 57. Hanski, I.: Metapopulation ecology. Oxford Univ. Press, Oxford (1999) 58. Dieckmann, U., Law, R., Metz, J.A.J.: The geometry of ecological interactions: simplifying spatial complexity. Cambridge Unive. Press, Cambridge (2000) 59. Tilman, D., Karieva, P.: Spatial ecology: the role of space in population dynamics and interspecific interactions. Princeton Univ. Press, Princeton (1997) 60. Matsuda, H., Ogita, A., Sasaki, A., Sato, K.: Statistical mechanics of population: the lattice Lotka-Volterra model. Prog. Theor. Phys. 88, 1035–1049 (1992) 61. Katori, M., Konno, N.: Upper bounds for survival probability of the contact process. J. Stat. Phys. 63, 115–130 (1991) 62. Tainaka, K.: Paradoxical effect in a three-candidate voter model. Phys. Lett. A 176, 303–306 (1993) 63. Iwasa, Y., Sato, K., Nakashima, S.: Dynamic modeling of wave regeneration (Shimagare) in subalpine Abies forests. J. Theor. Biol. 152, 143–158 (1991) 64. Harada, Y., Ezoe, H., Iwasa, Y., Matsuda, H., Sato, K.: Population persistence and spatially limited social interaction. Theor. Popul. Biol. 48, 65–91 (1994) 65. Harada, Y., Iwasa, Y.: Lattice population dynamics for plants with dispersing seeds and vegetative propagation. Res. Popul. Ecol. 36, 237–249 (1994) 66. Moran, P.A.P.: Notes on continuous stochastic phenomena. Biometrika 37, 17– 23 (1950) 67. Hui, C., McGeoch, M.A.: A self-similarity model for the occupancy frequency distribution. Theor. Popul. Biol. 71, 61–70 (2007) 68. Hui, C., McGeoch, M.A.: Modeling species distributions by breaking the assumption of self-similarity. Oikos 116, 2097–2107 (2007) 69. McGeoch, M.A., Gaston, K.J.: Occupancy frequency distributions: patterns, artefacts and mechanisms. Biol. Rev. 77, 311–331 (2002)

196

C. Hui

70. De Grave, S., Casey, D.: Influence of sample shape and orientation on density estimates on intertidal macrofauna. J. Marine Biol. Assoc. UK 80, 1125–1126 (2000) 71. He, F., Hubbell, S.P.: Percolation theory for the distribution and abundnce of species. Phys. Rev. Lett. 91, 198103 (2003) 72. Hui, C., McGeoch, M.A.: Capturing the “droopy-tail” in the occupancyabundance relationship. Ecoscience 14, 103–108 (2007) 73. Meynard, C.N., Quinn, J.F.: Predicting species distributions: a critical comparison of the most common statistical models using artificial species. J. Biogeogr. 34, 1455–1469 (2007) 74. Holt, A.R., Gaston, K.J., He, F.: Occupancy-abundance relationships and spatial distribution: a review. Basic Appl. Ecol. 3, 1–13 (2002) 75. Kunin, W.E.: Extrapolating species abundance across spatial scales. Science 281, 1513–1515 (1998) 76. Wilson, R.J., Thomas, C.D., Fox, R., Roy, D.B., Kunin, W.E.: Spatial patterns in species distributions reveal biodiversity change. Nature 432, 393–396 (2004) 77. Hartley, S., Kunin, W.E.: Scale dependency of rarity, extinction risk, and conservation priority. Cons. Biol. 17, 1559–1570 (2003) 78. Fotheringham, A.S.: Scale-independent spatial analysis. In: Goodchild, M.F., Gopal, S. (eds.) Accuracy of spatial databases, pp. 221–228. Taylor and Francis, London (1989) 79. Scheiner, S.M.: Six types of species-area curves. Global Ecol. Biogeogr. 12, 441–447 (2003) 80. Bell, G.: The co-distribution of species in relation to the neutral theory of community ecology. Ecology 86, 1757–1770 (2005) 81. Gotelli, N.J., Graves, G.R.: Null models in ecology. Smithsonian Institution Press, Washington (1996)

Fuzzy Logic Control in Communication Networks Chrysostomos Chrysostomou and Andreas Pitsillides Department of Computer Science, University of Cyprus, 75 Kallipoleos Street, P.O. Box 20537, 1678 Nicosia, Cyprus [email protected], [email protected]

Summary. The problem of network congestion control remains a critical issue and a high priority, especially given the increased demand to use the Internet for time/delay-sensitive applications with differing Quality of Service (QoS) requirements (e.g. Voice over IP, video streaming, Peer-to-Peer, interactive games). Despite the many years of research efforts and the large number of different control schemes proposed, there are still no universally acceptable congestion control solutions. Thus, even with the classical control system techniques used from various researchers, these still do not perform sufficiently to control the dynamics, and the nonlinearities of the TCP/IP networks, and thus meet the diverse needs of today’s Internet. Given the need to capture such important attributes of the controlled system, the design of robust, intelligent control methodologies is required. Consequently, a number of researchers are looking at alternative non-analytical control system design and modeling schemes that have the ability to cope with these difficulties in order to devise effective, robust congestion control techniques as an alternative (or supplement) to traditional control approaches. These schemes employ fuzzy logic control (a well-known Computational Intelligence technique). In this chapter, we firstly discuss the difficulty of the congestion control problem and review control approaches currently in use, before we motivate the utility of Computational Intelligence based control. Then, through a number of examples, we illustrate congestion control methods based on fuzzy logic control. Finally, some concluding remarks and suggestions for further work are given.

1 Introduction It is generally accepted that the problem of network congestion control remains a critical issue and a high priority. Despite the many years of research efforts and the large number of different control schemes proposed, there are still no universally acceptable congestion control solutions. Network congestion control is a complex problem, which is becoming even more difficult, A.-E. Hassanien et al. (Eds.): Foundations of Comput. Intel. Vol. 2, SCI 202, pp. 197–236. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com

198

C. Chrysostomou and A. Pitsillides

given the increased demand to use the Internet for high speed, delay-sensitive applications, with differing Quality of Service (QoS) requirements. Designing effective congestion control strategies is known to be difficult because of the complexity of the structure of the network and the variety of the dynamic network parameters involved. In addition to these, the uncertainties involved in identification of the network parameters lead to the difficulty of obtaining realistic, cost effective, analytical models of the network. Thus, even with the classical control system techniques used from various researchers, these still do not perform sufficiently to control the dynamics, and the nonlinearities of the TCP/IP networks, and thus meet the diverse needs of today’s Internet. Consequently, a number of researchers are looking at alternative nonanalytical control system design and modelling schemes that have the ability to cope with these difficulties in order to devise effective, robust congestion control techniques as an alternative (or supplement) to traditional control approaches. These schemes employ fuzzy logic control (a well-known Computational Intelligence technique). In this chapter, we firstly define network congestion, and discuss the difficulty of the congestion control problem. We then review current approaches on congestion control in the world of Internet. We propose that fuzzy logic control (a well-known Computational Intelligence technique) should have an essential role to play in designing this challenging control system. Finally, we present illustrative examples, based on documented published studies, of successful application of fuzzy logic in controlling congestion, and conclude with some suggestions.

2 Congestion Control in Internet Protocol Networks Congestion control is a critical issue in Internet Protocol (IP) networks. Many research proposals can be found in the literature to provide means of avoiding and/or controlling the congestion. The fundamental principles of congestion, and different approaches to avoid or/and control congestion are widely discussed. In this Section, the main functionalities of the standard Transmission Control Protocol (TCP) congestion control mechanisms are explained. In addition, many variants of TCP which have been proposed to meet Internet’s today needs are briefly described. As there is strong trend to progressively move the controls inside the network, closer to where it can be sensed, we discuss the use of router support to congestion control, either by having explicit single-bit feedback, or multi-bit feedback. Due to its current practical significance, in this chapter we focus on explicit single-bit feedback. 2.1

Defining Congestion

Congestion is a complex process to define. Despite the many years of research efforts in congestion control, currently there is no agreed definition. One may

Fuzzy Logic Control in Communication Networks

199

refer to the ongoing discussion between the active members of the networking community as to give the right definition for congestion [1]. Two perspectives on network congestion are the user perspective and the network perspective. Keshav [2] states that ”Network congestion is a state of degraded performance from the perspective of a particular user. A network is said to be congested from the perspective of a user if that user’s utility has decreased due to an increase in network load”. The user experiences long delays in the delivery of data, perhaps with heavy losses caused by buffer overflows. Thus, there is degradation in the quality of the delivered service, with the need for retransmissions of packets (for services intolerant to loss). In the event of retransmissions, there is a drop in the throughput, which leads to a collapse of network throughput, when a substantial part of the carried traffic is due to retransmissions (in that state not much useful traffic is carried). In the region of congestion, queue lengths, hence queuing delays, grow at a rapid pace - much faster than when the network is not heavily loaded. Yang and Reddy [3] give a network-centric definition of congestion, as a network state in which performance degrades due to the saturation of network resources, such as communication links, processor cycles, and memory buffers. For example, if a communication link delivers packets to a queue at a higher rate than the service rate of the queue, then the size of the queue will grow. If the queue space is finite then in addition to the delay experienced by the packets until service, losses will also occur. Observe that congestion is not a static resource shortage problem, but rather a dynamic resource allocation problem [4]. Networks need to serve all users requests, which may be unpredictable and bursty in their behaviour. However, network resources are finite, and must be managed for sharing among the competing users. Congestion will occur, if the resources are not managed effectively. The optimal control of networks of queues is a well-known, much studied, and notoriously difficult problem, even for the simplest of cases (e.g., [5], [6]). Figure 1(a) shows the throughput-load relationship in a packet-switching network [7], [14]. This plot shows the effect of excessive loading on the network throughput for three cases: no control, ideally controlled, and practically controlled. In the case of ideal control, the throughput increases linearly until saturation of resources, where it flattens off and remain constant, irrespective of the increase of loading beyond the capacity of the system. Obviously, this type of control is impossible in practice. Hence for the practically controlled case, we observe some loss of throughput, as there is some communication overhead associated with the controls, possible some inaccuracy of feedback state information as well as some time delay in its delivery. Finally, for the uncontrolled case, congestion collapse may occur whereby as the network is increasingly overloaded the network throughput collapses, i.e. very little useful network traffic is carried - due to retransmissions or deadlock situations. Figure 1(b) shows the corresponding delay-load relationship. The delay (response time) plot follows a similar pattern as the throughput plot. At first,

200


(a)

(b) Fig. 1. Network throughput and delay vs offered load [4]

the delay rises slowly with the offered load even for fast increments of the throughput. Then after the knee point is reached (i.e., the queues start building), the delay curve jumps significantly while the throughput stays flat. Finally, the delay grows indefinitely when the network becomes congested (i.e., the queues start overflowing). 2.2

Congestion Control Principles

Chiu and Jain [8] classify most congestion control approaches into two categories: approaches for congestion avoidance and approaches for congestion recovery. Congestion avoidance mechanisms allow a network to operate in the optimal region of low delay and high throughput, thus, preventing the network from becoming congested. In contrast, the congestion recovery


201

mechanism allows the network to recover from the congested state of high delay and losses, and low throughput. Even if a network adopts a strategy of congestion avoidance, congestion recovery schemes would still be required to retain throughput in the case of abrupt changes in a network that may cause congestion. Both types of approaches are basically resource management problems. They can be formulated as system control problems, in which the system senses its state and feeds this back to its users who adjust their control [8]. This simple classification only provides a very general picture of common properties between separating groups of approaches. A number of taxonomies of congestion control were/could be considered. A detailed taxonomy for congestion control algorithms is proposed by Yang and Reddy [3], which focuses on the decision-making process of individual congestion control algorithms. The main categories introduced by the Yang and Reddy taxonomy are [3]: •

•

Open loop: These are the mechanisms in which the control decisions of algorithms do not depend on any sort of feedback information from the congested spots in the network, that is, they do not monitor the state of the network dynamically. Closed loop: These are the mechanisms that make their control decisions based on some sort of feedback information to the sources. With the provision of feedback, these mechanisms are able to monitor the network performance dynamically. The feedback involved may be implicit or explicit. In the explicit feedback scheme, feedbacks have to be sent explicitly as separate packets (or can be piggybacked ) (e.g. [9]). If there is no necessity of sending the feedback explicitly, the scheme is said to be an implicit feedback scheme. Some examples of such implicit feedbacks are time delays of acknowledgment or timeouts, and packet loss (e.g. [10] - an implicit binary feedback scheme). – The feedback can be further categorized into binary and ”full” feedback. A single bit in the packet header is used as a binary feedback mechanism (e.g. [9] - an explicit binary feedback scheme). ”Full” feedback incorporates use of more than one bit in the packet header that are used to send a whole (i.e. ”full”) information about the status of the network, like the exact sending rate, the round-trip time, etc (e.g. [11] - an explicit multi-bit (”full”) feedback scheme).

A congestion control system should be preventive, if possible. Otherwise, it should react quickly and minimise the spread of congestion and its duration. A good engineering practice will be to design the system in such a way as to avoid congestion. But taken to the extreme (i.e. to guarantee zero losses and zero queuing delay), this would not be economical. For example, assuring zero waiting at a buffer implies increasing the service rate at its limit to infinity. A good compromise would be to allow for some deterioration of performance, but never allow it to become intolerable (congested). The challenge is to keep

202


the intolerance at limits acceptable to the users. Note the fuzziness present in defining when congestion is actually experienced. The difficulty of the congestion control problems has caused a lot of debate as to what are appropriate control techniques for the control of congestion, and depending on one’s point of view, many different schools of thought were followed, with many published ideas and control techniques. 2.3

Internet Congestion Control

The Internet Protocol (IP) architecture is based on a connectionless end-toend packet service. Transmission Control Protocol (TCP) is an end-to-end transport protocol that provides reliable, in-order service. Congestion control is implemented via a reactive, closed-loop, dynamic window control scheme [10]. This window-based scheme operates in the hosts to cause TCP connections to ”back off” during congestion. That is, TCP flows are responsive to congestion signals (i.e. dropped packets indicated by a timeout or a triple duplicate acknowledgment) from the network. It is primarily these TCP congestion avoidance algorithms that prevent the congestion collapse of today’s Internet. A fundamental aspect of TCP is that it obeys a ”conservation of packets” principle, where a new segment is not sent into the network until an old segment has left. TCP implements this strategy via a self-clocking mechanism (acknowledgements received by the sender are used to trigger the transmission of new segments). This self-clocking property is the key to TCP’s congestion control strategy. Other elements of TCP’s congestion control include the congestion avoidance algorithm, the congestion recovery algorithm (i.e. slow-start), and the fast retransmit/recovery algorithms [12]. A TCP sender additively increases its rate when it perceives that the end-path is congestion-free, and multiplicatively decreases its rate when it detects (via a loss event) that the path is congested. Thus, in such situations, TCP congestion control deploys the so called additive-increase, multiplicativedecrease (AIMD) algorithm. The linear increase phase of TCP’s congestion control protocol is known as congestion avoidance. The value of congestion window repeatedly goes through cycles during which it increases linearly and then suddenly drops to half its current value (when a loss event occurs, and particularly a triple duplicate acknowledgment), giving rise to a saw-toothed pattern in long-lived TCP connections [13]. During the initial phase of TCP’s congestion control, which is called slowstart, the TCP sender begins by transmitting at a slow rate but increases its sending rate exponentially, until a slow-start threshold is reached, where the congestion avoidance phase begins. In the case of a loss event, the AIMD saw-toothed pattern begins. The TCP congestion control reacts differently to a loss event that is detected via a timeout event, than it does to a loss event detected via receipt of a triple duplicate acknowledgment (ACK). After a triple duplicate ACK,


203

Fig. 2. Responsiveness and smoothness of the control [8]

the congestion window is cut in half and then increases linearly (i.e. AIMD). However, after a timeout event, a TCP sender enters a slow-start phase, where the congestion window is set to 1, and then it grows exponentially, until it reaches one half of the value it had before the timeout event. At that point, the TCP enters congestion avoidance phase. TCP Feedback Signalling Scheme The TCP feedback signalling scheme in the current Internet is binary and implicit (i.e., network congestion is detected at the sources by loss events). Due to the binary nature of the feedback, and consequently the AIMD sawtoothed pattern, the system does not generally converge to a single steady state. The system reaches an ”equilibrium”, in which it oscillates around the optimal state [8] (see Fig. 2). The time taken to reach the equilibrium (that determines the responsiveness of the control), and the size of the oscillations (that determines the smoothness of the control) jointly determine the convergence. Ideally, we would like the time as well as oscillations to be small. Therefore, the controls with smaller time and smaller amplitude of oscillations are called more responsive and smoother, respectively. TCP Evolution The congestion control mechanisms continue to be enhanced as TCP/IP evolves to meet new and more demanding requirements. The early version of TCP, known as TCP Tahoe, enters the slow-start phase irrespective of the type of loss event. The newer version of TCP, TCP Reno, cancels the slow-start phase after a triple duplicate ACK, which is called fast recovery, and resends the lost packet, without waiting for a timeout event, that is a fast retransmit occurs.

204


The TCP NewReno [15] improves the Reno implementation regarding the fast recovery mechanism. The aim of TCP NewReno is to prevent a TCP sender from reducing its congestion window multiple times in case several packets are dropped from a single window of data (a problem Reno has). The NewReno remains in fast recovery until all of the data outstanding by the time the fast recovery was initiated have been acknowledged. NewReno can retransmit one lost packet per round trip time (RTT), until all the lost packets from a particular window of data have been retransmitted. Thus NewReno avoids multiple reductions in the congestion window, or unnecessary retransmit timeout with slow start invocation. Another proposed modification to TCP, the TCP Selective Acknowledgement (SACK) [16] allows a TCP receiver to acknowledge out-of-order packets selectively rather than just cumulatively acknowledging the last correctly received, in-order packet. Thus TCP Sack may recover multiple lost packets in a window of data in just one single RTT. The behaviour of TCP/IP congestion control still remains a matter of continuous research interest in the TCP/IP world (highlighted by the frequent Internet Engineering Task Force - IETF - Request for Comments - RFCs, and many published papers in various journals and conferences, proposing fixes or new solutions). Recently, there is an ongoing research towards enhancing the TCP congestion control mechanisms in order for TCP to fully exploit the network capacity of fast, long-distance networks (i.e. high-speed networks operating at 622 M bps, 2.5 Gbps, or 10 Gbps, which have a high bandwidthdelay product). Network-Assisted Congestion Control With network-assisted congestion control, routers provide explicit feedback to the sender regarding the congestion state in the network. This feedback may be as simple as a single bit indicating congestion at a link, or more complex as a multi-bit feedback giving to the source ”full” information about the network state (e.g., the exact sending rate). Congestion information is typically conveyed from each router at the path from the sender to the receiver, by marking/updating a field in a packet’s header, to indicate congestion, and then fed back from the receiver to the sender as a form of notification. It has become clear [17] that the existing TCP congestion avoidance/control mechanisms and its variants, while necessary and powerful, are not sufficient to provide good service in all circumstances. Basically, there is a limit to how much control can be accomplished from the edges of the network. Some additional mechanisms are needed in the routers to complement the endpoint congestion avoidance/control methods, as suggested by various researchers ([9], [17], [18]). Note that the need for router control was realised early; e.g. see [10], where for future work the router side is advocated as necessary. A clear trend is observed: to progressively move the controls inside the network, closer to where it can be sensed.


205

By using network-assisted congestion control, TCP does not need to await a loss event - due to buffer overflow - to detect congestion and slow down properly. Instead, it is informed by the intermediate nodes (routers) when incipient congestion starts, and reacts accordingly. Binary Feedback The simplest method of assisting the TCP from the network point of view is to provide a binary feedback to the source about the network state. The use of Explicit Congestion Notification (ECN) was proposed [9] in order to provide TCP an alternative to packet drops as a mechanism for detecting incipient congestion in the network. The ECN proposal works together with the addition of active queue management (AQM) to the Internet infrastructure, where routers detect congestion before the queue overflows (see discussion of AQM in detail in Sect. 2.4). The ECN scheme requires both end-to-end and network support. An ECNenabled router can mark a packet by setting a bit in the packet’s header, if the transport protocol is capable of reacting to ECN. Specifically, the ECN proposal requires specific flags in both IP and TCP headers. Two bits are used in each header for proper signalling among the sender, routers, and the receiver. In the IP header, the two bits (ECN field) results in four ECN codepoints (see Table 1). The ECN-Capable Transport (ECT) codepoints ’10’ and ’01’ are set by the data sender to indicate that the end-points of the transport protocol are ECN-capable. Routers treat both codepoints as equivalent; senders are free to use either of the two to indicate ECT. The not-ECT codepoint ’00’ indicates a packet that is not using ECN. The ECN codepoint ’11’ is set by a router to indicate congestion to the end nodes (i.e. marks the packet); this is called the Congestion Experienced (CE) codepoint. Upon the receipt by an ECN-capable transport of a single CE packet, the congestion control algorithms followed at the end nodes must be essentially the same as the congestion control response to a single dropped packet. In the TCP header, two new flags are introduced. The ECN-Echo (ECE) flag is used by the receiver to inform the sender that a CE packet has been received. This is done in the ACK packet sent. Similarly, the sender uses the Congestion Window Reduced (CWR) flag to announce to the receiver that its congestion window has been reduced, as a consequence of the reception of the ECE ACK. Table 1. The ECN field in IP ECN FIELD 0 0 1 1

0 1 0 1

Not-ECT ECT(1) ECT(0) CE

206


The use of ECN for notification of congestion to the end-nodes generally prevents unnecessary packet drops, and thus is appealing to be used in the Internet. 2.4

Active Queue Management in TCP/IP Networks

The TCP congestion avoidance/congestion control mechanisms have been very successful, as the Internet has evolved from a small-scale research network to today’s interconnected millions of networks. However, the increased demand to use the Internet for time/delay-sensitive applications with differing QoS requirements, questions the efficiency and the feasibility of such an end-to-end implicit feedback based congestion control. Thus the need for a robust enough controller to capture the dynamics, the highly bursty network traffic, and the nonlinearities of the controlled system leads to the introduction of Active Queue Management (AQM) mechanisms to assist the TCP congestion control to obtain satisfactory performance. AQM mechanisms are proposed to provide high link utilization with low loss rate and queuing delay, while responding quickly to load changes. Due to the adherence to the current Internet standards next we focus on AQM mechanisms, which either drop or mark packets to indicate congestion, and also keep the TCP’s window increase and decrease mechanism at the sources unchanged. Active Queue Management Principles The AQM approach can be contrasted with the Drop Tail (DT) queue management approach, employed by common Internet routers, where the discard policy of arriving packets is based on the overflow of the output port buffer. Contrary to DT, AQM mechanisms [17] start dropping or marking packets earlier in order to notify traffic sources about the incipient stages of congestion (TCP interprets dropped packets as congestion). AQM allows the router to separate policies of dropping packets from the policies for indicating congestion. In the case of dropping of packets the TCP congestion controller relies on the implicit feedback signal (generated by the lost packet as a timeout) to reduce the TCP congestion window. In the case of packet marking packets are not dropped, rather a bit is set on their header indicating congestion (hence termed Explicit Congestion Notification, ECN [9]), and returned via the destination to the source. The main AQM performance characteristics include [20]: • •

Efficient queue utilization: the queue should avoid overflow that results in lost packets and undesired retransmissions or emptiness that results in link underutilization. Queuing Delay: It is desirable to keep small both the queuing delay and its variations.


•

207

Robustness: AQM scheme needs to maintain robust behaviour in spite of varying network conditions, such as variations in the number of TCP sessions, and variations in the propagation delay and link capacity.

Examples of Active Queue Management Mechanisms Several schemes have been proposed to provide congestion control in TCP/IP networks (e.g., [18], [19], [20], [21], [22], [63], [64], [65], [66]). Below we briefly review some of the prominent proposals and their limitations. Random Early Detection (RED) [18] was the first AQM algorithm proposed. It sets some minimum and maximum drop/mark thresholds in the router queues. In case the average queue size exceeds the minimum threshold, RED starts randomly dropping/marking packets, based on a linear heuristicbased control law, with a drop/mark probability depending on the average queue length, whereas if it exceeds the maximum threshold every packet is dropped. Adaptive-RED (A-RED) [19], proposed by one of the authors of RED, attempts to solve the problem of the need for continuously (re)tuning RED parameters. In particular, A-RED adjusts the value of the maximum drop/mark probability to keep the average queue size within a target range half way between the minimum and maximum thresholds. Thus, A-RED aims to maintain a desired average target queue length (TQL) twice the minimum threshold (if the maximum threshold is kept three times the minimum threshold). The adjustment of the maximum drop/mark probability is based on an additive fixed increase step when the average queue length exceeds the desired average queue, and on a multiplicative fixed decrease step when the average queue length goes below the desired average value, following a linear AIMD approach. The Proportional-Integral (PI) controller is proposed in [20], based on linear control theory. Three key network parameters - the number of TCP sessions, the link capacity and the round-trip time (RTT) - are related to the underlying feedback control system. The key feature is that PI control is based on formal feedback based linear control theory, drawing from its vast experience in controlling systems. It allows one to explicitly set the network queuing delay by introducing a desired queue length so as to stabilize the router queue length around this value. In [21] a new AQM scheme was proposed, namely Random Exponential Marking (REM). The key idea behind this AQM design is to stabilize both the input rate around link capacity and the queue length around a small target. The mark probability calculated is based on an exponential law. Further, in [22] an Adaptive Virtual Queue (AVQ) -based dropping/marking scheme for AQM was proposed. AVQ uses a modified token bucket model as a virtual queue to regulate link utilization, rather than the actual queue length. The AVQ scheme detects congestion solely based on the arrival rate of the packets at the link.

208


Limitations of Existing AQM Mechanisms The properties of RED have been extensively studied in the past few years. Issues of concern include: problems with performance of RED under different scenarios of operation and loading conditions; the correct tuning of RED parameters implies a global parameterization that is very difficult, if not impossible to achieve. Some researchers have advocated against using RED, in part because of this tuning difficulty [23]; the sensitivity to the averaging of the queue size [20]; also the linearity of the dropping function has been questioned by a number of researchers (e.g., [24]). As the RED-based algorithms control the macroscopic behaviour of the queue length (looking at the average) they often cause sluggish response and fluctuation in the instantaneous queue length [25]. As a result, a large variation in end-to-end delays is observed. Further, the linear drop/mark probability of RED itself is not robust enough for the highly bursty network traffic. The motivation should be to find a proper nonlinear function, rather than to find RED parameters appropriately tuned for a specific operating point for the original linear RED function. A-RED attempts to tune the RED parameters for a robust behavior but fails to do so in various dynamic cases [25] due to the fact that A-RED retains RED’s basic linear structure. Thus, fine tuning of the RED parameters is not sufficient to cope with the undesired RED behavior. The PI controller behaves in a similar way by exhibiting sluggish response to varying network conditions. This can be explained due to the fact that the fixed/static PI parameters are dependent on network parameters, like the number of flows and RTT, and thus it is difficult to get a stable operation in a broad range of dynamic varying traffic conditions. An illustrative example of how the PI AQM mechanism requires careful configuration of non-intuitive control parameters can be found in [26], where the PI controller shows weaknesses to detect and control congestion under dynamic traffic changes, and a slow response to regulate queues. Similarly, the AVQ control parameters are dependent on network parameters, like the round-trip delay and the number of flows. Thus, it is difficult to get a stable operation as stated above. The REM controller follows a function equivalent the price control function of the PI controller, thus it is also found to exhibit sluggish response to varying network conditions. The correct configuration of REM control parameters is still an issue for further investigation, concerning the dynamic, time-varying nature of TCP/IP networks. In general, the existing AQM mechanisms still require a careful configuration of non-intuitive control parameters. The dynamics of TCP/AQM models are mostly studied with the aid of linearization around equilibrium points of the nonlinear model developed, in order to study TCP/AQM stability around equilibrium. However, linearization fails to track the system trajectories across different regions dictated by the nonlinear equations


209

derived. As stated in [27], linearization ”assumes, and hence requires that the system always stays within a certain operating regime”. Moreover, the equations modeled are dependent on various network parameters, such as the number of flows and the round trip delays, which vary substantially in today’s Internet. Therefore, the linearization around a specific operating point and the dependence on varying network parameters make it difficult to get a stable and robust operation in the case of TCP/IP networks with dynamic load and delay changes. Hence, a major weakness of the proposed models is that the configuration of control parameters is done for a specific operating point, for which various system parameters are assumed to be known, and certain important dynamics are ignored. As stated in [24], even if the assumptions regarding the input parameters fit the specific scenario, the applicability of the AQM algorithm would be restricted to a small range of the assumed values only. Therefore, the configured parameter set and stability conditions introduced by the proposed models lack applicability to all possible real scenarios with varying dynamics of network conditions. In addition, even if the linearized system is made stable at equilibrium, there is no guarantee that the nonlinear system will remain stable [24], especially if the deviations from the equilibrium are at times large. 2.5

Differentiated Services Congestion Control

The Differentiated Services (Diff-Serv) approach [28] proposes a scalable means to deliver IP QoS based on handling of traffic aggregates. It operates on the premise that complicated functionality should be moved toward the edge of the network with very simple functionality at the core. The Diff-Serv framework enables QoS provisioning within a network domain by applying rules at the edges to create traffic aggregates and coupling each of these with a specific forwarding path treatment in the domain through the use of a codepoint in the IP header. The Diff-Serv Working Group (WG) of IETF (Internet Engineering Task Force) has defined the general architecture for differentiated services and has focused on the forwarding path behaviour required in routers. The WG has also discussed the functionality required at Diff-Serv domain edges to select and condition traffic according to a set of rules. The Diff-Serv architecture aims to provide aggregated QoS. Our focus in this chapter is on the development of differential dropping/marking algorithms for network core routers to support this objective. Further, AQM mechanisms, we believe, can be usefully employed at the core of the Diff-Serv domain to provide bandwidth assurance, with low loss and bounded delay to various (aggregated) service classes. Currently, there is ongoing work between members of the networking community for creating configuration guidelines recommended for Diff-Serv Service Classes [29]. In this chapter, we concentrated on the managing of the Assured Per-hop Behavior (AF PHB) [30] that specifies a forwarding behavior in which packets are expected to see a very small amount of loss. The AF PHB group is a means

210


to offer different levels of forwarding assurances for IP packets, and it provides delivery of IP packets in four independently forwarded AF classes (AF1, AF2, AF3, and AF4). In each Diff-Serv node, each AF class is allocated a certain amount of forwarding resources (buffer space and bandwidth), and should be serviced to achieve the configured service rate (bandwidth). Within each AF class, IP packets are marked with one of three possible drop precedence values (e.g., AF11, AF12, AF13). In case of congestion, the drop precedence of a packet determines the relative importance of the packet within the AF class. A congested Diff-Serv node tries to protect packets with a lower drop precedence value from being lost by preferentially discarding packets with a higher drop precedence value; thus it differentiates flows with different drop preference levels. The most popular algorithm used for Diff-Serv implementation is based on RED [18]. The RED implementation for Diff-Serv, called RED In/Out (RIO) [31], implements that we have different thresholds for different drop precedence levels. ”In” and ”Out” means packets are in or out of the connection conformance agreement. RIO uses the same mechanism as in RED, but it is configured with two different sets of parameters, one for ”In” (high priority - low drop precedence level) packets, and one for ”Out” (low priority - high drop precedence level) packets. Upon each packet arrival at the router, RIO checks whether the packet is tagged as ”In” or ”Out”. If it is an ”In” packet, RIO calculates the average queue length of ”In” packets only; otherwise (i.e., the packet is tagged as ”Out”) RIO calculates the total average queue length (i.e., of both ”In” and ”Out” arriving packets). The probability of dropping/marking an ”In” packet depends on the average queue length of ”In” packets, whereas the probability of dropping/marking an ”Out” packet depends on the total average queue length. The discrimination against ”Out” packets is created by carefully choosing the parameters of minimum and maximum thresholds, and maximum drop/mark probability. It drops ”Out” packets much earlier than ”In” packets; this is achieved by choosing the minimum threshold for ”Out” packets smaller than the minimum threshold for ”In” packets. It also drops/marks ”Out” packets with a higher probability, by setting the maximum drop/mark probability for ”Out” packets higher than the one for ”In” packets. However, as RIO is the implementation of RED for Diff-Serv, it still suffers from the undesired RED behaviour, as this was discussed in Sect. 2.4. In [32], based on analytic evaluation of the loss probability, it is concluded that the ”choice of different RIO parameter values can have a major impact on performance”. RIO also retains RED’s basic linear structure (between minimum and maximum average queue threshold values) that itself is not robust enough for the bursty network traffic, and cannot capture the dynamics and nonlinearities of TCP/IP networks. Furthermore, RIO’s decision for dropping/marking a packet of any level of drop precedence is not based on the total buffer occupancy; this may be a drawback if we want to have a bounded delay for the queue as a whole, and under any congestion level.


211

Beyond RIO, another popular algorithm, proposed for Diff-Serv AQMbased congestion control, is based on the standard PI AQM [20]. In particular, a two-level AQM controller (TL-PI) is proposed in [33] for providing differential marking probabilities at the Diff-Serv core. The PI AQM scheme proposed in [20] is used to preferentially drop/mark high-level of drop precedence, rather than low-level, by introducing two set points (TQLs) for the core queue, which correspond to the two levels of drop precedence used, respectively. The drop/mark probability for both levels is computed by two PI AQM controllers, using the same parameter values, except for the TQL. In order to preferentially drop/mark packets of high drop precedence during congestion the TQL of the low-level of drop precedence is set higher than the TQL of the high-level of drop precedence. However, as the two-level PI controller is actually the PI implementation for Diff-Serv congestion control, it still suffers from the undesired PI behavior, discussed in Sect. 2.4 (e.g., the dependency of PI control parameters on dynamic network parameters, like the number of flows and the round trip propagation delays, and the linearity of the control law). 2.6

The Need for the Alternative

As discussed in Sect. 2.3, the current Internet feedback mechanism for congestion control is binary and implicit and the network provides a best effort service. However, the existing TCP congestion avoidance/control mechanisms and its variants, while necessary and powerful, are not sufficient to provide good service in all circumstances [17]. Therefore, network-assisted mechanisms have been introduced (e.g., ECN) to provide a more responsive feedback mechanism. The pressing need to better capture the dynamics and the highly bursty network traffic, and nonlinearities of TCP has lead to the design of AQM mechanisms as router support to the TCP congestion control. While many AQM mechanisms (see Sect. 2.4) have recently been proposed in the best effort TCP/IP environment, these require careful configuration of non-intuitive control parameters that are dependent on network/traffic parameters, and show weaknesses to detect and control congestion under dynamic traffic changes, and exhibit a slow response to regulate queues [26]. Based on the above identified limitations it is evident [26] that by using a nonlinear drop/mark probability function, which does not require knowledge of dynamic system/network parameters, an effective and robust AQM system can be designed to drive quickly the system to be controlled into the steady-state. This should be contrasted with the linear drop/mark probability function that itself is not robust enough for the highly bursty network traffic and cannot capture the dynamics and nonlinearities of TCP/IP networks. For example, during high load conditions a disproportionately higher drop/mark probability is required than in a low load condition, in order to keep the queue length in the same range, a requirement met only by a nonlinear drop/mark function.

212


Thus the complexity of these problems and the difficulties in implementing conventional controllers to eliminate those problems, as identified in Sect. 2.4, motivate the need to investigate intelligent control techniques, such as fuzzy logic, as a solution to controlling systems in which dynamics and nonlinearities need to be addressed. This work supplements the standard TCP to obtain satisfactory performance in a best-effort environment. Fuzzy logic control [38] has been widely applied to control nonlinear, time-varying systems, in which they can provide simple and effective solutions. The capability to qualitatively capture the attributes of a control system based on observable phenomena is a main feature of fuzzy logic control and has been demonstrated in various research literature and commercial products. The main idea is that if the fuzzy logic control is designed with a good (intuitive) understanding of the system to be controlled, the limitations due to the complexity system’s parameters introduce on a mathematical model can be avoided. A common approach in the networking literature is to either ignore such complex parameters in the mathematical model (e.g., ignoring the slow-start phase in the nonlinear model derived in [34]), or to simplify the model (e.g., ignoring the timeout mechanism and linearization of the model derived in [35]) to such an extent (in order to obtain tractable model for controller design and/or stability results), which render the designed controllers and their derived stability bounds overly conservative.

3 Fuzzy Logic Control Fuzzy logic is a logical system, which is an extension and generalization of multivalued logic systems. It is one of the family of tools of what is commonly known as Computational Intelligence (CI). CI is an area of fundamental and applied research involving numerical information processing. While these techniques are not a panacea (and it is important to view them as supplementing proven traditional techniques), there is a lot of interest not only from the academic research community (e.g. [4], [36]) but also from industry, including the telecommunications industry (e.g. [37]), due to its successful deployment in controlling difficult systems. Fuzzy Logic Control (FLC) [38] denotes the field in which fuzzy set theory [39] and fuzzy inference are used to derive control laws. A fuzzy set is defined by a membership function that can be any real number in the interval [0, 1], expressing the grade of membership for which an element belongs to that fuzzy set. The concept of fuzzy sets enables the use of fuzzy inference, which in turn uses the knowledge of an expert in a field of application to construct a set of ”IF-THEN” rules. Fuzzy logic becomes especially useful in capturing human expert or operator’s qualitative control experience into the control algorithm, using linguistic rules. The idea of FLC was initially introduced by Zadeh [40] and first applied by Mamdani [41] in an attempt to control systems that are difficult to model mathematically and hence design controllers. FLC may be viewed as a way of


213

designing feedback controllers in situations where rigorous control theoretic approaches are too difficult and time consuming to use, due to difficulties in obtaining a formal analytical model, while at the same time some intuitive understanding of the process is available. Fuzzy logic control has strengths in controlling highly nonlinear, complex systems, which are commonly encountered in product design, manufacturing and control. Fuzzy logic provides a set of mathematical methods for representing information in a way that resembles natural human communication, and for handling this information in a way that is similar to human reasoning. The control algorithm is encapsulated as a set of linguistic rules, leading to algorithms describing what action should be taken based on system behaviour observations. FLC has been applied successfully for controlling numerous systems in which analytical models are not easily obtainable or the model itself, if available, is too complex and possibly highly nonlinear (e.g. [42]). Therefore, FLC concentrates on attaining an intuitive understanding of the way to control the process, incorporating human reasoning in the control algorithm. A fuzzy logic controller can be conceived as a nonlinear controller whose input-output relationship is described in linguistic terms that can be better understood and easily modified (tuned). It is independent of mathematical models of the system to be controlled. It achieves inherent robustness and reduces design complexity. This is in contrast with conventional control approaches that concentrate on constructing a controller with the aid of an analytical system model that in many cases is overly complex, uncertain, and sensitive to noise. 3.1

Application of Fuzzy Logic in Networks

Fuzzy Logic Control (FLC) has been successfully used in a wide variety of applications in engineering, science, business, medicine and other fields. A number of research papers using fuzzy logic investigating solutions to congestion control issues in networking, especially in Asynchronous Transfer Mode (ATM) networks, have been published. Given the complexity of ATM networks, rich variety of traffic sources that operate on them, and difficulty of obtaining formal models for in depth analysis, it is not surprising to see that FLC was adopted by many researchers. For example, [4], [43], [44] and [45], since early 90’s, have successfully used the concept of FLC for congestion control in ATM, as an alternative to conventional counterparts. A survey of some of these techniques is given in [36]. Based on the vast experience of successful implementations of FLC in the design of control algorithms, as indicated above, and the reported strength of fuzzy logic in controlling complex and highly nonlinear systems, FLC was also used in the IP world. To the best of our knowledge, fuzzy logic, in the concept of AQM congestion control in TCP/IP networks, was introduced in the early 2000s by Pitsillides, Rossides, Chrysostomou, et al. (e.g. [46], [47], and [48]). Their proposed scheme concerned a fuzzy-based RED variant illustrated in

214


a Diff-Serv environment with input variables the error on the queue length, and the rate of change of the error, while the output is the packet drop probability. This earlier research demonstrated that the application of fuzzy control techniques to the problem of congestion control in TCP/IP networks is worthy of further investigation. Lately, we are witnessing an increase of research papers focusing on the use of fuzzy logic in various fields of the IP world. Fengyuan, Yong, and Xiuming [49] have proposed a fuzzy controller for AQM in best effort IP networks, with input variables the error on the queue length, and the rate of change of the error, while the output is the increment step of the packet drop/mark probability. They clearly presented guidelines towards to the design of the FLC, implemented in NS-2 simulator [58], and compared performance under various scenarios with the PI controller [20]. The proposed FLC has superior steady and transient performance, and provides robustness against noise and disturbance, as well as adaptability to the variances of link delay and capacity. Wang, Li, Sohraby, and Peng [50] proposed a fuzzy controller for best effort AQM, with only one input - the queue length, while the output is the drop probability. This scheme is implemented by keeping the RED’s algorithm [18] semantics (they use the same threshold-based method as that in RED, i.e., when the queue length is less than a minimum threshold the probability is zero; when the queue length is between the minimum and maximum threshold the drop probability is computed, and when the queue length is greater than the maximum threshold then the same gentle mechanism in RED is used). The important feature of their proposed scheme is that it also designs an adaptive mechanism to dynamically readjust the fuzzy rule so as to make the scheme itself extensively stable for many dynamic environments. Simulation results [58] show that their scheme can effectively and stably control the queue length to the expected value or so. Compared with RED [18] and PI [20] algorithm, they obtain higher goodput and stable queue length than RED and PI, even with the introduction of UDP flows. Aul, Nafaa, Negru, and Mehaoua [51] proposed a fuzzy controller for best effort AQM, with input variables the error on the queue length, and the rate of change of the error, while the output is the drop probability. Comparing to traditional AQM algorithms (e.g., RED [18]), their proposal avoids buffer overflows/underflows, and minimizes packets dropping. Further, an on-line adaptation mechanism is proposed that captures fluctuating network conditions, while classical AQM algorithms require static tuning. Di Fatta, Hoffmann, Lo Re, and Urso [52] proposed a fuzzy PI-controller for AQM, where the gains are tuned by a genetic algorithm with respect to optimal disturbance rejection. The analytical design of the proposed controller is carried out in analogy with a PI [20] controller. The main objectives of the controller design method are fast response to high load variations and disturbance rejection in steady-state behavior. The experimental results demonstrate that the proposed controller outperforms the other AQM policies ([18], [20]) under various operating conditions, especially for traffic that


215

exceeds the nominal bandwidth causing severe overload on the node. The improvement in terms of response time and link utilization is due to the fact that the nonlinear fuzzy controller has a variable gain that allows the AQM to recover faster from large variation in traffic loads. Chrysostomou et al. [59], [60], [25], [26] have proposed a generic AQM control methodology in TCP/IP networks, based on FLC. A simple, effective and efficient nonlinear control law is built, using a linguistic model of the system, rather than a traditional mathematical model, which is easily adapted in different network environments (e.g. Best-Effort and Differentiated-Services architectures). It is demonstrated, via extensive simulative evaluation [26], that the proposed fuzzy control methodology offers inherent robustness with effective control of the system under widely differing operating conditions, without the need to (re)tune the settings for two different architectures (BestEffort and Differentiated-Services). As demonstrated, this is in contrast with the well-known conventional counterparts of A-RED [19], PI [20], REM [21], AVQ [22] for Best-Effort, and two-level PI [33], RIO [31] for Diff-Serv based networks, where the proposed approach outperforms all tested counterparts in each different architecture. A detailed overview is presented in Sect. 3.2. Fuzzy logic control has also been used, beside AQM, in other fields concerning today’s Internet. Siripongwutikorn, Banerjee, and Tipper [53] have proposed an adaptive bandwidth control algorithm based on fuzzy control to maintain the aggregate loss QoS. Habetha and Walke [54] developed a new clustering scheme concerning mobility and load management, based on fuzzy logic. Wang et al. [55] presented a fuzzy-based dynamic channel-borrowing scheme to maximize the number of served calls in a distributed wireless cellular network. Savoric [56] proposed a fuzzy explicit window adaptation scheme that can decrease the advertised receiver window in TCP acknowledgements if necessary in order to avoid congestion and packet losses. Oliveira and Braun [57] proposed a technique for packet loss discrimination using fuzzy logic over multihop wireless networks. 3.2

An Illustrative Example: A Generic Fuzzy AQM Control Methodology in TCP/IP Networks

In this section, the operation of the unified fuzzy congestion controller ([59], [60], [25], [26]) for best effort and Diff-Serv networks is summarized. Fuzzy Explicit Marking (FEM) Control System Design The nonlinear fuzzy logic-based control system (FLCS) is designed to operate in TCP/IP Best-Effort networks, and specifically in the IP routers’ output port buffer. However, the aim is to also design a generic control methodology that can be easily adopted in other network environments as well, as for example in TCP/IP Diff-Serv. The proposed FLCS [25] (see Fig. 3 - details are discussed below) is based on an AQM approach, which implements a drop probability function, and

216


Fig. 3. Fuzzy logic based AQM (FEM) system model [25]

supports ECN in order to mark packets, instead of dropping them. It uses feedback from the instantaneous queue length sampled frequently and is driven by the calculated errors between a given queue reference for the present and previous periods. The end-to-end behaviour of TCP is retained, with the TCP increase and decrease algorithm responding to ECN marked packets. The principal aim of the proposed nonlinear FLCS is to achieve the following goals: •

• • •

Dynamic and effective fast system response with robustness to the time-varying, dynamic nature of the controlled system, under differing operating conditions, without the need for (re)tuning, and thus provide acceptable QoS. High link utilization (based on the useful throughput). Minimal packet losses. Bounded-regulated queue fluctuations and delays (mean and variation).

The bounded mean queuing delay and delay variation can be achieved by regulating the queues of the output port buffers of IP routers at predefined levels. This will, as a consequence, have low losses and maintain high utilization as well. By having a nonlinear control law, based on fuzzy logic, the aim is to effectively deal with the high variability appearing in the network, and thus exhibit fast system response and robust behavior in spite of varying network conditions. The proposed FLCS in TCP/IP Best-Effort networks, called Fuzzy Explicit Marking (FEM) controller, provides a new nonlinear probability function that marks packets at IP routers in order to regulate queues at predefined levels, by achieving a specified target queue length (TQL). In order to design the FEM controller the following standard steps have been followed by the authors: • •

Identify the inputs and their ranges (universe of discourse) Identify the output and its range


• • • •

217

Construct the rule base (knowledge) that the system will operate under Create the degree of fuzzy membership function for each input and output Decide how the action will be executed for each rule Combine the rules and defuzzify the output

There is no systematic procedure to design the fuzzy controller [38]. The most commonly used approach is to define membership functions of the inputs and output based on a qualitative understanding of the system, together with a rule data base and to test the controller by trial-and-error until satisfactory performance is achieved. The authors rely on the use of heuristic expertise and study of the plant dynamics about how to best configure the control law. The focus is on the achievement of the design goals indicated above, whilst keeping the design of the controller as simple and generic as possible. As noted by the authors, as the fuzzy controller is nonlinear, it is very difficult to examine analytically the influence of certain parameters. Usually, extensive simulation and experimentation are used to investigate its behaviour. The authors’ aim is to ensure that the controller will have the proper information available to be able to make good decisions, and will have proper control inputs to be able to steer the controlled system in the directions needed, so that it achieves a high-performance operation, as pointed out above. Some of the design choices are briefly described below. Input-Output Selection and Scaling Since multiple inputs are usually used to capture the dynamic state of the controlled system more accurately, and also to offer better ability to linguistically describe the system dynamics [38], the authors utilize a two-input, single-output (the simplest of the Multiple Input Single Output (MISO) model based) fuzzy controller on the buffer of each output port of a router in TCP/IP networks. There is a need to choose the right inputs and output with generic normalized universe of discourse, applicable in any network/traffic environment. Thus, the decision made by the authors is to use the error on the instantaneous queue length from a target value for two consecutive sampling intervals. Sampling at every packet arrival, just like RED [18] does, is in the authors’ opinion an overkill and provides no perceptible benefit. By measuring the queue at two consecutive sampling intervals (the current and the past), it is attempted to estimate the future behavior of the queue. It is well known that the difference between the input arrival rate at a buffer and the link capacity at discrete time intervals can be approximated and visualized as the rate at which the queue length grows when the buffer is non-empty. Thus, as it is usually easier to sample queue length than rate in practice, the change of the queue length for two consecutive discrete time intervals is tracked. The system converges only when both sampled queue lengths reach the TQL (i.e. the errors on the queue length go to zero). The errors converging to zero imply that the input rate has been matched to the

218


link capacity, and there is no growth or drain in the router queue level. This has the effect of decoupling the congestion measure from the performance measure by keeping as congestion indices the queue length and the input rate (which is approximated with queue growth rate, as discussed above). Further, the output of the controller is selected as a nonlinear mark probability that is given as input of the controlled system in order to decide whether to mark a particular packet. After all the inputs and the output are defined for the FEM controller, the fuzzy control system shown in Fig. 3 is specified, where all quantities are considered at the discrete instant kT : • •

T is the sampling period e(kT ) is the error on the controlled variable queue length, q(kT ), from a specified TQL (qdes ), at each sampling period kT , defined in (1). e(kT ) = qdes − q(kT )

• • • •

(1)

e(kT − T ) is the error on queue length with a delay T (at the previous sampling period) p(kT ) is the calculated packet mark probability qdes is the specified desired TQL. SGi1,2 (kT ) and SGo (kT ) are the input and output scaling gains, respectively.

In fuzzy control theory, the range of values of inputs or outputs for a given controller is usually called the ”universe of discourse”. Often, for greater flexibility in fuzzy controller implementation, the universe of discourse for each process input is ”normalized” to the interval [−1, +1] by means of constant scaling factors [38]. For FEM controller design, the scaling gains SGi1 (kT ), SGi2 (kT ) and SGo (kT ), shown in Fig. 3, are employed to normalize the universe of discourse for the controller inputs error e(kT ) and e(kT − T ), and for the controller output p(kT ), respectively. The input gains SGi1,2 (kT ) are chosen so that the range of values of SGi1 (kT )×e(kT ) and SGi2 (kT )×e(kT −T ) lie on [−1, 1], and SGo (kT ) is chosen by using the allowed range of inputs to the plant in a similar way. The range of values of the controller’s output lies between 0 and 1 (i.e., p(kT ) ∈ [0, 1]). In order to achieve a normalized range of the FEM input variables from −1 to 1, the input scaling gain SGi1 (kT ) is set as shown in (2). 1 − qdes −Buf if e(kT ) < 0 f erSize SGi1 (kT ) = (2) 1 otherwise qdes The SGi1 (kT ) values are taken by considering the lower and upper bounds of the queue length. When the instantaneous queue length takes its maximum value (i.e., is equal to the buffer size), then the error on the queue length e(kT ) (see (1)) should have its minimum value of qdes − Buf f erSize. On the other hand, when the instantaneous queue length takes its minimum value, that is,


219

zero, then the error on the queue length has its maximum value that is equal to qdes . Similarly for SGi2 (kT ), however, e(kT − T ) is used. The output scaling gain SGo (kT ) is determined so that the range of outputs that is possible is the maximum, as well as to ensure that the input to the plant will not saturate around the maximum. Following the approach in [19], SGo (kT ) is dynamically set to a value indicating the maximum mark probability (initially set to e.g. a value of 0.1) in response to changes of the instantaneous queue length, q(kT ), as shown in (3). ⎧ ⎨ SGo (kT − T ) + 0.01 if q(kT ) > 1.1T QL and SGo (kT ) < 0.5 SGo (kT )= SGo (kT − T ) × 0.01 if q(kT ) < 0.9T QL and SGo (kT ) > 0.01 ⎩ SGo (kT − T ) otherwise (3) Rule Base Selection The multi-input fuzzy inference engine uses linguistic rules to calculate the mark probability. These linguistic rules form the control knowledge - rule base of the controller and provide a description of how best to control the system, under differing operating conditions. Hence, linguistic expressions are needed for the inputs and the output, and the characteristics of the inputs and the output. Linguistic variables are used (that is, symbolic descriptions of what are in general time-varying quantities) to describe fuzzy system inputs and output. The linguistic variables take on ”linguistic values” that change dynamically over time and are used to describe specific characteristics of the variables. Linguistic values are generally descriptive terms such as ”positivebig”, ”zero” and ”negative-small”. The linguistic variables and values provide a language to express the authors’ ideas about the control decision-making process in the context of the framework established by the authors’ choice of FEM controller inputs and output. In order to determine the linguistic values of the input and output variables, the authors define partitions over the input and output space so that they will adequately represent the linguistic variables. Since the inputs of the FEM controller deal with the queue evolution, which is dynamic and time-varying in nature, the authors need to have as ”many” operating regions - state partitions as possible, in order to capture as much detail of the dynamics and the nonlinearities of the TCP/IP plant. The authors also need to keep the controller as simple as possible by not increasing the number of linguistic values - state partitions beyond a number, which offers insignificant improvement on the plant performance. The same applies for the output of the FEM controller, the mark probability. The model of the FEM control system, comprising the control rules and the values of the linguistic variables, is obtained through an offline intuitive tuning process that starts from a set of the initial insight considerations and progressively modifies the number of linguistic values of the system until it reaches a level of adequate performance. The design objective is to keep the

220


controller as simple as possible to start with, and only increase complexity, by adding more linguistic values, if required. An adequate number of linguistic values is needed to describe the nonlinear behavior of the system accurately enough. Adding more rules, as expected, increases the accuracy of the approximation, which yields an improved control performance. But beyond a certain point the improvement is marginal. By choosing the simplest MISO controller, the authors have avoided the exponential increase of the rule base, and subsequent increase in the complexity of the controller, when the number of input variables increases. A careful design of the rule base is done based on two goals: • •

Completeness: all kinds of situations of system behavior are taken into consideration, i.e., all kinds of combinations of input variables results in an appropriate output value. Consistency: The rule base does not contain any contradiction. A set of rules is inconsistent if there are at least two rules with the same antecedents-part and different consequent-part.

The philosophy behind the knowledge base of the FEM scheme is that of being aggressive when the queue length deviates from the TQL (where congestion starts to set in and quick relief is required), but on the other hand being able to smoothly respond when the queue length is around the TQL. All other rules can represent intermediate situations, thus providing the control mechanism with a highly dynamic action. A convenient way to list all possible IF-THEN control rules is to use a tabular representation (see Table 2). These rules reflect the particular view and experiences of the designer, and are easy to relate to human reasoning processes and gathered experiences. Note that the actual number of rules implemented in FEM is reduced, since when the current error on queue length is negative-very-big, then the output control signal is always huge, irrespective Table 2. FEM Linguistic rules - Rule base [25] Table Notations: negative/positive very big (NVB/PVB), negative/positive big (NB/PB), negative/positive small (NS/PS), zero (Z), huge (H), very big (VB), big (B), small (S), very small (VS), tiny (T) p(kT) NVB NB Qerror NS (kT) Z PS PB PVB

Qerror (kT-T) NVB VB NS Z PS PB PVB H B T Z Z Z Z

H B VS Z Z Z Z

H B S Z Z Z Z

H VB S T Z Z Z

H VB B VS T Z Z

H H VB S T Z Z

H H VB B VS T Z


221

(a) linguistic input variables

(b) linguistic output variable Fig. 4. Membership functions of the linguistic values representing the input variables ”normalized error on queue length for two consecutive sample periods”, and the output variable ”mark probability” [25]

of the status of the past error on queue length. The same applies when the current error on queue length is positive-very-big, then the output control signal is always zero. Membership Functions Selection The membership functions of the linguistic variables are determined by using an intuitive and pragmatic choice and not an analytic approach (this is one of the reported advantages of fuzzy logic controllers compared to the conventional counterparts). The choice of membership function shape is open. Many shapes are often found in studies (see, e.g. [38]). Due to computational simplicity, the authors select triangular or trapezoidal shaped membership functions in FEM control system. These types of shapes are a standard choice used in many industry applications due to the mathematical simplicity of

222


the expressions representing them. The selected membership functions representing the linguistic values for both the inputs and the output of the FEM controller are shown in Fig. 4. The amount of overlapping between the membership functions’ areas is significant. The left and right half of the triangular membership functions for each linguistic value is chosen to provide membership overlap with adjacent membership functions. The chosen method is simple in that symmetric-andequally spaced membership functions are used, where the sum of the grade of membership of an input value, concerning the linguistic values of a specific input variable, is always one (see (4)). m

µk (xi ) = 1

(4)

k=1

where µk (xi ) is the membership value of the input value xi taken from the membership function of the linguistic value k, (1 < k < m, where m is the number of linguistic values of a linguistic variable), of the input variable of concern. This results in having at most two membership functions overlapping, thus we will never have more than four rules on/activated at a given time. This offers computational simplicity on the implementation of the FEM controller, a design objective. The overlapping of the fuzzy regions, representing the continuous domain of each control variable, contributes to a well-behaved and predictable system operation; thus the fuzzy system can be very robust [25]. The FEM controller is a Mamdani-based model. Mamdani’s fuzzy inference method is the most commonly used fuzzy methodology [61]. This approach is adopted due to its simplicity and effectiveness. The calculated output control signal of the nonlinear fuzzy controller, shown in (5), uses the center of gravity - the most common defuzzification method [38], of the aggregated fuzzy output set C. yµ (y)dy C = pk (5) µC (y)dy where, µC (y) = max(µ1 (y), µ2 (y), ..., µN (y)) is the membership degree of y in the aggregated fuzzy set C (which is found using the max-operation over all N implicated output fuzzy sets), and N is the number of linguistic rules. The limits of integration correspond to the entire universe of discourse Y of output mark probability values, to which y belongs. To reduce computations, the output universe of discourse Y is discretized to m values, Y = {y1 , y2 , ..., ym }, which gives the discrete fuzzy centroid (see (6)). m

pk =

j=1

yj × µC (yj )

m

j=1

µC (yj )

(6)


223

Fig. 5. Control-decision surface of the fuzzy inference engine of FEM controller. The nonlinear control surface is shaped by the rule base and the linguistic values of the linguistic variables [25].

Note that the use of symmetric triangular and trapezoidal membership functions makes the computation of the equation easy. The nonlinear control-decision surface implemented by the FEM controller is shaped by the constructed rule base and the linguistic values of the inputs and output variables (see Fig. 5). This surface represents in a compact way all the information in the fuzzy controller. An inspection of this nonlinear control surface and the linguistic rules shown in Table 2 provides hints on the operation of FEM. The mark probability behaviour under the region of equilibrium (i.e., where the error on the queue length is close to zero) is smoothly calculated. On the other hand, the rules are aggressive by increasing the probability of packet marking sharply in the region beyond the equilibrium point, where congestion starts to set in and quick relief is required. Thus the inference process of FEM controller dynamically calculates the mark probability behaviour based on the two inputs. The dynamic way of calculating the mark probability by the inference process comes from the fact that according to the error of queue length for two consecutive sample periods, a different set of fuzzy rules and so inference apply. Based on these rules and inferences, the mark probability is expected to be more responsive than other AQM approaches, (as for e.g. [18], [19], [20], [21], and [22]) due to the human reasoning and the inbuilt non linearity. Fuzzy Explicit Marking In/Out (FIO) Control System Design for Diff-Serv Congestion control at the core of a Dif-Serv network benefits from AQM schemes to preferentially drop/mark packets based on the level of precedence they belong to, by giving priority to low drop precedence against high drop

224


Fig. 6. FIO system model [25]

precedence traffic. At the same time, it is significant to provide adequate QoS-centric performance objectives, in terms of bounded delays, with high link utilization and minimal losses overall. As discussed in Sect. 2.5, the existing AQM schemes for Diff-Serv congestion control show weaknesses to respond to such objectives. The authors build on FEM, the fuzzy controller designed for Best-Effort service, and investigate its extension and suitability to provide effective congestion control for Diff-Serv environments. The goals are to achieve differentiated treatment of traffic aggregates, ensuring at the same time bounded queuing delays, low losses, and high link utilization in overall; hence offering (differentiated) QoS in traffic aggregates. In accomplishing these goals, fast system response with robustness to the time-varying dynamic nature of the controlled system play a significant role and hence are important design requirements. Furthermore, low complexity is also sought. A two-level of precedence FEM controller structure is formulated (see Fig. 6), which is designed to operate on the core routers’ buffer queue, called Fuzzy Explicit Marking In/Out (FIO), where ”In” and ”Out” terms are used to distinguish packets that are classified into different precedence traffic aggregates, distinguished by the drop/mark precedence level they belong to. ”In” packets belong to the low drop/mark precedence (i.e., high-priority traffic), while the ”Out” packets belong to the high drop/mark precedence (i.e., low-priority traffic). Both high- and low-priority traffic aggregates share a FIO queue. FIO comprises of two identical FEM controllers, one for each traffic aggregate, and two different TQLs are introduced (T QLhigh and T QLlow ), on the total queue length, one for each FEM controller. The TQL for low-priority traffic (T QLlow ) is lower than the one for high-priority traffic (T QLhigh ). Therefore, low-priority packets are more likely to be marked than the high-priority ones.


225

The idea behind this is to regulate the queue at the lower TQL. In this case, the mark probability of the high-priority traffic is closer to zero, as the TQL is set higher and thus it is less likely that high priority packets will be marked. In the presence of a small amount of high-priority packets, the queue would be mostly regulated at the lower TQL and thus marking of high-priority packets would be less likely. If however, the high-priority traffic is very high in comparison to the low-priority traffic, then the queue is regulated at the higher TQL (as there is not enough low priority traffic to ensure the lower TQL is maintained). In this case the mark probability for low-priority traffic is closer to one. In either case, the lower-priority traffic is marked at a higher rate. Therefore, both differentiation as well as a bounded delay - by regulating the queue between the two TQLs, depending on the dynamic network traffic conditions - can be accomplished. It is therefore expected that FIO can achieve an adequate differentiation between the two precedence traffic aggregates in the presence of congestion, by preferentially marking the lowest-priority packets, and giving priority/preference to highpriority-tagged traffic, while controlling the queue at the predefined levels, and thus providing QoS assurances for delay, loss, and link utilization. It is noted by the authors that even though FIO is introduced with two drop/mark precedence levels, it is easy to extend the fuzzy logic control methodology to multiple drop/mark precedence, due to the generic control methodology adopted. Performance Evaluation The authors have done extensive simulations to demonstrate the effectiveness and robustness of the AQM-based nonlinear fuzzy logic control methodology implemented in both Best-Effort and Diff-Serv TCP/IP environments. A comparison is also made with other published results selecting some representative well-known AQM schemes: A-RED [19], PI [20], REM [21], and AVQ [22] for TCP/IP Best-Effort networks, and RIO [31] and TL-PI [33] in the case of TCP/IP Diff-Serv networks. The performance of the AQM schemes is evaluated using the most widely used network simulator, the Network Simulator NS-2 [58]. A number of scenarios are selected as realistic and practical as possible, which aim to stress the tested approaches. The authors use both single- and multiple-congested (tandem) links (bottlenecks) network environments (including topologies with congestion at peripheral links), and also widely differing operating conditions, in order to examine the following effects, on the AQM schemes: • • • • •

dynamic traffic changes - time-varying dynamics traffic load factor heterogeneous propagation delays different propagation delays at bottleneck links different link capacities

226

• • •


introduction of noise-disturbance (background traffic) to the network (e.g. short-lived TCP connections) introduction of reverse-path traffic different types of data streams, like TCP/FTP and TCP/Web-like traffic, as well as unresponsive traffic (UDP-like).

The performance metrics that are used for evaluating the performance of the tested AQM schemes are: • • •

Bottleneck link utilization (based on the useful throughput, also commonly called goodput) Loss rate Mean queuing delay and its standard deviation.

Due to lack of space, what follows are some indicative results. More comprehensive results can be found in [25] and [26]. Best Effort Scenario

This scenario [25] provides a realistic network topology, where multiplecongested (tandem) links occur. The network topology of a multiple-bottleneck link is shown in Fig. 7. AQM is used in the queues of all core links from router-A to router-F. All other links (access links) have a simple Drop Tail queue. The link capacities and propagation delays are set as follows: (C1, d1) = (C8, d8) = (C9, d9) = (100 M bps, 5 ms), (C2, d2)=(C4, d4)= (C6, d6)= (15 M bps, 10 ms), (C3, d3)= (15 M bps, 60 ms), (C5, d5) = (15 M bps, 30 ms), and(C7, d7) = (C10, d10) = (C11, d11) = (200 M bps, 5 ms). N 1 flows end up at destination 1, whereas N 2 flows end up at destination 2, and N 3 flows end up at destination 3 creating cross traffic. The results in [25] and [26] show that both bottleneck links, where the cross traffic exists, (i.e., between router-B and router-C, and between router-D and router-E) exhibit similar behaviour, as far as the performance comparison is concerned. Therefore, the bottleneck link between router-D and router-E is chosen to show the results obtained.

src N2 flows

src N3 flows

(C8,d8)

src N1 flows Router A

(C9,d9)

Router B

Router C

Router D

Router E

Router F iMac

(C2,d2) (C1,d1)

(C3,d3)

(C4,d4) (C10,d10) iMac

dest 2

(C5,d5)

(C6,d6) (C11,d11) iMac

dest 3

Fig. 7. Multiple-bottleneck network topology [25]

(C7,d7) dest 1

Fuzzy Logic Control in Communication Networks Buffer Size

Buffer Size

500

500 PI

450

450

400

400

350

350 Buffer Size (Packets)

Buffer Size (Packets)

FEM

300 250 200

300 250 200

150

150

100

100

50

50

0

0 0

10

20

30

40

50

60

70

80

90

100

0

10

20

30

40

Time (Seconds)

50

60

70

80

90

100

Time (Seconds)

(a) FEM

(b) PI

Buffer Size

Buffer Size

500

500 A-RED

REM

450

450

400

400

350



227

300 250 200

300 250 200

150

150

100

100

50

50

0

0 0

10

20

30

40

50

60

70

80

90

100

0

10

20

30

40

Time (Seconds)

50

60

70

80

90

100

Time (Seconds)

(c) A-RED

(d) REM Buffer Size

500 AVQ 450 400


350 300 250 200 150 100 50 0 0

10

20

30

40

50

60

70

80

90

100

Time (Seconds)

(e) AVQ Fig. 8. Best Effort Scenario: Queue lengths for bottleneck prop. delay = 200 ms [25]

The performance of AQM schemes is investigated under variation of the bottleneck link propagation delays. Specifically, the effect of the RTT is examined by increasing the propagation delay router-D and router-E from 30 ms to 120 and 200 ms, thus increasing the RTT from 260 ms to 600 ms. The number of long-lived flows is N 1 = 500, N 2 = 100, and N 3 = 200. Also timevarying dynamics are introduced on the network, by stopping half of all the flows at time t = 40 s, and resuming transmission at t = 70 s. Figure 8 shows the queue length evolution for the case of 200 ms bottleneck propagation delay. From the results, it can be observed the superior steady performance of FEM with stable queue dynamics, with graceful performance degradation

228

C. Chrysostomou and A. Pitsillides Loss Rate vs Propagation Delay 14 FEM PI A-RED REM AVQ

12

Loss Rate (%)

10

8

6

4

2

0 20

40

60

80

100

120

140

160

180

200

Propagation Delay (msec)

Fig. 9. Best Effort Scenario: Loss Rate vs Propagation Delay (bottleneck prop. delay of 30, 120, 200 ms) [25]

Utilization vs Delay Variation 100 FEM PI A-RED REM AVQ

98

Utilization (%)

96

94

92

90

88

86 20

30

40

50 60 70 80 Queuing Delay Variation (msec)

90

100

110

Fig. 10. Best Effort Scenario: Utilization vs Delay Variation (bottleneck prop. delay of 30, 120, 200 ms) [25]

as the bottleneck propagation delay increases up to a value of 200 ms (note that there is a total of 600 ms round-trip propagation delay). FEM has the highest utilization, the lowest losses and the shortest delay variation (even though for 200 ms, FEM exhibits larger variation around the TQL than in previous situations found in [25] and [26], it still behaves much better that the other schemes). PI, REM, A-RED and AVQ exhibit large queue fluctuations, and show weakness to react quickly to dynamic changes resulting in degraded utilization and high variance of queuing delay. Thus, these mechanisms are shown to be sensitive to variations of RTT within the range of interest. This is clearly illustrated in Fig. 9, where the loss rate as the propagation delay

Fuzzy Logic Control in Communication Networks Buffer Size

Buffer Size

500

500 FIO

TL-PI

450

450

400

400

350



229

300 250 200

300 250 200

150

150

100

100

50

50

0

0 0

10

20

30

40

50

60

70

80

90

100

0

10

20

30

40

Time (Seconds)

50

60

70

80

90

100

Time (Seconds)

(a) FIO

(b) TL-PI Buffer Size

500 RIO 450 400


350 300 250 200 150 100 50 0 0

10

20

30

40

50

60

70

80

90

100

Time (Seconds)

(c) RIO Fig. 11. Diff-Serv Scenario: Queue lengths (bottleneck prop. delay = 120 ms - highpriority traffic consists of 1.33% of the total traffic passing through the bottleneck link) [25]

increases is shown. FEM shows robustness by having minimal, and the lowest among the others, losses. Figure 10 shows the utilization with respect to the queuing delay variation. FEM outperforms the other AQMs, in managing to achieve high utilization, and at the same time regulating the queue and thus providing bounded mean delay, and delay variation. Diff-Serv Scenario The same network topology of multiple-bottleneck links as the one used in the Best Effort Scenario,with the same network parameters is used (see Fig. 7). The bottleneck link between router-B and router-C is chosen to show the results obtained [25]]. The performance of AQM schemes under variation of the bottleneck link propagation delays is investigated. The effect of the RTT is examined by varying the bottleneck propagation delay between router-B and router-C from 30, to60, and120ms, thus increasing the RTT from 200 ms to 380 ms. All sources (N 1 = 100, N 2 = 50, and N 3 = 100 flows) are greedy sustained File Transfer Protocol applications. A limited number of flows tagged as high-priority traffic is considered (2% of N1 flows, whereas the rest 98% are tagged as low-priority). N2 and N3 flows are considered as being of

230

C. Chrysostomou and A. Pitsillides High-priority Utilization vs Mean Delay 45 FIO TL-PI RIO

40

High-priority Utilization (%)

35 30 25 20 15 10 5 0 40

60

80

100

120

140

160

180

200

220

240

Mean Queuing Delay (msec)

Fig. 12. Diff-Serv Scenario: Utilization of high-priority traffic vs mean queuing delay (bottleneck propagation delay varies from 30, 60, and 120 ms - high-priority traffic consists of 1.33% of the total traffic passing through the bottleneck link). The TQLlow value of 100 packets is equivalent to 53, 33 ms [25].

low-priority (i.e., about 1.33% of the traffic passing through the bottleneck link, under consideration, belongs to the high-priority level). Figure 11 shows the queue length evolution for the case of 120 ms bottleneck propagation delay. From the results, the superior steady performance of FIO can be observed with stable queue dynamics, irrespective of the increase of RTT. FIO has the highest utilization, and the lowest losses and the shortest delay variation ([25], [26]). On the other hand, RIO exhibits larger queue fluctuations as the RTT increases that result in degraded utilization and high variance of queuing delay [25], [26]. Also, TL-PI suffers from a slow response to regulate the queue, and has higher delay variation than FIO has [25], [26]. Thus, these mechanisms are shown to be sensitive to variations of RTT. Figure 12 shows the utilization of the bottleneck link regarding the high-priority traffic with respect to the mean queuing delay. Despite the fact that the high-priority traffic consists only of 1.33% of the total traffic passing through the particular link, FIO outperforms the other AQMs, in managing to achieve a considerable amount of utilization for the high-priority; thus it achieves a much higher differentiation between the two drop precedence levels compared with the other schemes, and at the same time regulating the queue and thus providing bounded mean delay, and delay variation [25], [26]. On the other hand, the other schemes exhibit larger delays, and provide no differentiation between high- and low-priority traffic.

4 Conclusions Network management and control is a complex problem, which is becoming even more difficult with the increased demand to use the Internet for time/delay-sensitive applications with differing QoS requirements (e.g. Voice


231

over IP, video streaming, Peer-to-Peer, interactive games). The existing TCP congestion avoidance/control mechanisms, while necessary and powerful, are not sufficient to provide good service in all circumstances. The insufficiencies of the implicit end-to-end feedback adopted by the TCP paradigm necessitate the design and utilization of new effective congestion control algorithms, to supplement the standard TCP based congestion control, since the replacement of the current TCP congestion control algorithm does not appear to be realistic at this point in time. Further, given the need for providing adequate QoS new network architectures have been proposed, such as the Differentiated Services architecture to deliver aggregated QoS in IP networks. Basically, there is a limit to how much control can be accomplished from the edges of the network of such an end-to-end implicit feedback based congestion control. Some additional mechanisms are needed particularly in the routers to complement the endpoint congestion control methods. Thus the need for router control has recently led to the concept of active queue management (AQM). The problem of network congestion control remains a critical issue and a high priority; despite the many years of research efforts and the large number of different control schemes proposed, there are still no universally acceptable congestion control solutions. Current solutions of existing AQM mechanisms, introduced to assist the TCP congestion control, are ineffective to meet the diverse needs of today’s Internet, due to the dynamic, time-varying nature of TCP/IP networks. It is widely accepted that they have serious limitations and drawbacks. Thus, despite the classical control system techniques used from various researchers, these still do not perform sufficiently to control the dynamics, and the nonlinearities of the TCP/IP networks. Given the need to capture such important attributes of the controlled system, the design of robust, intelligent control methodologies is required. Hence, given the need for such control methodology - to capture the dynamics, the highly bursty network traffic, and the nonlinearities of the TCP/IP system, under widely differing operating conditions - we show the usefulness of fuzzy logic control to meet such objectives. Fuzzy Logic Control can be considered as suitable candidate for AQM-based control mechanism due to its reported strength in controlling nonlinear systems using linguistic information. The capability to qualitatively capture the attributes of a control system based on observable phenomena is a main feature of fuzzy logic control and has been demonstrated in various places in the research literature as well as in commercial products. The main idea is that if the fuzzy logic control is designed with a good (intuitive) understanding of the system to be controlled, the limitations due to the complexity system’s parameters introduced on a mathematical model can be avoided. Therefore, the application of fuzzy control techniques to the problem of congestion control in TCP/IP networks is worthy of investigation, due to the difficulties in obtaining a

232


precise enough mathematical model (amicable to analysis) using conventional analytical methods, while some intuitive understanding of congestion control is available. We have presented illustrative examples of using fuzzy logic to control congestion. These and the literature we review on fuzzy logic methods applied to networks show that fuzzy logic can be effective in the control of congestion. There is no doubt that we will see more and more use of these techniques, including new challenging networking areas (e.g., sensor networks, 3G and beyond mobile networks, etc). We also expect that, as in other commercial products, fuzzy logic techniques will finally make it into real products in this area, and we expect with tremendous success. Of course many challenges to the control of congestion using fuzzy logic remain unresolved. Much work remains for the analytical study of fuzzy logic, particularly in the area of stability and performance analysis. Most proposed fuzzy logic controllers in literature do not have any stability analysis because of the difficulty in analysis. This is mainly due to the existence of the nonlinearity in the control structure that usually makes it difficult to conduct theoretical analysis to explain why fuzzy logic controllers in many instances achieve better performance than the conventional counterparts, especially for highly nonlinear processes. However, as elegantly pointed out by Mamdani [62], overstressing the necessity of mathematically derived performance evaluations may be counter productive and contrary to normal industry approach (e.g. prototype testing may suffice for accepting the controlled systems performance). Nevertheless, a certain degree of safety concerning fuzzy logic applied in networks can be examined.

References 1. ICCRG, Internet Congestion Control Research Group (2006), http://oakham. cs.ucl.ac.uk/mailman/listinfo/iccrg 2. Keshav, S.: Congestion Control in Computer Networks. Ph.D. Thesis, University of California Berkeley (1991) 3. Yang, C.Q., Reddy, A.V.S.: A taxonomy for congestion control algorithms in packet switching networks. IEEE Network Magazine 9(4), 34–45 (1995) 4. Pitsillides, A., Sekercioglu, A.: Congestion Control. In: Pedrycz, W., Vasilakos, A. (eds.) Computational Intelligence in Telecommunications Networks, pp. 109– 158. CRC Press, Boca Raton (2000) 5. Hassan, M., Sirisena, H.: Optimal control of queues in computer networks. In: IEEE International Conference on Communications, vol. 2, pp. 637–641 (2001) 6. Andrews, M., Slivkins, A.: Oscillations with TCP-like flow control in networks of queues. In: IEEE Infocom 2006, pp. 1–12 (2006) 7. Schwartz, M.: Telecommunication networks: Protocols, modelling, analysis. Addison-Wesley, Reading (1988) 8. Chiu, D.M., Jain, R.: Analysis of the increase and decrease algorithms for congestion avoidance in computer networks. Computer Networks and ISDN Systems 17, 1–14 (1989)


233

9. Ramakrishnan, K., Floyd, S., Black, D.: The addition of explicit congestion notification (ECN) to IP. Request for Comments RFC 3168, Internet Engineering Task Force (2001) 10. Jacobson, V.: Congestion avoidance and control. In: ACM SIGCOMM 1988, pp. 314–329 (1988) 11. Katabi, D., Handley, M., Rohrs, C.: Congestion control for high bandwidthdelay product networks. In: ACM SIGCOMM 2002, vol. 32(4), pp. 89–102 (2002) 12. Stevens, W.: TCP slow start, congestion avoidance, fast retransmit, and fast recovery algorithms. Request for Comments RFC 2001, Internet Engineering Task Force (1997) 13. Lakshman, T.V., Madhow, U.: The performance of TCP/IP for networks with high bandwidth delay products and random loss. IEEE/ACM Transactions on Networking 5, 336–350 (1997) 14. Kurose, J.F., Ross, K.W.: Computer networking: a top-down approach featuring the Internet. Addison-Wesley, Reading (2005) 15. Floyd, S., Henderson, T., Gurtov, E.A.: The NewReno modification to TCP’s fast recovery algorithm. Request for Comments RFC 3782, Internet Engineering Task Force (2004) 16. Mathis, M., Mahdavi, J., Floyd, S., Romanow, A.: TCP Selective Acknowledgement options. Request for Comments RFC 2018, Internet Engineering Task Force (1996) 17. Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering, S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G., Partridge, C., Peterson, L., Ramakrishnan, K., Shenker, S., Wroclawski, J., Zhang, L.: Recommendations on queue management and congestion avoidance in the Internet. Request for Comments RFC 2309, Internet Engineering Task Force (1998) 18. Floyd, S., Jacobson, V.: Random early detection gateways for congestion avoidance. IEEE/ACM Trans on Networking 1(4), 397–413 (1993) 19. Floyd, S., Gummadi, R., Shenker, S.: Adaptive RED: An Algorithm for Increasing the Robustness of RED’s Active Queue Management. Technical report, ICSI (2001) 20. Hollot, C.V., Misra, V., Towsley, D., Gong, W.B.: Analysis and Design of Controllers for AQM Routers Supporting TCP Flows. IEEE Transactions on Automatic Control 47(6), 945–959 (2002) 21. Athuraliya, S., Li, V.H., Low, S.H., Yin, Q.: REM: Active Queue Management. IEEE Network Magazine 15(3), 48–53 (2001) 22. Kunniyur, S., Srikant, R.: An adaptive virtual queue (AVQ) algorithm for active queue management. IEEE/ACM Transactions on Networking 12(2), 286–299 (2004) 23. May, M., Bolot, J., Diot, C., Lyles, B.: Reasons Not to Deploy RED. In: 7th International Workshop on Quality of Service, pp. 260–262 (1999) 24. Plasser, E., Ziegler, T.: A RED Function Design Targeting Link Utilization and Stable Queue Size Behaviour. Computer Networks Journal 44, 383–410 (2004) 25. Chrysostomou, C., Pitsillides, A., Sekercioglu, A.: Fuzzy Explicit Marking: A Unified Congestion Controller for Best Effort and Diff-Serv Networks. Computer Networks Journal (accepted for publication) (2008) 26. Chrysostomou, C.: Fuzzy Logic Based AQM Congestion Control in TCP/IP Networks. PhD Thesis, University of Cyprus (2006), http://www.netrl.cs. ucy.ac.cy/images/thesis/chrysostomou-phd-thesis-sep06.pdf

234


27. Guirguis, M., Bestavros, A., Matta, I.: Exogenous-Loss Awareness in Queue Management - Towards Global Fairness. Techical Report, Computer Science Departrment, Boston University (2003) 28. Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., Weiss, W.: An architecture for Differentiated Services. Request For Comments RFC 2475, Internet Engineering Task Force (1998) 29. Babiarz, J., Chan, K., Baker, F.: Configuration guidelines for DiffServ service classes. Request for Comments RFC 4594, Internet Engineering Task Force (2006) 30. Heinanen, J., Baker, F., Weiss, W., Wroclawski: Assured Forwarding PHB Group. Request for Comments RFC 2597, Internet Engineering Task Force (1999) 31. Clark, D., Fang, W.: Explicit Allocation of Best Effort Packet Delivery Service. IEEE/ACM Transactions on Networking 6(4), 362–373 (1998) 32. May, M., Bolot, J.C., Jean-Marie, A., Diot, C.: Simple perfomance models of differentiated services schemes for the Internet. In: IEEE INFOCOM 1999, New York, pp. 1385–1394 (1999) 33. Chait, Y., Hollot, C.V., Misra, V., Towsley, D., Zhang, H., Lui, C.S.: Providing throughput differentiation for TCP flows using adaptive two-color marking and two-level AQM. In: IEEE INFOCOM 2002, New York, vol. 2, pp. 837–844 (2002) 34. Misra, V., Gong, W.B., Towsley, D.: Fluid-based Analysis of a Network of AQM Routers Supporting TCP Flows with an Application to RED. In: ACM SIGCOMM 2000, pp. 151–160 (2000) 35. Hollot, C.V., Misra, V., Towsley, D., Gong, W.B.: A control theoretic analysis of RED. In: IEEE Infocom 2001, vol. 3, pp. 1510–1519 (2001) 36. Sekercioglu, A., Pitsillides, A., Vasilakos, A.: Computational intelligence in management of ATM networks. Soft Computing Journal 5(4), 257–263 (2001) 37. Azvine, B., Vasilakos, A.: Application of soft computing techniques to the telecommunication domain. In: Tselentis, G. (ed.) ERUDIT Roadmap, pp. 89– 110 (2000) 38. Passino, K., Yurkovich, M.: Fuzzy Control. Prentice Hall, Englewood Cliffs (1998) 39. Zadeh, L.A.: Fuzzy Sets. Information and Control 8, 338–353 (1965) 40. Zadeh, L.A.: Outline of a new approach to the analysis of complex systems and decision processes. IEEE Transactions on Systems, Man, and Cybernetics 3(1), 28–44 (1973) 41. Mamdani, E.H.: Applications of fuzzy algorithms for simple dynamic plant. Proceedings of IEE 121(12), 1585–1588 (1974) 42. Morales, E., Polycarpou, M., Hemasilpin, N., Bissler, J.: Hierarchical Adaptive and Supervisory Control of Continuous Venovenous Hemofiltration. IEEE Transactions on Control Systems Technology 9(3), 445–457 (2001) 43. Sekercioglou, A., Pitsillides, A., Egan, G.K.: Study of an adaptive fuzzy controller based on the adaptation of relative rule weights. In: Proceedings of ANZIIS 1994, Brisbane, Queensland, Australia, pp. 204–208 (1994) 44. Pitsillides, A., Sekercioglou, A., Ramamurthy, G.: Effective Control of Traffic Flow in ATM Networks Using Fuzzy Explicit Rate Marking (FERM). IEEE Journal on Selected Areas in Communications 15(2), 209–225 (1997) 45. Douligeris, C., Develekos, G.: A fuzzy logic approach to congestion control in ATM networks. In: IEEE ICC 1995, Washington, USA, pp. 1969–1973 (1995)


235

46. Rossides, L., Sekercioglu, A., Kohler, S., Pitsillides, A., Phuoc, T.G., Vassilakos, A.: Fuzzy Logic Controlled RED: Congestion Control for TCP/IP Diff-Serv Architecture. In: 8th European Congress on Intelligent Techniques and Soft Computing, Aachen, Germany, pp. 263–269 (2000) 47. Rossides, L., Chrysostomou, C., Pitsillides, A., Sekercioglu, A.: Overview of Fuzzy-RED in Diff-Serv Networks. In: Bustard, D.W., Liu, W., Sterritt, R. (eds.) Soft-Ware 2002. LNCS, vol. 2311, pp. 1–13. Springer, Heidelberg (2002) 48. Chrysostomou, C., Pitsillides, A., Rossides, L., Polycarpou, M., Sekercioglu, A.: Congestion Control in Differentiated Services Networks using Fuzzy-RED. IFAC Control Engineering Practice (CEP) Journal 11(19), 1153–1170 (2003); special Issue on Control Methods for Telecommunication Networks 49. Fengyuan, R., Yong, R., Xiuming, S.: Design of a fuzzy controller for active queue management. Computer Commmunications 25, 874–883 (2002) 50. Wang, C., Li, B., Sohraby, K., Peng, Y.: AFRED: An adaptive fuzzy-based control algorithm for active queue management. In: 28th IEEE International Conference on Local Computer Networks (LCN 2003), pp. 12–20 (2003) 51. Aul, Y.H., Nafaa, A., Negru, D., Mehaoua, A.: FAFC: Fast adaptive fuzzy AQM controller for TCP/IP networks. In: IEEE Globecom 2004, vol. 3, pp. 1319–1323 (2004) 52. Di Fatta, G., Hoffmann, F., Lo Re, G., Urso, A.: A Genetic Algorithm for the Design of a Fuzzy Controller for Active Queue Management. IEEE Transactions on Systems, Man, and Cybernetics, Special Issue on Computational Intelligence in Telecommunications Networks and Internet Services: Part I 33(3), 313–324 (2003) 53. Siripongwutikorn, P., Banerjee, S., Tipper, D.: Adaptive bandwidth control for efficient aggregate QoS provisioning. In: IEEE Globecom 2002, vol. 3, pp. 2435–2439 (2002) 54. Habetha, J., Walke, B.: Fuzzy rule-based mobility and load management for self-organizing wireless networks. International journal of wireless information networks 9(2), 119–140 (2002) 55. Wang, C., Li, B., Hou, Y.T., Sohraby, K., Lin, Y.: LRED: A Robust Active Queue Management Scheme Based on Packet Loss Ratio. In: IEEE Infocom 2004, vol. 1, pp. 1–12 (2004) 56. Savoric, M.: Fuzzy explicit window adaptation: a method to further enhance TCP performance. Technical Report TKN-03-010, Telecommunication Networks Group, Technical University Berlin (2003) 57. Oliveira, R., Braun, T.: A delay-based approach using fuzzy logic to improve TCP error detection in ad hoc networks. In: IEEE Wireless Communications and Networking conference, Atlanta, USA, vol. 3, pp. 1666–1671 (2004) 58. Network Simulator (1989), http://nsnam.isi.edu/nsnam/ 59. Chrysostomou, C., Pitsillides, A., Hadjipollas, G., Polycarpou, M., Sekercioglu, A.: Fuzzy Logic Control for Active Queue Management in TCP/IP Networks. In: 12th IEEE Mediterranean Conference on Control and Automation Kusadasi, Aydin, Turkey, 6 pages (2004) (CD ROM Proceedings) 60. Chrysostomou, C., Pitsillides, A., Hadjipollas, G., Polycarpou, M., Sekercioglu, A.: Congestion Control in Differentiated Services Networks using Fuzzy Logic. In: 43rd IEEE Conference on Decision and Control, Bahamas, pp. 549–556 (2004) (CD ROM Proceedings - ISBN: 0-7803-8683-3, IEEE Catalog Number: 04CH37601C)

236


61. Mamdani, E.H., Assilian, S.: An experiment in linguistic synthesis with a fuzzy logic controller. International Journal of Man-Machine Studies 7(1), 1–13 (1975) 62. Mamdani, E.H.: Twenty years of fuzzy logic: experiences gained and lessons learned. In: IEEE International conference on fuzzy systems, San Franscisco, pp. 339–344 (1975) 63. Andrew, L.H., Hanly, S.V., Chan, S., Cui, T.: Adaptive Deterministic Packet Marking. IEEE Comm. Letters 10(11), 790–792 (2006) 64. Thommes, R.W., Coates, M.J.: Deterministic packet marking for time-varying congestion price estimation. IEEE/ACM Transactions on Networking 14(3), 592–602 (2006) 65. Liu, S., Basar, T., Srikant, R.: Exponential-RED: A Stabilizing AQM Scheme for Low- and High-Speed TCP Protocols. IEEE/ACM Transactions on Networking 13(5), 1068–1081 (2005) 66. Ariba, Y., Labit, Y., Gouaisbaut, F.: Design and Performance Evaluation of a State-Space Based AQM. In: International Conference on Communication Theory, Reliability, and Quality of Service, pp. 89–94 (2008)

Adaptation in Classification Systems Abdelhamid Bouchachia Group of Software Engineering & Soft Computing, Dept. of Informatics, University of Klagenfurt, Austria [email protected]

Summary. The persistence and evolution of systems essentially depend of their ability to self-adapt to new situations. As an expression of intelligence, adaptation is a distinguishing quality of any system that is able to learn and to adjust itself in a flexible manner to new environmental conditions. Such ability ensures selfcorrection over time as new events happen, new input becomes available, or new operational conditions occur. This requires self-monitoring of the performance in an ever changing environment. The relevance of adaptation is established in numerous domains and by versatile real world applications. The primary goal of this contribution is to investigate adaptation issues in learning classification systems form different perspectives. Being a scheme of adaptation, life long incremental learning will be examined. However, special attention will be given to adaptive neural networks and the most visible incremental learning algorithms (fuzzy ARTMAP, nearest generalized exemplar, growing neural gas, generalized fuzzy minmax neural network, IL based on function decomposition) and their adaptation mechanisms will be discussed. Adaptation can also be incorporated in the combination of such incremental classifiers in different ways so that adaptive ensemble learners can be obtained too. These issues and other pertaining to drift will be investigated and illustrated by means of a numerical simulation.

1 Introduction The continuity of systems rests on their ability to adapt to new situations. In real life, such an ability is one of the key features of any living organism and can be seen as an expression of intelligence. Undoubtedly, adaptation should feature any system that is able to adjust itself in a flexible manner to new environmental conditions through self-correction over time as new events happen, new input becomes available, or new operational conditions occur. This implies a continuous improvement and at least non-degradation of the system performance in an ever changing environment. Hence, building adaptive systems that are able to deal with nonstandard settings of learning and which are flexible in interacting with their environment at any time in an open-ended cycle of learning is an eminent research issue. Adaptation is particularly manifest in intelligent applications where learning from data is at the heart of system modeling and identification. The goal is to cope with non-stationary changing situations by employing adaptive mechanisms A.-E. Hassanien et al. (Eds.): Foundations of Comput. Intel. Vol. 2, SCI 202, pp. 237–258. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com

238

A. Bouchachia

Fig. 1. Steps of the adaptive incremental learning

to accommodate changes in the data. This becomes more important when storage capacities (memory) are very limited and when data arrives over long periods of time. In such situations, the system should adapt itself to the new data samples which may convey a changing situation and at the same time should keep in memory relevant information that had been learned in the remote past. In this contribution, we aim at studying one of the fundamental aspects of adaptation , that is, adaptive incremental learning (AIL) which seeks to deal with data arriving over time or with (static) huge amounts of data that exceed the storage capacities. Thus, processing of data at once is not feasible. Most of the available literature on machine learning reports on learning models that are one-shot experience and, therefore, lack adaptation . Therefore, learning algorithms with an adaptive incremental learning ability are of increasing importance in many nowadays on-line data streams and time series applications (e.g., text streams, video streams, stock market indexes, user profile learning, computer intrusion, etc.) but also discrete data-starved applications where the acquisition of training data is costly and requires much time. An illustrative scenario would be that typifying the particular case of running expensive chemical or physical experiments that my take long periods of time in order to collect a training sample. The concept of AIL we are interested in pertains to classification and clustering, though the focus is more on the first aspect. In such a context, the issue bears on devising adaptive learning mechanisms to induce new knowledge without ’catastrophic forgetting’ and/or to refine the existing knowledge. This raises the question of how to adaptively accommodate new data in an incremental way while keeping the system under use. Figure 1 illustrates the type of frameworks considered in this work. In particular, this paper we explore, among others, the efficiency of some neural learning systems that are implicitly based on neural constructivism ideas [32][37]. Investigations based on these ideas have allowed to understand how the lateral connectivity in the cortex have emerged and, more importantly, how the cortex can be seen as a continuously-adapting dynamic system formed by competitive and

Adaptation in Classification Systems

239

cooperative lateral interactions. In other words, the cortex is shaped through the dynamic and adaptive interaction between the neural progressive evolution and the neural activity owed to the environment. Following these cognitive and biological motivations, the neural learning systems discussed here are classification oriented architectures that suggest an adaptive incrementality based on algorithmic tuning. From a classification and clustering perspective, it is worth defining the notion of incrementality since the literature review shows that this term is often used in many different contexts and is subject to confusion. A classification learning system is said to be incremental if it has the following characteristics: • • • • • • • •

Ability of on-line (life-long) learning, Ability to deal with the problem of plasticity and stability, Once processed, there is no capacity to save individual historical data points in subsequent stages, No prior knowledge about the topological structure of the neural network is needed, Ability to incrementally tune the network’s structure, No prior knowledge about the data and its statistical properties is needed, No prior knowledge about the number of classes and prototypes per class is required, and No particular prototype initialization is required.

All incremental learning algorithms are confronted with the plasticity-stability dilemma. This dilemma establishes the tradeoff between catastrophic interference (or forgetting) on one hand and the ability to incrementally and continually accommodate new knowledge in the future whenever new data samples become available. The former aspect is referred to as stability, while the latter is referred to as plasticity. In a nutshell, the stability-plasticity dilemma is concerned with learning new knowledge without forgetting the previously learned one. This problem has been thoroughly studied by many researchers [10][14][29] [33][36]. From another perspective, incrementality assumes phenomena that evolve over time and change their known evolution schemes. This refers in the first to the problem of concept drift. To deal with such a problem, most often dedicated techniques to drift detection and handling use either full memory (e.g. the system has a memory and therefore has access to already seen data in the past) or partial memory (e.g. temporal window of data). It seems however quite appealing to investigate the problem of concept drift with no memory (i.e., data is processed online without any full or temporal storage). This paper aims at looking closely at this approach. Moreover, as we are interested in studying a collection of incremental learning algorithms, it sounds legitimate to observe adaptation from the perspective of ensemble learning. In fact, by considering such line of investigation, the aim is to achieve a 3-level adaptation mechanism: •

Adaptation due the nature of the classifiers, they are self-adaptive by construction,

240

• •

A. Bouchachia

Adaptation due to proportional (weighted) contribution of each classifier in the ensemble decision, and Adaptation due to the structural update (dynamically changing structure) of the ensemble.

Before delving into the details of each adaptation level, we highlight the structure of the paper. Section 2 describes the incremental classifiers used, their differences and similarities. Section 3 looks at the problem of ensemble classifiers before discussing the problem of concept drift and adaptation consequences in Section 4. Section 5 describes an approach that unifies ensemble learning and drift handling from the perspective of adaptation. Section 6 provides an evaluation of the various adaptation levels mentioned earlier.

2 Roadmap through AIL Algorithms There exists a certain number of incremental learning algorithms that are known to be lifelong learning algorithms. For the sake of exhaustiveness, we select five most illustrative algorithms. These include: fuzzy ARTMAP (FAM) [7][14], nearest generalized exemplar (NGE) [35], generalized fuzzy min-max neural networks (GFMMNN) [13], growing neural gas (GNG) [12][28], and incremental learning based on function decomposition (ILFD) [3]. These incremental algorithms are chosen due to their characteristics including different types of prototypes, generation mechanisms, operations on the prototypes (shrinking, deletion, growing, overlap control), noise resistance, and data normalization requirement. It is, however, important to recall that some of the algorithms require recycling over the data to achieve more stability. This will be avoided in this study so that the spirit of incremental learning as defined earlier is preserved. Table 1. Characteristics of AIL algorithms Hyperbox-based Characteristics

Point-based

FAM GFMMNN NGE GNG ILFD

Online learning y Type of prototypes hbox Generation control y Shrinking n Deletion n Overlap y y Growing Noise resistance u Sensitivity to data order y Normalization y

y hbox y y n n y y y y

Legend

n:no,

y:yes,

y y y hbox node center y y y y u u n y y n u u y u u u u u y y y y/n n y/n u:unkown


241

Table 1 shows some of the characteristics of the studied algorithms. Each of these algorithms is capable of online learning and produces a set of prototypes per class (in the case of classification). The algorithmic steps in all these algorithms are theoretically the same. A prototype is generated when the incoming data point is sufficiently dissimilar to the existing prototypes. Otherwise, an adjustment of some of the existing prototypes is conducted. The first characteristic that distinguishes these algorithms is the type of prototypes. In fact, we propose to categorize them into 2 classes: hyperbox-based algorithms (HBAs) and point-based algorithms (PAs). The HBAs class includes FAM, NGE and GFMMNN. Many variations of these algorithms exist. For instance, there exist some attempts to generalize the FAM categories to different shapes. The PAs class includes GNG and ILFD. While prototypes in GNG are nodes of a graph, in ILFD they are cluster centers in the sense of radial basis function neural networks and can take different geometrical shapes depending on the type of distance used. Usually, in these IL algorithms the decision of assigning new data points to either an existing prototype or to a new prototype is based on some control parameters (or procedures) like the size of hyperboxes, the error threshold and the similarity threshold. A further observation is that all HBAs allow the hyperboxes to grow. Moreover, they all, except FAM, allow the hyperboxes to shrink. Quite appealing is the fact that the growth of the hyperboxes is controlled in GFMMNN, but not in the other HBAs, in order to avoid any overlap of hyperboxes with different labels (classes). Note also that, while PAs are equipped with prototype deletion mechanisms, HBAs are not. Worth pointing out is that all algorithms suffer from the problem of sensitivity to the presentation order of data and that not all algorithms require standardization of data. FAM and GFMMNN require data normalization to fall in [0,1], while ILFD works better with normalized data in contrast to the other algorithms which do not require any standardization. On the other hand, the desired properties of any AIL algorithm are stability, plasticity and ability to track the evolution [4]. Plasticity is already fulfilled by all HBAs and PAs since they are able to accommodate new data points. However, the way to quantify it has not been yet discussed in the relevant literature. Its importance pertains to the complexity of the model. Usually too much plasticity leads to the problem of prototype proliferation. As noted earlier, generation is controlled by some parameter(s) and the question that arises is, what is the appropriate value of such parameters? Another related issue is, how can the model distinguish between rare events and noisy data? These questions remain open issues. On the other hand stability aims at equipping the AIL algorithms with the ability of preserving the prototypes already learned. Usually the trend of the data changes and the system has to adapt in order to be consistent with the current configuration of the environment. However, this adaptation may result in forgetting some of the learned knowledge. Quantifying stability can be done through measuring amount of forgetting. This has not been clearly studied in all of the existing AIL algorithms. The third aspect that is of relevance in AIL is the ability to track changes. This is related to the problem of plasticity in the sense of distinguishing between rare events and noise. The challenge lies in scenarios where the system at a time t makes

242

A. Bouchachia

Fig. 2. Logical structure of an online learning system

a prediction p given a sample x, gets adapted afterwards several times and then at time t due to some external feedback as portrayed in Fig. 2, it turns out that p was a wrong decision. The question is then, how can AIL reconsider their wrong decisions by undoing adaptations that took place between time t and t . Current algorithms are not able to adjust the systems by re-examining old decisions. To deal with this problem and as minimum requirement, AIL systems have to be able to retrieve some of their past decisions. The present paper does not deal explicitly with such issues. However, for the sake of completeness, their reference here aims at opening new research perspectives. 2.1

Fuzzy ARTMAP

Fuzzy ARTMAP (FAM) is one of many adaptive resonance network models introduced by Grossberg and Carpenter [7][14]. FAM has some interesting properties like the fast convergence and the capacity of incremental learning. It is a supervised network which is composed of two Fuzzy ARTs which are identified as ARTa and ARTb. ARTa consists of two layers of neurons, the input layer F1 and the output layer F2 which are fully interconnected. A neuron of F2 represents one prototype formed by the network and is characterized by its weight vector. It learns by placing hyperboxes in the 2*M , where M is the size of layer F1 . Each prototype is defined by a box. The first M positions of a weight vector (prototype) represent one corner of the box (the closest to the origin) and the other M positions represent the opposite one. When an input is presented to the network, it goes through the following steps. First, the smallest box including the input is selected using a choice function which checks if the box is a fuzzy subset of the input. If no such box exist, either the one which needs to be the less expanded to enclose the point is selected or a new one is created. Once a neuron is selected, a vigilance criterion is checked. It serves for choosing another box if the one selected is already too large compared to a vigilance parameter ρ. If the criterion is satisfied, the network learns the input, otherwise it selects the next neuron with the highest choice function and reevaluates the vigilance criterion. These two steps are repeated until the vigilance criterion is met. It is said that the network is in resonance and all neurons of F2 are set to 0 except the winning neuron which is set to 1. Then, ARTb compares the mapped class of the input with the actual input’s label. If the labels are different, ARTa creates a new prototype and connects it to the input’s label.


2.2

243

Nearest Generalized Exemplar

Similar to ART, NGE uses class exemplars in the form of hyperboxes to perform classification. During training, hyperboxes are incrementally either created, shrunk or expanded along one or more dimensions. Specifically, the new input is first matched against each of the existing hyperboxes. This is done as follows. If the sample lies outside of all existing hyperboxes, a weighted Euclidean distance is computed. In this distance measure, both features and hyperboxes are weighted. Initially the weight, wh , of each hyperbox, h, is set to 1 and is updated incrementally. At each step: wh is set to A/B where A is the number of times h has been used and B is the number of times h has made a correct classification. Likewise, each feature is weighted and is initially set to 1. This latter can be either increased or decreased depending on its contribution to the correct classification of samples. If the sample falls inside a hyperbox, its distance to that hyperbox is simply zero. If the sample is equidistant to several hyperboxes, the smallest of these is chosen. Once the nearest hyperbox is found, its label is checked against that of the new input. If they have the same label, the hyperbox is extended to include the input sample. If they have distinct labels, the second nearest hyperbox is searched and its label is again compared with that of the input sample. If they meet, the hyperbox is updated. Should both the first and second nearest hyperboxes have different labels than the new sample, a new hyperbox is created and will consist solely of the input sample. Note that in [42], a data normalization step is executed ahead of the training. 2.3

Growing Neural Gas

Growing neural gas (GNG) [12] generates prototypes within the data space by building a graph that consists of a set of nodes and a set of connections. Each connection is characterized by an age and each node is characterized by a local error variable. Given a new input x, the first winner, n1 and the second winner, n2 are determined. If there is no connection between these two nodes, then it is created. The connection age is set to 0. The squared distance between the first winner and the input is added to a local error variable. The weight vectors of the first winner and its direct topological neighbors are adapted by fractions η1 and ηi of the distance to x. The age of all connections leading from the winner neuron are increased by 1. Connections with age greater than a given max, are then removed. If some neurons become disconnected, then they are also removed. If the number of processed inputs reached an integer multiple of given parameter λ, a new neuron q is inserted between the neuron n with the largest error and the neuron m among the neighbors of n with the largest error. The weight of q is set to the mean value of the n and m weights. A connection between q and each of n and m is set and that between n and m is removed. Error of n and m is decreased by a given fraction, while that of the new node is set to the mean value of the errors of n and m. Then, the error of all neurons is decreased by a given fraction. The process is repeated until a predefined termination condition is met which can be the size of the network or any performance measure.

244

2.4

A. Bouchachia

Generalized Fuzzy Min-Max Neural Network

The generalized fuzzy min-max neural network (GFMMNN) [13] is a classifier belonging to the class of HBAs. GFMMNN is a neural network that consists of 3 layers: F1 , F2 , and F3 . The input layer, F1 , consists of 2 ∗ n processing nodes, twice as many dimensions as of the input. The hidden layer, F2 consists of nodes in the form of hyperbox fuzzy set. These nodes are created during training. The connections between F1 and F2 represent the min-max points of hyperboxes, while the transfer function of the F2 nodes is the hyperbox membership function. The min and max points are stored in the matrices V and W respectively. The connections are adjusted using a learning algorithm. The layer, F3 , is the class layer. Learning consists of three steps: hyperbox expansion, hyperbox overlap test, and hyperbox contraction. Given an input, x, expansion aims at identify the hyperbox that can be expanded. If such a hyperbox does not exist, a new hyperbox is created which will be consisting of x. An expandable hyperbox has to satisfy three conditions: (1) providing the highest degree of membership, (2) representing the same class as the actual input and (3) meeting an expansion criterion which is a function of the hyperbox size (that is θ). The overlap test aims at checking the existence of overlap between hyperboxes from different classes. Contraction is executed if overlap exists. It aims at eliminating that overlap using an adjustment procedure. 2.5

Incremental Learning via Function Decomposition

ILFD [3] aims at enabling lifelong classification of data lying in different regions of the space allowing to generate non-convex and disconnected partitions. Each class is approximated by a certain number of prototypes centered around their prototypes. ILFD tries to infer a classification rule that is obtained by composition of 2 functions. The first function, G, mapping the input X onto prototypes W is called a clustering function. The second function H mapping the prototypes onto class labels Y is called a labeling function. These functions are realized by a 3-layered neural network: input layer, prototype layer and class layer. The topology of the network is updated either by adding new prototype nodes, adding new class nodes or by deleting redundant and dead units of L(2) . A prototype node is redundant if it can be covered by other nearby nodes. A prototype node is dead if it remains inactive for a long period. Deletion is controlled by two checks called dispersion test and staleness test. Given a new input xk with a label yi , if no class node of that label is available, then a new prototype node j and a corresponding new class node i are inserted in the net and the weight inter-layers are accordingly adjusted. If the label of the new input is known, this input is compared against all prototypes of the same label as xk . If no prototype is sufficiently close to xk , a new prototype Pji is created and weights are accordingly adjusted. If there is a matching prototype node, the weights Wj are updated using the rival penalized competitive learning rule. The idea here is to move the competitor with a different label away from the region of the input vector so that the classes of the competing winners are kept as distinct as possible.


245

3 Combination of the AI Learners Although many of the classifiers are proven to be universal non-linear function approximators (e.g., radial basis function, multi-layer perceptron, etc.), due to the diversity and the definition range of their parameters, their performance may vary strongly. To alleviate the effect of parameter setting, it seems appealing to combine in a symbiotic way several classifiers. The idea is that even if the performance of one or few neural networks may not be that much satisfactory, the ensemble of the algorithms can still predict the correct output. Usually, when the task is relatively hard, multiple predictors are used following the conquer-and-divide principle [25, 45, 18, 17, 26, 20].

Fig. 3. Combining online learners

It is important to note that there are two combination schemes: 1. The individual classifiers (based on the same model) are trained on different data sets randomly generated sets (re-sampling from a larger training set) before they are combined to perform the classification. These include stacking [44], bagging [5] and boosting [11]. 2. The ensemble contains several classifiers trained on the same data but they are of different types (neural networks, decision trees , etc.), with different parameters (e.g. in multi-layer neural networks: different number of hidden layers, different number of hidden neurons, etc.), and trained using different initial conditions (e.g. weight initialization in neural networks, etc.) [25, 8].

246

A. Bouchachia Table 2. Combination rules

Output type Class label

Actual output

Rule

Expression

Majority voting

D(x) = arg maxj

Weighted maj. voting

D(x) = arg maxj

Product Rule

Oj (x) =

Sum Rule

Oj (x) =

Average Rule

Oj (x) =

Generalized average

Oj (x) =

Max Rule (optimistic)

j Oj (x) = maxN i=1 Oi (x)

Min Rule (pessimistic)

j Oj (x) = minN i=1 Oi (x)

1 N

N i=1

δ(Di (x), Cj )

N i=1

wi δ(Di (x), Cj )

N

j i=1 Oi (x) j i=1 Oi (x) N j 1 i=1 Oi (x) N

N

1 N

N i=1

Oij (x)

η η1

Both schemes seek to ensure a high diversity of the ensemble. Since the present paper is about different online algorithms, it is natural to focus on the second combination scheme. The ensemble consists of FAM, GFMMN, NGE, GNG, and ILFD. The important issue is then how to combine such classifiers taking the notion of adaptation into account. Basically there are two combination strategies as shown in Tab. 2: 1. Class labels: In this strategy, the combination rules use the individual decisions (assignments) of the N classifiers to generate the final decision of the ensemble. The most representative rule in this strategy is the voting rule and its weighted version. The class receiving the highest number of votes is retained as the decision of the ensemble. In the weighted version the contribution of each classifier is represented by a weight, wi that is obtained based on background knowledge (e.g., performance of the classifier) or via training using a separate algorithm. Note that δ(.) in Tab. 2 indicates the kroneker function. Moreover there exist other class-label based combination strategies such as BKS and Borda count [16] which are less popular. 2. Actual outputs: The decision of the ensemble can also be obtained by combining directly the outputs of each classifier rather than the labels corresponding to those outputs. The resulting outputs are then used to infer the final assignment decision (winning class). The widely used rules are those shown in Tab. 2. The most general one is the generalized mean rule that subsumes the minimum rule (when η → −∞), the geometric mean (variant of the product rule) (when η = 0), the average rule ( η = 1), and the maximum rule (when η → ∞). Because the output, Oji by a classifier i with respect to a class j corresponds to the estimate of the posterior probability P (Cj |x) for j, it is important to normalize the outputs so that they sum up to 1. To do that, usually the softmax function is applied:


247

j

Oi ˜j = e O i C k=1

k

eOi

j = 1···C

(1)

There exist other combination rules that take the assignment decisions of the classifiers during training into account. These include decision templates and Dempster-Shafer [24] which are more demanding from the computational point of view. Moreover, there has been several investigations to find the best combination rule [1, 20, 23, 40]. Although there is no general conclusion about such issue, it seems that the most admitted rules are the majority voting (for class label) and the average (continuous output) and their weighted versions. These are preferred due to their advantageous tradeoff between accuracy and computational simplicity. Therefore, in the present study, the weighted majority voting rule will be applied to combine the diverse classifiers. In addition to the previously mentioned reasons, the choice of such combination rule is motivated by the aim of incorporating adaptation at this level too. The weights stand for a tuning mechanism that is not imposed but rather learned through time as data arrives and classifiers evolve. Hence, we see weighted majority voting as a second level of adaptation. It is the second element of the hybrid approach proposed in this paper that consists of self-adaptive basic classifiers, to be combined by a dynamic combination rule and where the structure of the ensemble is dynamic too. More details will follow in the next sections.

4 Concept Drift In dynamic environments where data arrives over time, very often the data distribution drifts over time leading to performance deterioration of the off-line trained system. Particularly, the drift makes the model built using old data inconsistent with the new data. To tackle this problem, the classifier needs to be equipped with appropriate mechanisms to detect and handle concept drift. Such mechanisms help monitor the performance of the system efficiently by instantly updating the underlying model. The state-of-art techniques in the context of concept drift are rather data driven, meaning that drift is handled only from the perspective of data. Several techniques based on this strategy exist: (i) instance weighting, (ii) instance selection, and (iii) ensemble learners. The idea underlying instance weighting [43] consists of decreasing with time the importance of samples. It is, however, hard to decide which instances should be assigned higher weights, though some heuristics using aging and typicality can be devised. Independently of drift handling, such an idea underlies some incremental classifiers discussed here. These include ILFD and GNG. Instance selection [21] is the best known technique and includes two methodologies: fixed window and adaptive window where the model is regenerated using the last data batches (the system possesses a memory). The challenge in instance selection technique is finding the appropriate window. This can be seen from the perspective of forgetting in that the classifier is adjusted over time by learning the

248

A. Bouchachia

new data and forgetting/unlearning outdated knowledge. As outlined in [43], the issue is quantifying the characteristics of forgetting, that is, the rate and the type of the drift (gradual, seasonal, abrupt). Often forgetting is simulated by re-training [34] the classifiers on a window of length n containing the new sample and the n − 1 old samples. However the size of the window is a critical issue. Small window leads to low stability due the few samples used to train the model and large window might lead to less responsiveness to the change. Simulating forgetting can also be done using an adaptive window whose size changes overtime. If a drift is detected, the window is downsized. But even with that, it is difficult to accurately modify the size of the window according to the pace of drift. The last known type of forgetting is density-based [38] where instances are deleted from the learning set if they are already reflected by the local regions (local models). This results in constant update of the high-density regions, while not removing the data representing rare events. More relevant to the present study is the ensemble learners 1 technique, especially incremental ensembles, according to which the idea of instance selection is generalized so that many classifiers vote. Their weight is changed so that the successful classifiers that detect drift are rewarded and/or those which do not detect drift are replaced by new ones trained on newly arriving data. Other ensemble-based approaches seek to learn a set of concept descriptions over different time intervals [22, 19, 39, 41]. Relying on weighted voting, the predictions of the learners are combined and the most relevant description is selected. To simplify the categorization provided by Kuncheva [24], ensemble learning algorithms for changing environments can be classified into one of the following three types: (i) dynamic combination where the base learners are trained in advance and dynamically combined to respond to the changes in the environment by changing the combination rule (Winnow and weighted majority are often used [27, 41, 43]); (ii) continuous update of the learners such that the learners are either retrained in batch mode or updated online using the new data [6, 9, 30] (the combination rule may or may not change in the process); (iii) structural update where new learners are added (or existing ones are activated if they deactivated) and inefficient ones are removed (or deactivated) [22, 39] relying on accuracy criteria. To fit the context of open-ended cycle of learning, we consider a hybrid approach that unifies: dynamic combination rule for classifiers that learn online and that can be structurally updated overtime as the accuracy of the ensemble starts to deteriorate. The adaptation of the ensemble structure is further described in the next section.

5 Online Mistake-Driven Ensemble Learning Following the description line in the previous section, the next goal is to define mechanisms that allow to achieve the online learning of the individual classifiers, 1

”Ensemble learners” and ”ensemble classifiers” are used interchangeably.


249

while seeking dynamic structural update of the ensemble that ensures adaptation in dynamically changing environments (e.g. in presence of concept drift). It is therefore important to apply appropriate combination techniques that fit the current incremental context. An early work by Littlestone [27] has focused on the idea of online mistake-driven. While such work was originally proposed in the context of attribute weighing so that a more accurate classifier can be obtained, its adoption in ensemble learning is straightforward as we will see shortly. The basic Winnow mistake-driven algorithm for individual classifiers is given as: • Initialize weights w1 , ..., wn • Given an input xt with its corresponding output yt , get the predicted label from the classifier update the classifier ◦ If no mistake, then w(t + 1) = w(t) // no modification ◦ Otherwise set w(t + 1) = wt eηyt xt /Zt // Zt is a normalization factor and η is a learning rate

Like in Perceptron, the Winnow algorithm updates the weights only when an input is misclassified. Moreover, the Winnow algorithm can be seen as a weighted majority algorithm (i.e. a learner replaces the attribute in the original version of Winnow). However the update rule changes [2]. This idea is portrayed in Alg. 1. As illustrated, the weights of the individual learners are updated when the ensemble misclassify the current input following the original scheme of the Winnow algorithm. In doing that, individuals that produce the right predictions are promoted (by increasing their weights) and those that fail to predict the correct class of the current input are demoted (i.e. penalized) by decaying their weight. In the current paper, we apply the following steps: Note that n indicates the number of learners, therefore its value is set to 5 in our case. Recall, however, that all classifiers are trained while the ensemble is tuned, all online. One aspect to be added to the portrayed algorithm 1 in the context of incremental open-ended learning cycle is the removal of inefficient learners and addition of new ones. This corresponds to the third adaptation level after self-adaptation and weighted contribution of individual classifiers. As explained earlier, there exist some attempts in the context of online ensemble to change the structure of the ensemble through the operation of addition and deletion of ensemble members [22, 39]. In this study, Alg. 1 is modified resulting in the version shown in Alg. 2. While the delete operation in Step 7 of Alg. 2 is well defined the add operation in Step 9 needs full specification. The learners used in this study already ensure the desired high diversity of the ensemble, more reinforcement of such a diversity can be still achieved. To do that, if a new learner is to be appended to the ensemble at some time, its type (i.e., FAM, NGE, GFMMN, GNG, ILFD) can be decided based on the current variability of the ensemble. For each type, a learner is virtually added and the variability of the ensemble is computed. The type that produces the highest variability is effectively added. Of course one can consider another alternative, that

250

A. Bouchachia

Algorithm 1. The Winnow algorithm as a weighted majority voting algorithm 1: - Initialize weights w1 , ..., wn // n learners, the promotion parameter α (α > 1) and the demotion parameter β (0 < β < 1), the weights wi s.t. n i wi = 1 2: - Present the current input xt to each learner (the corresponding output is yt ) 3: - Get predictions from the learners (pt1 , · · · ptn ) 4: - Compute the weighted majority vote (the decision of the ensemble) n

y˜t = arg max wj [yj = ptj ] j

j=1

5: if y˜ = y then 6: for all Learners j = 1 · · · n do 7: if pj = y t then (t+1) (t) 8: - wj = wj ∗ α 9: else (t+1) (t) = wj ∗ β 10: - wj 11: end if (t+1) 12: end for wj wj = n 13: - Normalize the weights: (t+1) i=1 wj 14: end if 15: - Train each learner on the the input (xt , yt ) t

t

is, the accuracy reflected by the associated weight of each type in the current configuration of the ensemble. Such a weight can be used as a factor for deciding which type of the learners is to be instantiated. In the present paper, we focus on the latter alternative. The former one will be the subject of future work. The idea of adding and deleting learners aims at dealing with data drift. If the arriving data has a different probabilistic distribution compared to that of the data already seen, the learners might not able to handle such situations efficiently despite their adaptation capabilities. Therefore, the option of online creation of new learners and deletion of inefficient ones is highly desirable.

6 Numerical Evaluation For the sake of illustration, in this study 3 real-world data sets pertaining to classification problems are used whose characteristics are shown in Tab. 3. These sets are: defect dataset that describes software defect detection [31], breast cancer, and spam data sets from the UCI repository [15]. In order to simulate drift in the real world data sets that do not originally contain drift, we proceed by first sorting the data according to a certain feature and then we delete it from the data set. This is an easy way for appending drift to a given data set and to make that drift somehow implicit. In the case of the defect data set, the attribute representing the total McCabe’s line count of code was used to generate the drift, while in the case of spam and cancer, ”capital run length total” (representing the total number of capital letters in the e-mail) and ”radius” (representing the


251

Algorithm 2. The Winnow algorithm as a weighted majority voting algorithm 1: - Initialize weights w1 , ..., wn // n learners, the promotion α (α > 1) and the parameter 1 demotion parameter β (0 < β < 1), the weights wi1 s.t. n i wi = 1, and the maximum number of learners L 2: - Present the current input xt to each learner (the corresponding output is yt ) 3: - Get predictions from the learners (pt1 , · · · ptn ) 4: - Compute the weighted majority vote (the decision of the ensemble) n

y˜t = arg max wjt [j = ptj ] j

yj =1

5: if y˜ = y then 6: if n > L then 7: - Delete learner j with weight wjt = min[wit ]i=1···n , and set n = n − 1 8: end if 9: - Add a learner of a particular type (after some criterion) and set n = n + 1, wn = 1; 10: for all learners j = 1 · · · n − 1 // Ignoring the newly added learner do 11: if pj = y t then (t+1) (t) 12: - wj = wj ∗ α 13: else (t+1) (t) 14: - wj = wj ∗ β 15: end if (t+1) 16: end for wj wj = n 17: - Normalize the weights: (t+1) i=1 wj 18: end if 19: - Train each learner on the the input (xt , yt ) t

t

Table 3. Characteristics of the data Data Cancer Spam Defect

Size # Classes # Features 683 2 9 4601 2 57 2109 2 22

radius of the cell nucleus, that is the mean of distances from center to points on the perimeter) are respectively used to generate the drift. Note that in previous studies [34], we generated drifting data using some controlled formulas. We intend to study in this numerical evaluation, (1) evaluation of the algorithms in an incremental settings (data arrives over time and the algorithms can see the data only once), (2) their combination using the ensemble learning algorithms described by Alg. 1 (without dynamic update of the number basic classifiers), (3) their combination using Alg. 2 with dynamic update of the number basic classifiers, (4) their combination in presence of drift without update and (5) using their combination with update in presence of drift. Note that all results are averaged over 10 runs so that initial conditions have less effect on the general conclusions.

252

A. Bouchachia Table 4. Parameter settings of the networks FAM Baseline vigilance (ρa ) Vigilance of Map Field (ρab ) Choice parameter (α) GNG Learning rate - winner (ew ) Learning rate - neighbors (en ) Learning rate - output (η) Adaptation (insertion) (λ) Error decrease (all nodes) (α) Error decrease (insertion) (β) Maximal age amax

0.8 0.3 0.01 0.9 0.0001 0.08 100 0.1 0.5 300

GFMMNN Hyperbox size (θ) Sensitivity (γ)

0.1 0.05

ILFD Learning rate - winner (ew ) Learning rate - rival (en ) Confidence (R) Confusion (M ) Staleness (γ)

0.4 0.02 0.91 0.01 500

After initial experiments, we found out the values of the parameters shown in Tab. 4 to be the most fitting providing the highest classification accuracy. Note that NGE does not have any tuning parameter. 6.1

Performance of the Base Learners

Recall that training of the classifiers is done incrementally in an adaptive online manner. Hence, each of the classifiers sees a particular sample only once (i.e., classifiers have no data storage memory). Moreover, we use the accuracy ratio (the ratio of the correctly classified samples to the size of the testing set) to measure the performance of the classifiers. Considering the pre-specified parameter settings, the accuracy of the individual classifiers is computed on the testing set providing the results shown in Tab. 5. It is worth noticing that the classifiers perform differently from one data to another. The simplest data set is the cancer data while the most difficult on average is the defect data. In terms of performance, FAM seems to perform better than the others base classifiers on the average. However occasionally, the other classifiers perform better than FAM. 6.2

Performance of the Ensemble Classifiers

When training an ensemble classifiers consisting of the five basic incremental classifiers relying on Alg. 1 where the combination rule is the weighted majority voting, we obtain the results shown in Tab. 6 accompanying Tab. 5. The weight represents the contribution of each individual in making the ensemble’s decision. The higher the weight, the more importance it gets in the voting process. Having obtained the weights during the training phase, the ensemble can be evaluated on the testing data. The classification results (last column) reflect the general performance of the individual classifiers. Moreover, as one can notice from both Tabs. 5 and 6, the weights have clear effect. In the case of the cancer data


253

Table 5. Performance of individual classifiers FAM GFMMN NGE GNG ILFD Cancer 89.89 ± 0.63 94.81 ± 1.36 72.593 ± 1.72 86.67 ± 0.01 93.33 ± 1.02 Spam 86.96 ± 1.54 62.53 ± 1.92 70.91 ± 2.04 61.80 ± 0.03 72.67 ± 1.94 Defect 66.67 ± 0.63 68.57 ± 1.118 72.38 ± 1.02 79.05 ± 1.54 75.24 ± 2.66 Table 6. Performance of the ensemble classifiers (Winnow 1) - Weights and accuracy FAM GFMMN NGE Cancer 0.0564 0.1696 0.0015 Spam 0.2991 0.2181 0.1963 Defect 0.1527 0.0902 0.3192

GNG 0.0016 0.0684 0.1001

ILFD Ensemble Accuracy 0.7710 97.04 ± 1.31 0.2181 82.16 ± 0.78 0.3379 72.38 ± 0.83

Table 7. Performance of the adaptive ensemble classifiers - Winnow 2 Cancer Spam Defect Alg. type Weight Accur. Type Weight Accur. Type Weight Accuracy 1 FAM 0.0647 95.56± 0.93 NGE 0.0342 82.32± 1.23 FAM 0.0928 54.43± 3.03 2 GFMMN 0.0647 78.52± 1.24 FAM 0.0342 73.48± 1.92 GFMMN 0.0835 60.76± 1.05 3 NGE 0.0583 72.59± 1.72 FAM 0.0342 83.62± 2.04 NGE 0.0928 79.75± 1.00 4 GNG 0.0647 64.81± 1.13 FAM 0.0380 78.84± 1.03 GNG 0.0835 79.11± 1.65 5 ILFD 0.0647 91.85± 1.09 FAM 0.0380 84.93± 1.65 ILFD 0.0835 51.14± 4.11 6 FAM 0.0524 94.07± 1.31 FAM 0.0422 77.25± 1.78 FAM 0.0928 53.80± 3.03 7 FAM 0.0719 78.52± 1.23 FAM 0.0422 75.65± 1.09 FAM 0.1146 79.11± 1.65 8 FAM 0.0719 91.85± 0.65 FAM 0.0469 76.67± 1.09 FAM 0.1146 79.11± 1.65 9 FAM 0.0799 81.48± 1.23 FAM 0.0469 85.51± 1.65 FAM 0.1146 64.05± 1.93 10 FAM 0.0888 88.89± 1.23 FAM 0.0469 84.64± 1.00 FAM 0.1273 79.11± 1.48 11 FAM 0.1096 84.81± 0.93 FAM 0.0469 85.36± 1.78 12 FAM 0.0987 82.22± 0.93 FAM 0.0469 82.17± 1.78 13 FAM 0.1096 87.04± 1.23 FAM 0.0469 82.61± 1.00 14 FAM 0.0579 80.58± 1.92 15 FAM 0.0644 65.36± 1.65 16 FAM 0.0794 63.33± 3.01 17 FAM 0.1090 72.61± 1.93 18 GFMMN 0.0422 65.36± 1.42 19 FAM 0.0342 60.58± 2.78 20 GNG 0.0342 57.54± 4.63 21 ILFD 0.0342 59.42± 3.11 Ensemble 97.30±1.09 Ensemble 83.91 ± 2.54 Ensemble 79.75 ± 3.11

set, the contribution of the combination is apparent, while in the other two cases, there are individuals that outperform the ensemble. The reason for that is obviously the weights which have been adapted on the training data. But in all the ensemble always outperform the majority of the individuals. Therefore the ensemble can be considered as the most reliable.

254

A. Bouchachia Table 8. Performance of the ensemble classifiers in presence of drift - Winnow 1

Alg.

FAM GFMMN NGE GNG ILFD Ensemble 75.93 ± 1.23 74.70 ± 1.84 67.41 ± 1.32 97.04 ± 1.18 79.26 ± 2.39 Cancer 85.93 ± 1.56 0.1728 0.2370 0.2133 0.1399 0.2370 77.39 ± 1.53 63.62 ± 2.32 69.71 ± 2.04 59.04± 0.93 56.23 ± 2.05 Spam 79.57 ± 1.84 0.2448 0.2203 0.2448 0.0454 0.2448 66.33 ± 1.65 60.96 ± 1.13 67.59 ± 1.13 60.63 ± 1.63 67.00 ± 1.84 Defect 66.16 ± 1.04 0.1757 0.1757 0.2423 0.2306 0.1757

6.3

Performance of Adaptive Ensemble Classifiers

While in the previous experiments the base classifiers do not change during training, using Alg. 2, we can enhance the adaptation of the ensemble by a self-tuning mechanism that allows the classifiers to leave the ensemble and other to be created. Therefore, adaptation under this scenario is present in its three forms: self-adaptation of individual classifiers, weighting and structural update. Once training is exhausted, the final configuration of the ensemble is obtained. Such configuration of the ensemble is shown in Tab. 7 columns 2, 5 and 8 along with weights shown in columns 3, 6 and 9. Note that these experiments, we set the maximum number of individuals L (see Alg. 2) illustratively to 21. Testing of the individual classifiers and the resulting ensemble produces the accuracy values shown in Tab. 7 for each of the data sets (columns 4, 7 and 10). Comparing the results obtained without structure adaptation (Tab. 6) against those of the adaptive ensemble, the performance of the ensemble (i.e., the last row) shows that adaptation has a clear contribution irrespectively of the data sets. In the case of the defect data the contribution is even clearer. 6.4

Performance of Ensemble Classifiers in Presence of Drift

This experiment aims at examining the capability of the ensemble classifiers to deal with concept drift. After generating the drifting data according to the mechanism described in Sec. 6, we turn to the analysis of the performance of the incremental classifiers and their combination. Here we study the ensemble without adaptation. Table 8 shows the results obtained. Clearly the drift has an effect on the classifiers. When comparing Tab. 6 against Tab. 8, the performance of the ensemble decreases, but that is to some extent expected due to drift that often leads the prototypes generated by the networks to overlap. However, one can see clearly that the performance of the ensemble is very acceptable in presence of drift. It is also important to note that some of the classifiers such as NGE and GNG resist quite well to drift. 6.5

Performance of Adaptive Ensemble Classifiers in Presence of Drift

Considering adaptive ensemble in presence of drift, the idea is to observe whether equipping the ensemble classifier with incremental structural adaptation has an effect on the performance. The results obtained on each of the data sets are portrayed


255

Table 9. Performance of adaptive ensemble classifiers in presence of drift - Winnow 2 Cancer Spam Defect Alg. type Weight Accur. Type Weight Accur. Type Weight Accuracy 1 FAM 0.1090 95.56± 0.95 NGE 0.0313 76.52 ± 1.06 FAM 0.1243 65.19 ± 1.45 2 GFMMN 0.1211 81.48± 1.53 FAM 0.0347 76.96 ± 1.09 GFMMN 0.1119 60.63 ± 1.56 3 NGE 0.1211 97.04± 0.73 FAM 0.0476 69.28 ± 1.35 NGE 0.1119 60.63 ± 1.73 4 GNG 0.1211 96.30± 0.79 FAM 0.0386 71.01 ± 1.13 GNG 0.1119 57.47 ± 2.01 5 ILFD 0.1090 71.85± 1.58 FAM 0.0476 68.70 ± 1.54 ILFD 0.1243 59.37 ± 1.76 6 FAM 0.1346 76.30± 1.06 FAM 0.0476 63.77 ± 1.67 FAM 0.1243 62.53 ± 1.25 7 FAM 0.1346 64.81± 1.98 FAM 0.0476 68.99 ± 1.60 FAM 0.1381 61.90 ± 1.49 8 FAM 0.1495 86.67± 0.97 FAM 0.0476 69.42 ± 1.48 FAM 0.1534 60.00 ± 1.63 9 FAM 0.0653 63.77 ± 1.63 10 FAM 0.0529 67.68 ± 1.29 11 FAM 0.0476 66.96 ± 1.52 12 FAM 0.0529 79.28 ± 0.97 13 FAM 0.0807 58.99 ± 1.85 14 FAM 0.0726 71.74 ± 1.15 15 FAM 0.0726 70.72 ± 1.21 16 GFMMN 0.0386 70.87 ± 1.23 17 FAM 0.0347 71.74 ± 1.09 18 ILFD 0.0386 65.36 ± 1.73 19 FAM 0.0347 61.74 ± 1.69 20 FAM 0.0313 61.74 ± 1.82 21 GNG 0.0347 68.55 ± 1.61 Ensemble 94.81 Ensemble 81.45 Ensemble 69.82

in Tab. 9. As anticipated drift may impact the accuracy of the ensemble if we compare the results of Tab. 7 against those of Tab. 9. However, the accuracy remains very competitive. On the other hand, the accuracy of the individual incremental classifiers slightly decreases. Therefore one can conclude that the ensemble approach is justifiably worth considering. Moreover, it is quite interesting to note that when comparing results of Tab. 9 against the results of Tab. 8 (corresponding to non adaptive ensemble in presence of drift), the adaptation expressed by means of structural update in presence of drift is arguably rational.

7 Conclusions In this paper, the problem of adaptation has been discussed from three perspectives. The first concerns the self-organizing nature of the classifiers studied; the second is about the proportional (weighted) contribution of the classifiers when incorporated into an incremental ensemble classifiers; the third pertains to the incremental and dynamic update over time of the ensemble’s structure (i.e., the ensemble can grow and shrink). Extensive experiments have been conducted to study each of these adaptation forms showing in particular the rationality of considering the ensemble classifiers as an approach to deal with various incrementality scenarios. Further investigations are planned in the future such as equipping the individual classifiers and the ensemble with forgetting mechanisms, undertaking a closer look

256

A. Bouchachia

at the conditions under which new classifiers are added or deleted by considering various criteria such as diversity and accuracy performance, and comparing such incremental algorithms with known techniques such as retraining that require partial memory.

References 1. Battiti, R., Colla, A.: Democracy in neural nets: Voting schemes for classification. Neural Networks 7(4), 691–707 (1994) 2. Blum, A.: Empirical support for winnow and weighted-majority based algorithms: results on a calendar scheduling domain. Machine Learning 26, 5–23 (1997) 3. Bouchachia, A.: Incremental learning via function decomposition. In: Proc. of the Int. Conf. on machine learning and applications, pp. 63–68 (2006) 4. Bouchachia, A.: Incremental Learning. In: Encyclopedia of Data Warehousing and Mining, 2nd edn., Idea-Group (in press) (2008) 5. Breiman, L.: Bagging predictors. Machine Learning 24(2), 123–140 (1996) 6. Breiman, L.: Pasting small votes for classification in large databases and on-line. Machine Learning 36, 85–103 (1999) 7. Carpenter, G., Grossberg, S., Rosen, D.: Fuzzy ART: Fast Stable Learning and Categorization of Analog Patterns by an Adaptive Resonance System. Neural Networks 4(6), 759–771 (1991) 8. Dietrich, C., Palm, G., Schwenker, F.: Decision templates for the classification of bioacoustic time series. Information Fusion 4(2), 101–109 (2003) 9. Fern, A., Givan, R.: Machine learning. Machine Learning 53, 71–109 (2003) 10. French, R.: Catastrophic forgetting in connectionist networks: Causes, consequences and solutions. Trends in Cognitive Sciences, Trends in Cognitive Sciences 3(4), 128–135 (1999) 11. Freund, Y., Schapire, R.: A decision-theoretic generalization of on-line learning and an application to boosting. In: Proceedings of the Second European Conference on Computational Learning Theory, pp. 23–37 (1995) 12. Fritzke, B.: A growing neural gas network learns topologies. In: Advances in neural information processing systems, pp. 625–632 (1995) 13. Gabrys, B., Bargiela, A.: General fuzzy min-max neural network for clustering and classification. IEEE Trans. on Neural Networks 11(3), 769–783 (2000) 14. Grossberg, S.: Nonlinear neural networks: principles, mechanism, and architectures. Neural Networks 1, 17–61 (1988) 15. Hettich, S., Blake, C., Merz, C.: UCI repository of machine learning databases (1998), www.ics.uci.edu/˜mlearn/MLRepository.html 16. Huang, Y., Suen, C.: A method of combining multiple experts for the recognition of unconstrained handwritten numerals. IEEE Transactions on Pattern Analysis and Machine Intelligence 17(1), 90–94 (1995) 17. Jacobs, R.: Methods of combining experts’ probability assessments. Neural Computing 7, 865–888 (1995) 18. Jacobs, R., Jordan, M., Nowlan, S., Hinton, G.: Adaptive mixtures of local experts. Neural Computing 3, 79–87 (1991) 19. Stanley, K.: Learning concept drift with a committee of decision trees. Technical Report TR-03-302, Dept of Computer Science, University of Texas at Austin, USA (2003) 20. Kittler, J., Hatef, M., Duin, R., Matas, J.: On combining classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence 20(3), 226–239 (1998)


257

21. Klinkenberg, R.: Learning drifting concepts: example selection vs. example weighting. Intelligent Data Analysis 8(3), 281–300 (2004) 22. Kolter, J., Maloof, M.: Dynamic weighted majority: a new ensemble method for tracking concept drift. In: Proceedings of the 3rd International Conference on Data Mining ICDM 2003, pp. 123–130. IEEE CS Press, Los Alamitos (2003) 23. Kuncheva, L.: A theoretical study on six classifier fusion strategies. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(2), 281–286 (2002) 24. Kuncheva, L.: Classifier ensembles for changing environments. In: Proc. of the 5th international workshop on multiple classifier systems, pp. 1–15 (2004) 25. Kuncheva, L., Bezdek, J., Duin, R.: Decision templates for multiple classifier fusion: An experimental comparison. Pattern Recognition 34(2), 299–314 (2001) 26. Lam, L., Suen, C.: Optimal combinations of pattern classifiers. Pattern Recognition Letters 16, 945–954 (1995) 27. Littlestone, N.: Learning quickly when irrelevant attributes abound: A new linear threshold algorithm. Machine Learning 2, 285–318 (1988) 28. Martinetz, T., Berkovich, S., Schulten, K.: Neural gas network for vector quantization and its application to time-series prediction. IEEE Trans. Neural Networks 4(4), 558– 569 (1993) 29. McCloskey, M., Cohen, N.: Catastrophic interference in connectionist networks: the sequential learning problem. The psychology of learning and motivation 24, 109–164 (1999) 30. Oza, N.: Online Ensemble Learning. Phd thesis, University of California, Berkeley (2001) 31. Promise. Software engineering repository (May 2008), http://promise.site.uottawa.ca/SERepository 32. Quartz, S., Sejnowski, T.: The neural basis of cognitive development: a constructivist manifesto. Behavioral and Brain Sciences 20(4), 537–556 (1997) 33. Ratcliff, R.: Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological Review 97, 285–308 (1990) 34. Sahel, Z., Bouchachia, A., Gabrys, B.: Adaptive mechanisms for classification problems with drifting data. In: Apolloni, B., Howlett, R.J., Jain, L. (eds.) KES 2007, Part II. LNCS, vol. 4693, pp. 419–426. Springer, Heidelberg (2007) 35. Salzberg, S.: A nearest hyperrectangle learning method. Machine learning 6, 277–309 (1991) 36. Sharkey, N., Sharkey, A.: Catastrophic forgetting in connectionist networks: Causes, consequences and solutions. An analysis of catastrophic interference 7(3-4), 301–329 (1995) 37. Sirosh, J., Miikkulainen, R., Choe, Y. (eds.): Lateral Interactions in the Cortex: Structure and Function, The UTCS Neural Networks Research Group, Austin, TX, Electronic book (1996) 38. Slaganicoff, M.: Density-adaptive learning and forgetting. In: Proc. of the 10th Int. Conf. on Machine Learning, pp. 276–283 (1993) 39. Street, W., Kim, Y.: A streaming ensemble algorithm (sea) for large-scale classification. In: Proceedings of the 7th International Conference on Knowledge Discovery and Data Mining KDDM 2001, pp. 377–382 (2001) 40. Tax, D., van Breukelen, M., Duin, R., Kittler, J.: Combining multiple classifiers by averaging or by multiplying? Pattern Recognition 33(9), 1475–1485 (2000) 41. Tsymbal, A., Pechenizkiy, M., Cunningham, P., Puuronen, S.: Handling local concept drift with dynamic integration of classifiers: Domain of antibiotic resistance in nosocomial infections. In: Proc. of the 19th IEEE Symposium on Computer-Based Medical Systems, pp. 679–684 (2006)

258

A. Bouchachia

42. Wettschereck, D., Dietterich, T.: An experimental comparison of the nearest-neighbor and nearest-hyperrectangle algorithms. Machine Learning 19(1), 5–27 (1995) 43. Widmer, G., Kubat, M.: Learning in the presence of concept drift and hidden contexts. Machine Learning 23, 69–101 (1996) 44. Wolpert, D.: Stacked generalization. Neural Networks 5(2), 241–259 (1992) 45. Woods, K., Kegelmeyer, W., Bowyer, K.: Combination of multiple classifiers using local accuracy estimates. IEEE Transactions on Pattern Analysis and Machine Intelligence 19, 405–410 (1997)

Music Instrument Estimation in Polyphonic Sound Based on Short-Term Spectrum Match Wenxin Jiang1 , Alicja Wieczorkowska2, and Zbigniew W. Ra´s1,2 1 2

University of North Carolina, Department of Computer Science, Charlotte, NC 28223, USA Polish-Japanese Institute of Information Technology, Koszykowa 86, 02-008 Warsaw, Poland [email protected], [email protected], [email protected]

Summary. Recognition and separation of sounds played by various instruments is very useful in labeling audio files with semantic information. This is a non-trivial task requiring sound analysis, but the results can aid automatic indexing and browsing music data when searching for melodies played by user specified instruments. In this chapter, we describe all stages of this process, including sound parameterization, instrument identification, and also separation of layered sounds. Parameterization in our case represents power amplitude spectrum, but we also perform comparative experiments with parameterization based mainly on spectrum related sound attributes, including MFCC, parameters describing the shape of the power spectrum of the sound waveform, and also time domain related parameters. Various classification algorithms have been applied, including k-nearest neighbor (KNN) yielding good results. The experiments on polyphonic (polytimbral) recordings and results discussed in this chapter allow us to draw conclusions regarding the directions of further experiments on this subject, which can be of interest for any user of music audio data sets.

1 Introduction Recently, a number of acoustical features for the construction of a computational model for music timbre estimation have been investigated in Music Information Retrieval (MIR) area. Timbre is a quality of sound that distinguishes one music instrument from another, since there are a wide variety of instrument families and individual categories. It is rather a subjective quality, defined by ANSI as the attribute of auditory sensation, in terms of which a listener can judge that two sounds, similarly presented and having the same loudness and pitch, are different [1], [2]. Such definition is clearly subjective and not of much use for automatic sound timbre classification, although the footnote to the definition gives hints towards physical timbre description, stating that the timbre depends primarily upon the spectrum of the stimulus, but also upon the waveform, the sound pressure, the frequency A.-E. Hassanien et al. (Eds.): Foundations of Comput. Intel. Vol. 2, SCI 202, pp. 259–273. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com

260

W. Jiang, A. Wieczorkowska, and Z.W. Ra´s

location of the spectrum, and the temporal characteristics of the stimulus [2], [5]. Still, musical sounds must be very carefully parameterized to allow automatic timbre recognition. So far, there is no standard parameterization used as a classification basis. The sound descriptors applied are based on various methods of analysis in time domain, spectrum domain, time-frequency domain and cepstrum, with Discrete Fourier Transform (DFT) for spectral analysis being most common, e.g. Fast Fourier Transform (FFT), and so on. Also, wavelet analysis gains increasing interest for sound and especially for musical sound analysis and representation. Researchers explored different statistical summations to describe signatures of music instruments based on vectors or matrices in features, such as Tristimulus parameters, brightness, irregularity of the spectrum etc. [6], [14], [21]. Flattening these features for traditional classifiers increases the number of features. In [16] authors used a new set of features jointly with other popular features used in music instrument identification. They built a database of music instrument sounds for training a number of classifiers. These classifiers are used by MIRAI system to identify music instruments in polyphonic sounds. MIRAI is designed as a web-based storage and retrieval system which can automatically index musical input (of polyphonic, polytimbral type), transforming it into a database, and answer queries requesting specific musical pieces, see http://www.mir.uncc.edu/. When MIRAI receives a musical waveform, it divides this waveform into segments of equal size and then the classifiers incorporated into the system identify the most dominating musical instruments and emotions associated with that segment. A database of musical instrument sounds describing about 4,000 sound objects by more than 1,100 features is associated with MIRAI. Each sound object is represented as a temporal sequence of approximately 150-300 tuples which gives a temporal database of more than 1,000,000 tuples, each one represented as a vector of about 1,100 features. This database is mainly used to train classifiers for automatic indexing of musical instrument sounds. It is semantically reach enough (in terms of successful sound separation and recognition) so the constructed classifiers have a high level of accuracy in recognizing the dominating musical instrument and/or its type when music is polyphonic. Unfortunately, the loss of information on non-dominant instruments by the sound separation algorithm, due to the overlap of sound features, may significantly lower the recognition confidence of the remaining instruments in a polyphonic sound. This chapter shows that by identifying a weighted set of dominating instruments in a sequence of overlapping frames and using a special voting strategy, we can improve the overall confidence of the indexing strategy for polyphonic music, and the same improve the precision and recall of MIRAI retrieval engine.

Music Instrument Estimation in Polyphonic Sound

261

2 Sound Parameterization for Automatic Classification Purposes Sound wave can be described as a function representing amplitude changes in time. For digitally recorded sound, this function is quantized in time and in amplitude. Sampling rate describes how many values are recorded per time unit, and binary resolution in multi-bit recording describes how many bits are used to represent quantized amplitude axis, for each channel. Standard setting for CD for sampling rate is 44,100 samples per second, i.e. 44.1 kHz, with 16bit resolution for amplitude, i.e. with 216 quantization levels. Obviously, such data are not well suited to perform automatic classification, so usually sound parameterization is performed before further experiments on audio databases. There are numerous ways the sound can be parameterized. Audio data can be parameterized in time domain, in frequency domain, and time-frequency features can also be extracted. Parameterization can be based on Fourier analysis, particularly on amplitude spectrum, on cepstral analysis, also on wavelet analysis, and so on; various features can be extracted to describe the results of these analyzes [3], [4], [7], [8], [9], [11], [13], [19], [22]. In our research, we decided to base mainly on direct observation of sound spectrum (see Section 4), but we also performed experiments based on the following sound features [8], [22]: •

• •

•

• •

AudioSpectrumBasis - MPEG-7 descriptor, representing low-dimensional projection of a high-dimensional spectral space, to aid compactness and recognition [8]. AudioSpectrumBasis is a matrix derived from the SVD (singular value decomposition) of a power spectrum in normalized dB scale, i.e. in log scale with maximal value defining 0 dB. In our research, frequency axis for AudioSpectrumBasis was divided into 32 bands, with 1/4-octave resolution for 8 octaves; octave distance means doubling the fundamental frequency, i.e. the pitch of the sound. AudioSpectrumP rojection - projection of AudioSpectrumBasis [8], AudioSpectrumF latness - MPEG-7 parameter, calculated in our research for spectrum divided into 32 frequency bands, i.e. with 1/4-octave resolution for 8 octaves, and the length of this 32-element vector is added as 0th element of this 33-dimensional feature; if there is a high deviation from a flat spectral shape for a given band, it may signal the presence of tonal components [8], M F CC = {mf ccn : 1 ≤ n ≤ 13} - cepstral coefficients in mel scale; feature originating from speech processing, but also used for music analysis [12], [17]. 13 coefficients were used (the 0th one and the next 12), for 24 mel frequency scale hearing filters, using Julius software [10], HamonicP eaks = {HamoP kn : 1 ≤ n ≤ 28} - sequence of the first 28 local peaks of harmonics (in normalized dB scale) for a given frame T emporalCentroid - time instant where the energy of the sound is focused, calculated as energy weighted mean of the sound duration,

262

• •

• • •

•

• • •


LogSpecCentroid - AudioSpectrumCentroid from MPEG-7 standard [8]; this parameter represents the gravity center of a log-frequency power spectrum, LogSpecSpread - AudioSpectrumSpread descriptor from MPEG-7 [8]; calculated as RMS (Root Mean Square) value of the deviation of the power spectrum in log frequency scale with respect to the gravity center in a frame, Energy - energy of spectrum, averaged through all frames of the sound, ZeroCrossings - zero-crossing rate, i.e. number of sign changes of the wave form in a frame, averaged through all frames of the sound, SpecCentroid - calculated as HarmonicSpectralCentroid from MPEG7, representing power-weighted average of the frequency of the bins in the linear power spectrum, and averaged over all the frames of the steady state of the sound, SpecSpread - calculated as HarmonicSpectralSpread from MPEG-7, describing the amplitude-weighted standard deviation of the harmonic peaks of the spectrum, normalized by the instantaneous HarmonicSpectralCen− troid and averaged over all the frames of the steady state of the sound, RollOf f - averaged (over all frames) frequency below which an experimentally chosen percentage of the accumulated magnitudes of the spectrum is concentrated, F lux - difference between the magnitude of the amplitude spectrum points in a given and successive frame, averaged through the entire sound, LogAttackT ime - decimal logarithm of the sound duration from the time instant when the signal starts, to the time when it reaches its maximum value, or when it reaches its sustained part, whichever comes first.

3 Polyphonic Sound Estimation Based on Sound Separation and Feature Extraction The traditional way of pattern recognition in music sound is to extract features from raw signals in digital form, usually recorded as a sequence of integer samples representing quantized values of amplitude of a sound wave in consequent time instants. By feature extraction, the acoustic characteristics such as pitch and timbre are described by smaller and more structured dataset which is further fed to traditional classifiers to perform estimation. In case of polyphonic sounds, sound separation can be applied to extract the signal which is identified as one specific instrument at timbre estimation process. Then timbre estimation can be applied again on the residue of the signal to get other timbre information. Fig. 1 shows the process of music instrument recognition system based on feature extraction and sound separation. However, there are two main problems in this method. First of all, overlapping of the features make it difficult to perform timbre estimation and sound separation. Secondly, during the classification process, only one instrument is picked up from all candidates, which makes the estimation inaccurate.


263

Polyphonic Sound Get pitch Feature extraction

FFT

Pitch Estimation

Classifier

Get spectrum

Timbre Estimation Sound separation

Get Instrument

New spectrum

Power Spectrum

Fig. 1. Flow chart of music instrument recognition with sound separation

3.1

Overlapping of the Features Both in Temporal and Spectral Space

Feature based datasets are easier and more efficient to work with classifiers, however, there is usually information loss during the feature extraction process. Feature is the abstract or compressed representation of waveform or spectrum, such as harmonic peaks, MFCC (Mel Frequency Cepstral Coefficients), zero-crossing rate, and so on. In the case of monophonic music sound estimation tasks with only singular non-layered sounds, the features can be easily extracted and identified. However, this is not the case in polyphonic, polytimbral sound. It is difficult or even often impossible to extract distinct clear features representing single instrument from polyphonic sound, because the overlapping of the signals and their spectra, especially when instruments have the similar patterns in their features space. 3.2

Classification with Single Instrument Estimation for Each Frame

The traditional classification process usually gives single answer, representing one class; in our case, it would be the name of instrument playing in an analyzed sample. In such a case, when only the best answer, i.e. the name of one (the only one or dominating) instrument playing for each frame of music sound is given, then information about other possibly contributing instruments is lost. In fact, it is common for the polyphonic music sound to have multiple instruments playing simultaneously, which means that in each frame, there

264


are representations of multiple timbres existing in the signal. Providing one only candidate leads to obtaining predominant timbre while ignoring other timbre information. And also, there could be no dominating timbre in each frame, when all instruments play equally loud. This means that classifier has to randomly choose one of the equally possible candidates. In order to find solution to this problem, authors introduce the Top-N winner strategy which gives multiple candidates for each evaluated frame.

4 Pattern Detection Directly from Power Spectrum The fact that discriminating one instrument from another depends on more details from raw signals leads to another way of pattern recognition: directly detecting distinct patterns of instruments based on lower representation of signal, such as power spectrum. Fig. 2 shows two different ways of pattern recognition. Since spectrum is very useful for timbre representation purposes, we propose the new strategy of instrument estimation based on short term power spectrum match. 4.1

Sub-Pattern of Single Instrument in the Mixture Sound Segment

Figure 3 shows the power spectrum of trumpet, piano and the mixture of those two instruments. As we can see, the spectrum of mixture preserves part of the pattern of each single instrument.

Fig. 2. Two different methods of pattern recognition


265

Fig. 3. Power spectrum of trumpet, piano and their mixture; frequency axis is in linear scale, whereas amplitude axis is in log [dB] scale

The same similarity of properties of the spectra is also observed e.g. for flute, trombone and their mixture, as Figure 4 shows. In order to index the polyphonic sound, we need to detect the instrument information in each small slice of music sound. Such detection is rather not feasible directly in time domain. Therefore, in our experiments, we have observed the short term spectrum space. This has been calculated via short time Fourier transform (STFT). Figure 5 shows another example of the spectrum slice for flute and trombone and their mixture sound. Each slice is 0.04 seconds long. As Figure 5 shows, the power spectrum patterns of single flute and single trombone can still be identified in mixture spectrum without blurring with each other (as marked in the figure). Therefore, we do get the clear picture of distinct pattern of each single instrument when we observe each spectrum slice of the polyphonic sound wave. 4.2

Classification Based on Power Spectrum Pattern Match

In order to represent accurately the short term spectrum with high resolution in frequency axis, allowing more precise pattern matching, long analyzing

266


Fig. 4. Power spectrum of flute, trombone and their mixture

frame with 8192 numeric samples was chosen. Fourier transform performed on these frames describes frequency space for each slice (or frame). Instead of parameterizing the spectrum (or time domain) and extracting a few dozens of features to represent sound, we decided to work directly on the power amplitude spectrum values (points). When a new sound is analyzed with a goal to find what instrument or instruments contributed to create this sound, even though their spectra overlap, we can still try to find the closest vectors from the training data set of singular sounds and discover which instrument sounds they represent. The traditional classification models such as decision trees, Naive Bayesian classifiers, and neural networks do not perform well in this case. It is because there are too many attributes (8192 numeric attributes) for those classifiers to yield good classification models, and also any classification model itself stands for some sort of abstraction, which is in conflict with any information preserving strategy. However, one of the most fundamental and simple classification methods, K Nearest Neighbor algorithm, needs no prior knowledge about the distribution of the data and it seems to be an appropriate classifier for numeric spectrum vectors.


267

Fig. 5. Sub-patterns of single instruments in the mixture sound slice for flute, trombone, and their mix

5 Top-N Winners from Classification Model As opposed to the traditional pattern matching or classification process, which uses classification model to choose the one with the highest confidence as the estimation result, we do not get the ”best answer” for an evaluated frame. Instead, we choose the multiple candidates from classification model according to the confidence measure. As it was already discussed, we know that the classifier during the classification process compares pattern from

268


each frame with standard instrument patterns in a training database. Since there are several different sub-patterns existing in the spectrum, classifier will assign the confidence to each recognized pattern. This way, we may identify which N matches have the highest confidence levels. They are our top N winners. Thus, at each frame we get n instruments Ii with the confidence level Ci and save them to the candidates pool for the voting process. After evaluating all the frames, we get weights for all the candidates from the candidates pool by adding up their confidences, and the final voting proceeds according to the weights Wj of each instrument. The following is the pseudo-code for the Top-N winners procedure: For each frame from the sound Get power spectrum by STFT For each candidate Xi from top-N winners of classifiers If Xi exists in candidates pool then Confidence[x] += Ci Else Add Xi into candidates pool Confidence[x] = Ci End If End For End For Select Top m candidates from candidates pool Some noise coming from errors occurred during the single frame estimation process could be minimized in terms of the whole music context. By keeping the original acoustical information of the music sound, we are getting much higher recognition rate for multiple instruments in polyphonic sound. Here are the steps of pattern matching process: 1. Use STFT and Hamming window to extract power spectrum for each 0.04s frame for all the standard single instrument sounds. 2. Save these spectra in a training database; since there is overlapping of 2/3 of frame length for Hamming window, the number of items in the dataset actually almost triples for each sound. 3. During the estimation process, use KNN to do the vector distance measure (8192 points) and decide which frame in the training dataset is the most similar to the unknown sound frame; when we give multiple matches, the multiple instrument candidates are saved for the overall weights calculation. Fig. 6 shows the new music instrument recognition system which has been developed with the strategy of Top-N winners based on short-term spectrum matching.


269

Polyphonic Sound

Training data Get frame

Timbre Estimation FFT

Power Spectrum

Classifier Get Instrument Candidates

Finish all the Frames Get Final winners

Voting process

estimation

Fig. 6. Flow chart of music instrument recognition system with new strategy

6 Experiment of Top-N-Winners Strategy Based on Short Term Spectrum Matching To simplify the problem, we only performed tests on the middle C instrument sounds, i.e. for pitch equal to C4 in MIDI notation, of frequency 261.6 Hz (for A4 tuned to 440 Hz). The training subset including 3323 objects has been selected from the entire training database. Each object is represented by the frame-wise (0.04 seconds) power spectrum extracted by short time Fourier transform from the following 26 single instrument sounds: Electric Guitar, Bassoon, Oboe, B-flat Clarinet, Marimba, C Trumpet, Eflat Clarinet, Tenor Trombone, French Horn, Flute, Viola, Violin, English Horn, Vibraphone, Accordion, Electric Bass, Cello, Tenor Saxophone, B-Flat Trumpet, Bass Flute, Double Bass, Alto Flute, Piano, Bach Trumpet, Tuba, and Bass Clarinet. To compare the results with the traditional feature based classification strategy, we have also extracted the following 5 groups of both temporal and spectral features (calculated for spectrum divided into 33 frequency bands), mainly originating from the MPEG-7 standard [8], [20], [22], and used decision tree classifier to perform the timbre estimation: Group1: BandsCoef f icient = {bandsCoefn : 1 ≤ n ≤ 33} - coefficients for 33 AudioSpectrumF latness bands. Group2: P rojections = {prjn : 1 ≤ n ≤ 33} - AudioSpectrumP rojection from MPEG-7,

270


Group3: M F CC = {mf ccn : 1 ≤ n ≤ 13} Group4: HamonicP eaks = {HamoP kn : 1 ≤ n ≤ 28} Group5: Other Features: • • • • • • • • • • •

•

T emporalCentroid, LogSpecCentroid, LogSpecSpread, Energy, ZeroCrossings, SpecCentroid, SpecSpread, RollOf f , F lux, bandsCoef Sum - AudioSpectrumF latness bands coefficients sum, prjmin, prjmax, prjsum, prjdis, prjstd - minimum, maximum, sum, distance, and standard deviation of AudioSpectrumP rojection calculated for AudioSpectrumBasis. Distance represents a dissimilarity measure: distance for a matrix is calculated as sum of absolute values of differences between elements of each row and column. Distance for a vector is calculated as the sum of dissimilarity (absolute difference of values) of every pair of coordinates in the vector, LogAttackT ime.

52 polyphonic audio files have been mixed (using Sound Forge sound editor [18]) from 2 of those 26 instruments sound. These mixture audio files have been used as test files. The system uses MS SQLSERVER2005 database to store training dataset and K nearest neighbor algorithm as the classifier. When the polyphonic sound is submitted to system, it provides several estimations as the final candidate answers. In our experiment, we gave 4 estimations for each submitted audio file. The performance of our algorithm was measured using recognition rate R, calculated as R = P/A where P is the positive response, i.e. the number of the correct estimations, and A is the actual number of instruments existing in the polyphonic sound. For comparison purpose, five experiments were performed independently. We applied feature-based sound separation strategy and we used a decision tree type classifier in our first two experiments. In experiment 1, only one candidate was chosen by a classifier for each frame. In the first step of experiment 2, top n candidates (with n = 2) were chosen by a classifier for each frame. In its second step, for each candidate, the confidences over all the frames were added to get the overall score used to identify the final n winners. In the remaining three experiments, we applied a new strategy of spectrum match based on KNN classifier. In experiment 3, we used KNN (k = 1) to


271

Table 1. Recognition rate of music instrument estimation based on various strategies experiment #

description

Recognition Rate

1

Feature-based and separation + Decision Tree (n=1)

36.49%

2

Feature-based and separation + Decision Tree (n=2)

48.65%

3

Spectrum Match + KNN(k=1;n=2)

79.41%

4

Spectrum Match + KNN(k=5;n=2)_ Spectrum Match + KNN(k=5;n=2) without percussion instrument_

82.43%

5

87.1%

choose the top 2 candidates as the winners for each frame. In experiment 4, we increased k from 1 to 5. In experiment 5, we ruled out the percussion instrument objects from the testing audio files, since they have less clear patterns in the spectrum envelope. From the results shown in Table 1, we get the following conclusions: 1. Using the multiple candidates for each frame yields better results than using single winner strategy. 2. Spectrum-based KNN classification improves the recognition rate of polyphonic sounds significantly. 3. Some percussion instrument (such as vibraphone, marimba) are not suitable for spectrum-based classification, but most instruments generating harmonic sounds work well with this new strategy.

7 Conclusion We have provided a new solution to an important problem of instrument identification in polyphonic music: The loss of information on non-dominant instruments during the sound separation process due to the overlapping of sound features. The new strategy is to directly detect sub-patterns from short term power spectrum, which is relatively lower-level and at the same time more efficient representation of the raw signals, instead of usually a few dozens (or maximally hundreds) of features, most often used for instrument recognition purposes. Next, we choose the multiple candidates from each frame during the frame-wise classification based on similarity of the spectrum, and weight them based on their possibility over all the sound period to get the more accurate estimation of multiple instruments which are playing simultaneously in the music piece. This approach also avoids extracting more compact feature patterns of multiple instruments from polyphonic sounds, which is difficult and inaccurate because of the information-loss during the abstraction process. Our experiments show that the sub-patterns detected from the power spectrum slices contain sufficient information for the multiple-timbre estimation tasks and improve the robustness of instrument identification as well.

272


Acknowledgments This work was supported by the National Science Foundation under grant IIS-0414815, and also by the Research Center of PJIIT, supported by the Polish National Committee for Scientific Research (KBN). We are grateful to Dr. Xin Zhang for many helpful discussions we had with her and for the comments she made which improved the quality and readability of the chapter.

References 1. Agostini, G., Longari, M., Pollastri, E.: Content-Based Classification of Musical Instrument Timbres. In: International Workshop on Content-Based Multimedia Indexing (2001) 2. American National Standards Institute, American national standard: Psychoacoustical terminology. ANSI S3.20-1973 (1973) 3. Aniola, P., Lukasik, E.: JAVA Library for Automatic Musical Instruments Recognition. AES 122 Convention, Vienna, Austria (2007) 4. Brown, J.C.: Computer identification of musical instruments using pattern recognition with cepstral coefficients as features. J. Acoust. Soc. Am. 105, 1933– 1941 (1999) 5. Fitzgerald, R., Lindsay, A.: Tying semantic labels to computational descriptors of similar timbres. In: Sound and Music Computing 2004 (2004) 6. Fujinaga, I., McMillan, K.: Real Time Recognition of Orchestral Instruments. In: International Computer Music Conference (2000) 7. Herrera, P., Amatriain, X., Batlle, E., Serra, X.: Towards instrument segmentation for music content description: a critical review of instrument classification techniques. In: International Symposium on Music Information Retrieval ISMIR (2000) 8. ISO/IEC JTC1/SC29/WG11, MPEG-7 Overview (2004), http://www.chiariglione.org/mpeg/standards/mpeg-7/mpeg-7.htm 9. Kaminskyj, I.: Multi-feature Musical Instrument Sound Classifier w/user determined generalisation performance. In: Proceedings of the Australasian Computer Music Association Conference ACMC, pp. 53–62 (2002) 10. Kawahara, T., Lee, A., Kobayashi, T., Takeda, K., Minematsu, N., Sagayama, S., Itou, K., Ito, A., Yamamoto, M., Yamada, A., Utsuro, T., Shikano, K.: Free software toolkit for Japanese large vocabulary continuous speech recognition. In: Proc. Int’l Conf. on Spoken Language Processing (ICSLP), vol. 4, pp. 476– 479 (2000) 11. Kitahara, T., Goto, M., Okuno, H.G.: Pitch-Dependent Identification of Musical Instrument Sounds. Applied Intelligence 23, 267–275 (2005) 12. Logan, B.: Mel Frequency Cepstral Coefficients for Music Modeling. In: Proceedings of the First International Symposium on Music Information Retrieval ISMIR 2000 (2000) 13. Martin, K.D., Kim, Y.E.: Musical instrument identification: A patternrecognition approach. In: 136-th meeting of the Acoustical Society of America, Norfolk, VA (1998)


273

14. Pollard, H.F., Jansson, E.V.: A Tristimulus Method for the Specification of Musical Timbre. Acustica 51, 162–171 (1982) 15. Ras, Z., Wieczorkowska, A., Lewis, R., Marasek, K., Zhang, C., Cohen, A., Kolczynska, E., Jiang, M.: Automatic Indexing of Audio With Timbre Information for Musical Instruments of Definite Pitch (2008), http://www.mir.uncc.edu/ 16. Ras, Z., Zhang, X., Lewis, R.: MIRAI: Multi-hierarchical, FS-tree based Music Information Retrieval System (Invited Paper). In: Kryszkiewicz, M., Peters, J.F., Rybinski, H., Skowron, A. (eds.) RSEISP 2007. LNCS (LNAI), vol. 4585, pp. 80–89. Springer, Heidelberg (2007) 17. Saha, G., Yadhunandan, U.: Modified Mel-Frequency Cepstral Coefficient. In: Proceedings of the IASTED 2004 (2004) 18. Sonic Foundry, Sound Forge. Software (2003) 19. Wieczorkowska, A.: Towards Musical Data Classification via Wavelet Analysis. In: Ohsuga, S., Ra´s, Z.W. (eds.) ISMIS 2000. LNCS (LNAI), vol. 1932, pp. 292–300. Springer, Heidelberg (2000) 20. Wieczorkowska, A., Ras, Z., Zhang, X., Lewis, R.: Multi-way Hierarchic Classification of Musical Instrument Sounds. In: Kim, S., Park, J., Pissinou, N., Kim, T., Fang, W., Slezak, D., Arabnia, H., Howard, D. (eds.) International Conference on Multimedia and Ubiquitous Engineering MUE 2007, Seoul, Korea. IEEE Computer Society, Los Alamitos (2007) 21. Wold, E., Blum, T., Keislar, D., Wheaten, J.: Content-based classification, search, and retrieval of audio. IEEE Multimedia 3(3), 27–36 (1996) 22. Zhang, X.: Cooperative Music Retrieval Based on Automatic Indexing of Music by Instruments and Their Types. PhD dissertation, The University of North Carolina at Charlotte, Charlotte (2007)

Ultrasound Biomicroscopy Glaucoma Images Analysis Based on Rough Set and Pulse Coupled Neural Network El-Sayed A. El-Dahshan1 , Aboul Ella Hassanien2 , Amr Radi1 , and Soumya Banerjee3 1

2

3

Physics Department, Faculty of Science, Ain Shams University, Abbassia, Cairo 11566, Egypt e @Yahoo.com, [email protected] Information Technology Department, FCI, Cairo University 5 Ahamed Zewal Street, Orman, Giza, Egypt [email protected], [email protected] Department of Computer Science, Birla Institute of Technology International Center, Mauritius [email protected]

Summary. The objective of this book chapter is to present the rough sets and pulse coupled neural network scheme for Ultrasound Biomicroscopy glaucoma images analysis. To increase the efficiency of the introduced scheme, an intensity adjustment process is applied first using the Pulse Coupled Neural Network (PCNN) with a median filter. This is followed by applying the PCNN-based segmentation algorithm to detect the boundary of the interior chamber of the eye image. Then, glaucoma clinical parameters have been calculated and normalized, followed by application of a rough set analysis to discover the dependency between the parameters and to generate set of reduct that contains minimal number of attributes. Finally, a rough confusion matrix is designed for discrimination to test whether they are normal or glaucomatous eyes. Experimental results show that the introduced scheme is very successful and has high detection accuracy.

1 Introduction Glaucoma is a disease that can cause a severe impairment of visual function and leads to irreversible blindness if untreated. About 60 million people worldwide will have glaucoma by 2010, and the number will increase to nearly 80 million by 2020, according to a recent study in the British Journal of Ophthalmology [1]. It has been estimated that one-half of the glaucoma patients are affected by angle closure glaucoma [2]. Angle closure glaucoma (ACG) has been called the most common form of glaucoma in the worldwide, and the leading cause of bilateral blindness [2, 3, 4]. If the disease is detected in its early stages, damage can be minimized and the long term prognosis for the patient is improved. A.-E. Hassanien et al. (Eds.): Foundations of Comput. Intel. Vol. 2, SCI 202, pp. 275–293. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com

276

E.-S.A. El-Dahshan et al.

Fig. 1. Schematic diagram for normal and glaucomatous eye

Fig. 2. Healthy and glaucomatous eye

Often the diagnosis of glaucoma depends on the type of glaucoma. There are two main types of glaucoma, ıopen angle and angle closure. The angle refers to the drainage area where the clear protective lining of the front chamber of the eye, the cornea, and the iris, the colored portion of the eye, join. If this area is closed or narrow, one could develop angle closure glaucoma. If this area is physically open and the individual has glaucoma, it is termed open angle [5]. Gonioscopy, Ultrasound Biomicroscopy (UBM), and Optical Coherence Tomography (OCT) are potentially important tools for diagnosing angle closure glaucoma (ACG) [6, 7]. Figure 1 shows the Schematic diagram for normal and glaucomatous eye. Figure 2 shows two cases, healthy and glaucomatous eye, the left image is a healthy eye with a wide anterior chamber angle and the right image is a glaucomatous eyes. UBM images can help the clinician visualize structures behind the iris. It is also of benefit when the anterior chamber structures cannot be clearly seen, such as through a cloudy cornea. The diagnostic utility of ultrasound biomicroscopy has been reported for anterior segment disorders such as glaucoma,

Ultrasound Biomicroscopy Glaucoma Images Analysis

277

iris tumor, corneal diseases, and ocular trauma. UBM can also detect small foreign bodies, including those missed by computed tomography and B-scan ultrasound [7]. The UBM has enabled clinicians to quantitatively assess the iris curvature and degree of angle opening, since it images a cross-section of angle structures similar to that of a low power microscope section. One can determine the state of closure of the entire angle, even when it cannot be visualized by gonioscopy. UBM acts at a frequency of 50 to 100 Hz with 20 to 60 μm resolution and 4 mm penetration [8, 9]. It produces high resolution images of the anterior part of the eye by which a qualitative and a quantitative evaluation of structures and their relation can be done [7]. In spite of recent advances in ultrasonic imaging, manually glaucoma clinical parameters assessment on UBM images by physicians is still a challenging task due to poor contrast, missing boundary, low signal-to-noise ration (SNR), speckle noise and refraction artifacts of the images. Besides, manual identification for glaucoma clinical parameters is tedious and sensitive to observer bias and experience. Thus, Semi- or automatic angle closure glaucoma clinical parameters measurements methods provide robust results with a certain degree of accuracy and can remove the physical weaknesses of observer interpretation within ultrasound images[10, 11]. This is essential for the early detection and treatment of glaucoma disease. Over the past two decades several traditional multivariate statistical classification approaches, such as the linear discriminant analysis and the quadratic discriminant analysis, have been developed to address the classification problem. More advanced and intelligent techniques have been used in medical data analysis such as neural network, Bayesian classifier, genetic algorithms, decision trees[12], fuzzy theory and rough set. Fuzzy sets [13] provide a natural framework for the process in dealing with uncertainty. It offers a problem-solving tool between the precision of classical mathematics and the inherent imprecision of the real world. Neural networks [14, 15] provide a robust approach to approximating real-valued, discrete-valued and vector-valued functions. The well-known algorithm Backpropagation, which uses gradient descent to tune network parameters to best fit the training set with input-output pair, has been applied as a learning technique for the neural networks. Other approaches like case based reasoning and decision trees [12]are also widely used to solve data analysis problems. Each one of these techniques has its own properties and features including their ability of finding important rules and information that could be useful for the medical field domain. Each of these techniques contributes a distinct methodology for addressing problems in its domain. Rough set theory [16, 17, 18] is a fairly new intelligent technique that has been applied to the medical domain and is used for the discovery of data dependencies, evaluates the importance of attributes, discovers the patterns of data, reduces all redundant objects and attributes, and seeks the minimum subset of attributes. Moreover, it is being used for the extraction of rules from databases. One advantage of the

278


rough set is the creation of readable if-then rules. Such rules have a potential to reveal new patterns in the data material. This chapter introduces a rough set scheme for Ultrasound Biomicroscopy glaucoma images analysis in conjunction with pulse coupled neural network. This chapter is organized as follows: Section 2 gives a brief mathematics background to pulse coupled neural network and to the rough sets. Section 3 discusses the proposed rough set data analysis scheme in detail. Section 4 discuss the Rough Sets Prediction Model. Experimental analysis and discussion of the results are described in section 5. Finally, conclusions and future work are presented in section 6.

2 Mathematics Background 2.1

Pulse Coupled Neural Network

The Pulse Coupled Neural Network(PCNN) [19, 20] are neural networks that are based on cats visual cortex and developed for high-performance biomimetic image processing. Eckhorn et al. [21, 22, 23] introduced a neural model to emulate the mechanism of cats’ visual cortex. The Eckhorn model provided a simple and effective tool for studying small mammals’ visual cortex and was soon recognized as having significant application potential in image processing. In 1994, Eckhorn model was adapted to be an image processing algorithm by Johnson who termed this algorithm PCNN. A PCNN is a two-dimensional neural network. Each neuron in the network corresponds to one pixel in an input image, receiving its corresponding pixels color information (e.g. intensity) as an external stimulus. Each neuron also connects with its neighboring neurons, receiving local stimuli from them. The external and local stimuli are combined in an internal activation system, which accumulates the stimuli until it exceeds a dynamic threshold, resulting in a pulse output. Through iterative computation, PCNN neurons produce temporal series of pulse outputs. The temporal series of pulse outputs contain information of input images and can be utilized for various image processing applications, such as image enhancement and segmentation [19, 20]. Pulse coupled neural network model is comprised of four parts that form the basis of the neuron. The first part is the feeding receptive field that receives the feeding inputs (i.e., image pixel values); the second part is the linking receptive field that receives the linking inputs from the neighbor neurons; the third part is modulation field, which the linking input added a constant positive bias, then it is multiplied by the feeding input; the last part is a pulse generator that consists of an output pulse generator and a threshold spike generator. When PCNN is applied to image processing, one neuron corresponds to one pixel. Figure 3 depicts the layout structure of PCNN and its components.


279

Fig. 3. The layout structure of PCNN and its components

2.2

Rough Set Theory

Rough set theory [16, 17, 18] is a fairly new intelligent technique for managing uncertainty that has been applied to the medical domain and is used for the discovery of data dependencies, evaluates the importance of attributes, discovers the patterns of data, reduces all redundant objects and attributes, seeks the minimum subset of attributes, recognizes and classifies objects in image processing. Moreover, it is being used for the extraction of rules from databases. Rough sets have proven useful for representation of vague regions in spatial data. One advantage of the rough set is the creation of readable if-then rules. Such rules have a potential to reveal new patterns in the data; furthermore, it also collectively functions as a classifier for unseen data sets. Unlike other computational intelligence techniques, rough set analysis requires no external parameters and uses only the information presented in the given data. One of the nice features of rough set theory is that it can tell whether the data is complete or not based on the data itself. If the data is incomplete, it suggests more information about the objects required to be collected in order to build a good classification model. On the other hand, if the data is complete, rough set approach can determine whether there are more than enough or redundant information in the data and find the minimum data needed for classification model. This property of rough set approach is very important for applications where domain knowledge is very limited or data collection is very expensive/laborious because it makes sure the data

280


collected is just good enough to build a good classification model without sacrificing the accuracy of the classification model or wasting time and effort to gather extra information about the objects [16, 17, 18]. In rough set theory, the data is collected in a table, called decision table. Rows of the decision table correspond to objects, and columns correspond to attributes. In the data set, we assume that the a set of examples with a class label to indicate the class to which each example belongs are given. We call the class label the decision attributes, and the rest of the attributes the condition attributes. Rough set theory defines three regions based on the equivalent classes induced by the attribute values lower approximation, upper approximation and boundary. Lower approximation contains all the objects, which are classified surely based on the data collected, and upper approximation contains all the objects which can be classified probably, while the boundary is the difference between the upper approximation and the lower approximation. So, we can define a rough set as any set defined through its lower and upper approximations. On the other hand, indiscernibility notion is fundamental to rough set theory. Informally, two objects in a decision table are indiscernible if one cannot distinguish between them on the basis of a given set of attributes. Hence, indiscernibility is a function of the set of attributes under consideration. For each set of attributes we can thus define a binary indiscernibility relation, which is a collection of pairs of objects that are indiscernible to each other. An indiscernibility relation partitions the set of cases or objects into a number of equivalence classes. An equivalence class of a particular object is simply the collection of objects that are indiscernible to the object in question. Here we provide an explanation of the basic framework of rough set theory, along with some of the key definitions. Reader’s may consult [16, 17, 18] for more fundamental details on rough set theory and applications.

3 Ultrasound Biomicroscopy Glaucoma Rough Sets Images Analysis Scheme Figure 4 illustrates the overall steps in the proposed Ultrasound Biomicroscopy Glaucoma Rough Sets Images Analysis Scheme using a UML Activity Diagram where a square or rectangular represents a data object, a rounded rectangular represents an activity, solid and dashed directed lines indicate control flow and data object flow respectively. Functionally, RBIS can be partitioned into three distinct phases. 3.1

Preprocessing Phase

In the first phase of the experiment the UBM eye images have been preprocessed to remove noise. Eye structure in UBM images are not very clear and this makes them very challenging to analysis, both for naked human eye and any automatic assessment algorithm. PCNN is a very powerful tool to enhance the boundaries in ultrasound images.


281

Fig. 4. Ultrasound Biomicroscopy Glaucoma Rough Sets Images Analysis Scheme

PCNN with the median filter noise reduction algorithm To increase efficiency of automating the boundary detection process, a preprocessing process should be considered to enhance the quality of the eye images before detection their boundaries. The median filter [19, 20] is used to reduce noise in an image. It operates one pixel in the image at a time and looks at its closest neighbors to decide whether or not it is representative of its surroundings. To begin with, one should decide the size of the window that the filter operates the image within. The size could, for example, be set to three, which means that the filter will operate on a centered pixel surrounded by a frame of 33 neighbors. Then the filter sorts the pixels contained in the image area surrounded by the window. The center pixel will be replaced by

282


the median, the middle value, of the ranking result. The advantage of the median filter, compared with other smoothing filters of similar size, is that it performs noise-reduction with considerably less blurring. Thus, the filter also preserves the edges in an image very well. The median filter works especially well for random noise. The algorithm works as follows: it first finds out the concrete position of the noised pixel according to the firing pattern and then removes the noise from the image with median filter. Initially the threshold of all of the neurons is set to zero, and at the first iteration all the neurons are activated which means all neurons receive the maximal linking input at the next iteration. So the proper set of the PCNN’s parameters will make the neurons correspond to noised pixels with high intensity fire before its neighborhood at the second iteration, and according to the current firing pattern the concrete position of noised pixels can be found out. Then the noised pixels can be removed with 33 median filter. The removal of noised pixels with low intensity is the same as the removal of noised pixels with high intensity if the intensity is inverted. Due to the fact that this algorithm can find out the concrete positions of noised pixels and apply median operation only on the noised regions, its ability to keep the details of the image is strong, for more details, reader may consult [19, 20]. PCNN boundary detection algorithm The success of the application of PCNNs to image segmentation depends on the proper setting of the various parameters of the network, such as the linking parameter β thresholds θ, decay time constants αθ , and the interconnection matrices M and W . The image can be represented as an array of M × N normalized intensity values. Then the array is fed in at the M × N inputs of the PCNN. If initially all neurons are set to 0, the input results in activation of all of the neurons at the first iteration. The threshold of each neuron, Θ, significantly increases when the neuron fires, then the threshold value decays with time. When the threshold falls below the respective neuron’s potential (U ), the neuron fires again, which again raises the threshold. The process continues creating binary pulses for each neuron. We observe Table 1. Chamber area decision table PCNN parameters β αF αL αΘ VF VL VΘ N

values 0.2 0.001 1 10 0.01 0.01 2 5


(a) anterior chamber angle

283

(b) glaucoma parameters

Fig. 5. Relationship between the Angle TIA and AOD500 Table 2. Summary of the definition of variables in this study Variables TIA

AOD500 ARA500

Definition Angle between the arms passing through a point on the trabecular meshwork 500 μm from the scleral spur and the point perpendicularly opposite on the iris Length of a perpendicular from the trabecular meshwork on the iris at a point 500 μm from the scleral spur The total area bounded by the iris and cornea at 500 μm from the scleral spur was calculated as the angle-recess area

that the visible difference between the enhanced image and the original image is not too drastic. The results corresponding to the segmentation without preprocessing does result in blank image but with preliminary preprocessing, it does not result in blank image. The PCNN parameter values used in this application is given in table (1). 3.2

Clinical Parameters Assessment Phase

The second phase of the experiment shows the clinical parameters assessment. The degree of angle opening was measured using the following variables: trabecular-iris angle (TIA), the angle-opening distance (AOD) at 500 micron from the scleral spur (AOD500), and angle-recess area (ARA500), as described by Palvin et al [24, 25]. Figure 5(a) shows UBM image of the anterior chamber angle demonstrating the angle-recess area and Figure 5(b) illustrates the Glaucoma parameters to be measured. A summary of the definition of variables is shown in Table 2. Clinical parameters assessment algorithm We designed an algorithm to identify the sclera spur, and then automatically calculate the distance along a perpendicular line drawn from the corneal

284


Algorithm 1. Clinical parameters assessment algorithm Input: the enhanced UBM glaucoma image. Output: normal or glaucomatous eye. 1: Draw the anterior chamber boundary 2: Locate the Sclera spur point (Apex point) 3: Draw a line of 25 pixels parallel to the x-axis and at the end of this line draw a perpendicular line to intersect the upper boundary and the lower boundary of the anterior chamber region, then calculate the distances d1 and d2 4: Calculate the distances xx and xxxx sing the Euclidian rule 5: Calculate the angle 1 and angle 2, then angle a = angle 1+angle2 6: From the apex point draw a line of 25 pixels on the upper boundary of the anterior chamber, then find the distance z= xx/cos (angle a), also find the distance y = sin (angle a) 7: Calculate the Angle-recess area (ARA500) =1/2(xy)

endothelial surface to the iris at 500 μm yielding the AOD500 μm. The total area bounded by the iris and cornea at 500 μm from the sclera spur was calculated as the angle-recess area (ARA500). Also, the TIA was measured from the apex point. Then the measured TIA and AOD500 parameters are fed to the classifier to classify the cases as normal and glaucomatous eye. Figure (5). shows a schematic diagram for the calculations of the glaucoma clinical Parameters and the main steps for clincal parametrs assesment are given in algorithm 1. The angles of patients were categorized as Grade 0 to Grade 4, using Shaffer’s classification[3]. These angles were quantified by ultrasound biomicroscopy (UBM) using the following biometric characteristics: Angle opening distance 500 μm (AOD500) from the scleral spur and Angle Recess Area (ARA) [3, 26]. The angles were further segregated as narrow angles (Schaffer’s Grade 2 or less) and open angles (Schaffer’s Grade 3 and 4). 3.3 •

•

•

Rough Set Data Analysis Phase Pre-processing stage(Activities in Dark Gray). This stage includes tasks such as extra variables addition and computation, decision classes assignments, data cleansing, completeness, correctness, attribute creation, attribute selection and discretization. Analysis and Rule Generating stage(Activities in Light Gray). This stage includes the generation of preliminary knowledge, such as computation of object reducts from data, derivation of rules from reducts, rule evaluation and prediction processes. Classification and Prediction stage(Activities in Lighter Gray). This stage utilize the rules generated from the previous phase to predict the stock price movement


285

Rough set pre-processing stage In this stage, the decision table required for rough set analysis is created. In doing so, a number of data preparation tasks such as data conversion, data cleansing, data completion checks, conditional attribute creation, decision attribute generation, discretization of attributes are performed. Data splitting is also performed creating two randomly generated subsets: one subset for analysis containing %75 of the objects in the data set and one validation containing the remainder %25 of the objects. It must be emphasized that data conversion performed on the initial data must generate a form in which specific rough set tools can be applied. 3.4

Data Completion Process

Often, real world data contain missing values. Since rough set classification involves mining for rules from the data, objects with missing values in the data set may have undesirable effects on the rules that are constructed. The aim of the data completion procedure is to remove all objects that have one or more missing values. Incomplete data or information systems exist broadly in practical data analysis, and approaches to complete the incomplete information system through various completion methods in the preprocessing stage that are normal in data mining and knowledge discovery. However, these methods may result in distorting the original data and knowledge, and can even render the original data to be un-minable. To overcome these shortcomings inherent in the traditional methods, we used the decomposition approach for incomplete information system ( i.e. decision table) proposed in [27]. 3.5

Data Discretization Process: RSBR Algorithm

Attributes in concept classification and prediction, may have varying importance in the problem domain being considered. Their importance can be pre-assumed using auxiliary knowledge about the problem and expressed by properly chosen weights. However, when using rough set approach for concept classification, rough set avoids any additional information aside from what is included in the information table itself. Basically, rough set approach tries to determine from the available data in the decision table whether all the attributes are of the same strength and, if not, how they differ in respect of the classifier power. Therefor, some strategies for discretization of real value attributes have to be used when we need to apply learning strategies for data classification with real value attributes (e.g. equal width and equal frequency intervals). It has been shown that the quality of learning algorithm depends on this strategy. Discretization uses data transformation procedure which involves finding cuts in the data sets that divide the data into intervals. Values lying within an interval are then mapped to the same value. Performing this process leads to reducing the size of the attributes

286


value set and ensures that the rules that are mined are not too specific. For the discretization of continuous-valued attributes, we adopt, in this chapter, rough sets with boolean reasoning (RSBR) algorithm proposed by Zhong et al. [27] The main advantage of RSBR is that it combines discretization of real valued attributes and classification. (more detalis refer to [13]). Analysis and Rule Generating Stage As we mentioned before, Analysis and Rule Generating stage includes generating preliminary knowledge, such as computation of object reducts from data, derivation of rules from reducts, and prediction processes. These stages lead towards the final goal of generating rules from information system or decision table. Reduce Irrelevant and Redundant Attributes In decision tables, there often exist conditional attributes that do not provide (almost) any additional information about the objects. These attributes need to be removed in order to reduce the complexity and cost of decision process [17, 18, 28]. A decision table may have more than one reduct. And any of these reducts could be used to replace the original table. However, finding all the reducts from a decision table is NP-complete but fortunately, in applications, it is usually not necessary to find all of them – one or a few of them are sufficient. Selecting the best reduct is important. The selection depends on the optimality criterion associated with the attributes. If a cost function could be assigned to attributes, then the selection can be based on the combined minimum cost criteria. But in the absence of such cost function, the only source of information to select the reduct from, is the contents of the table. In this chapter, we adopt the criteria that the best reducts are those with minimal number of attributes and if there are more such reducts with the least number of combinations of values of its attributes cf. [28]. In general, rough set theory provides useful techniques to reduce irrelevant and redundant attributes from a large database with a lot of attributes. The dependency degree (or approximation quality, classification quality) and the information entropy are two most common attribute reduction measures in rough set theory. In this chapter, we use the dependency degree measure to compute the significant features and measuring the effect of removing a feature from the feature sets. Computation of the Reducts A reduced table can be seen as a rule set where each rule corresponds to one object of the table. The rule set can be generalized further by applying rough set value reduction method. The main idea behind this method is to


287

drop those redundant condition values of rules and to unite those rules in the same class. Unlike most value reduction methods, which neglect the difference among the classification capabilities of condition attributes, we first remove values of those attributes that have less discrimination factors. Thus more redundant values can be reduced from decision table and more concise rules can be generated. Rule Generation from a Reduced Table The generated reducts are used to generate decision rules. The decision rule, at its left side, is a combination of values of attributes such that the set of (almost) all objects matching this combination has the decision value given at the rule’s right side. The rule derived from reducts can be used to classify the data. The set of rules is referred to as a classifier and can be used to classify new and unseen data. The quality of rules is related to the corresponding reduct(s). We are especially interested in generating rules which cover the largest parts of the universe U . Covering U with more general rules implies smaller size rule set. Classification and Prediction Stage Classification and prediction is the last stage of our proposed approach. We present a classification and prediction scheme based on the methods and techniques described in the previous sections. To transform a reduct into a rule, one only has to bind the condition feature values of the object class from which the reduct originated to the corresponding features of the reduct. Then, to complete the rule, a decision part comprising the resulting part of the rule is added. This is done in the same way as for the condition features. To classify objects, which has never been seen before, rules generated from a training set will be used. These rules represent the actual classifier. This classifier is used to predict to which classes new objects are attached. The nearest matching rule is determined as the one whose condition part differs from the feature vector of re-object by the minimum number of features. When there is more than one matching rule, we use a voting mechanism to choose the decision value. Every matched rule contributes votes to its decision value, which are equal to the t times number of objects matched by the rule. The votes are added and the decision with the largest number of votes is chosen as the correct class. Quality measures associated with decision rules can be used to eliminate some of the decision rules. Rule Strength Measures The global strength defined in [29] for rule negotiation is a rational number in [0, 1] representing the importance of the sets of decision rules relative to the considered tested object. Let us assume that T = (U, A (d)) is a given

288


decision table, ut is a test object, Rul(Xj ) is the set of all calculated basic decision rules for T , classifying objects to the decision class Xj (vdj = vd ), M Rul(Xj , ut ) ⊆ Rul(Xj ) is the set of all decision rules from Rul(Xj ) matching tested object ut . The global strength of decision rule set M Rul(Xj , ut) is defined by the following form [29]: j r⊂MRul(Xj ,ut ) |P red(r)|A ∩ |d = vd |A M Rul(Xj , ut ) = . (1) j |d = vd |A Measure of strengths of rules defined above is applied in constructing classification algorithm. To classify a new case, rules are first selected matching the new object. The strength of the selected rule sets is calculated for any decision class, and then the decision class with maximal strength is selected, with the new object being classified to this class.

4 Implementation and Results Evaluation 4.1

UBM Images Characteristic

The UBM images were from the New York Glaucoma Research Institute, obtained with the UBM Model 840, Paradigm Medical Industries Inc, with a 50 MHz transducer probe. The image has a lateral and axial physical resolution of approximately 50 μ and 25 μ respectively and a penetration depth of 4-5 mm, typically of dimensions 5 x 5 mm at a resolution of 440 x 240 pixels. Twenty images were used in the verification of the technique. The technique was implemented on PC with a 3 GHz P4 processor using MATLAB 7.01. 4.2

PCNN: Chamber Boundary Detection Results

Preprocessing results: In the first phase of the experiment, the UBM eye images have been preprocessed to remove noise. Eye structure in UBM images are not very clear which makes them very challenging to analysis, both for naked human eye as well as any automatic assessment algorithm. It can be seen that with the preprocessing module which removes image noise, smoothes images and enhances the image resolutions, the performance of the segmentation module can be significantly improved. Table 3 shows the results of the developed PCNN enhancement and boundary detection technique in 2D UBM images. Table 3(a) is the original image. After noise removal and image enhancement by the preprocessing module, the output image is shown in Table 3(b). Table 3(c) shows the boundary of the anterior chamber on the original image. Table 3(d) shows the boundary of the anterior chamber alon.


289

Table 3. Determination of chamber boundaries

a) Original

b) PCNN enhanced

c) segmentation

d) boundaries

Table 4. Chamber area decision table Angle-TIA 45.43 24.8 13.68 13.6 24.58 56.4 37.44

4.3

AOD500 28.161 11.78 6.13 6.05 11.52 48.19 20.61

ARA 63.04 150.17 77.66 75.89 145.03 771.28 277.53

Decision class 1 0 0 0 0 1 1

Rough Sets Data Analysis Results

Table (4) represents the Chamber area rough decision system. We reach the minimal number of reducts that contains a combination of attributes which has the same discrimination factor. The final generated reduct set which is used to generate the list of rules for the classification is: {TIA, with support 100%}

290


A natural use of a set of rules is to measure how well the ensemble of rules is able to classify new and unseen objects. To measure the performance of the rules is to assess how well the rules do in classifying new cases. So we apply the rules produced from the training set data to the test set data. The following present the generated rules in a more readable format: R1: IF TIA < 29.94 THEN Decision Class is 0.0 R2: IF TIA >= 29.94 THEN Decision Class is 1.0 Measuring the performance of the rules generated from the training data set in terms of their ability to classify new and unseen objects is also important. Our measuring criteria were Rule Strength and Rule Importance [30] and to check the performance of our method, we calculated the confusion matrix between the predicted classes and the actual classes as shown in Table (5). The confusion matrix is a table summarizing the number of true positives, true negatives, false positives, and false negatives when using classifiers to classify the different test objects. Several runs were conducted using different setting with strength rule threshold. Table (6) shows the number of generated rules using rough sets and for the sake of comparison we have also generated rules using neural network. Table (6) indicates that the number of rules generated using neural networks is much larger than the rough sets. Comparative analysis To evaluate the efficiency of the classification developed method, we compared the results obtained using our classifier with those manually defined by an expert. It is noticed that for the most part, our measurement and the radiologist measurement agree. Also, Our analysis has been compared with the analysis of the anterior chamber angle using UBM Pro2000 software (Paradigm Medical Industries Inc, Salt Lake City, Utah) [refer Figure 6] [31, 32]. After the observer selects the scleral spur, the program automatically detects the border and calculates the angle recession area at 500 μm anterior to the scleral spur. Table 5. Model Prediction Performance (Confusion Matrix) Actual Class 0 Class 1

Predict Class 0 17 0 1.0

Predict Class 1 0 32 1.0

Accuracy 1.0 1.0 1.0


291

Table 6. Number of generated rules Method Neural networks 37 Rough sets 2

Generated rule numbers

Fig. 6. UBMPro2000 [31, 32]

5 Conclusion and Future Works We have developed an advanced hybrid rough pulse coupled neural network scheme for Ultrasound Biomicroscopy glaucoma images analysis and provided a methodology for assessing the clinical parameters of angle closure glaucoma based on UBM images of the eye. To increase the efficiency of the introduced hybrid scheme, an intensity adjustment process is applied first, based on the Pulse Coupled Neural Network with a median filter. This is followed by applying the PCNN-based segmentation algorithm to detect the boundary of the prostate image. Combining the adjustment and segmentation enables us to eliminate PCNN sensitivity to the setting of the various PCNN parameters whose optimal selection can be difficult and can vary even for the same problem. Then, chamber boundary features have been extracted and normalized, followed by application of a rough set analysis to discover the dependency between the attributes and to generate set of reduct that contains minimal number of attributes. Experimental results showed that the introduced scheme is very successful and has high detection accuracy. It is believed that the proposed automatic scheme for glaucoma clinical parameters assessment and classification for the UBM images is a promising approach, which provides an efficient and robust assessment and diagnosis strategy and acts as second opinion for the physician’s interpretation of glaucoma diseases. In conclusion, the analysis of the UBM images is a useful method for evaluating the chamber angle structure of eye.

292


Acknowledgments The authors wish to acknowledge the valuable feedback provided by Prof. Taymoor, The University of Ain Shams, during the development of the proposed algorithm. They would also like to acknowledge the help provided by Prof. Adel Abdel-Shafeek from Faculty of medicin-Ain Shams University for the manual measurements of ultrasound images. They wish to thank Dr. Zaher Hussein (Ophthalmology-Glaucoma Specialist) from the (New York Glaucoma Research Institute) New York Eye and Ear infirmary, Glaucoma Service, for providing the UBM images and reports.

References 1. Quigley, H.A., Broman, A.T.: The number of people with glaucoma worldwide in 2010 and 2020. Br. J. Ophthalmol. 90(3), 262–267 (2006) 2. Razeghinejad, M.R., Kamali-Sarvestani, E.: The plateau iris component of primary angle closure glaucoma. Developmental or acquired Medical Hypotheses 69, 95–98 (2007) 3. Kaushik, S., Jain, R., Pandav, S.S., Gupta, A.: Evaluation of the anterior chamber angle in Asian Indian eyes by ultrasound biomicroscopy and gonioscopy. Indian Journal of Ophthalmology 54(3), 159–163 (2006) 4. Quigley, H.A.: Number of people with glaucoma worldwide. Br. J. Ophthalmol. 80, 389–393 (1996) 5. Glaucoma, http://www.theeyecenter.com 6. Nishijima, K., Takahashi, K., Yamakawa, R.: Ultrasound biomicroscopy of the anterior segment after congenital cataract surgery. American Journal of Ophthamology 130(4), 483–489 (2000) 7. Radhakrishnan, S., Goldsmith, J., Huang, D., Westphal, V., Dueker, D.K., Rollins, A.M., Izatt, J.A., Smith, S.D.: Comparison of optical coherence tomography and ultrasound biomicroscopy for detection of narrow anterior chamber angles. Arch. Ophthalmol. 123(8), 1053–1059 (2005) 8. Urbak, S.F.: Ultrasound Biomicroscopy. I. Precision of measurements. Acta Ophthalmol Scand 76(11), 447–455 (1998) 9. Deepak, B.: Ultrasound biomicroscopy ”An introduction”. Journal of the Bombay Ophthalmologists Association 12(1), 9–14 (2002) 10. Zhang, Y., Sankar, R., Qian, W.: Boundary delineation in transrectal ultrasound image for prostate cancer. Computers in Biology and Medicine 37(11), 1591–1599 (2007) 11. Youmaran, R., Dicorato, P., Munger, R., Hall, T., Adler, A.: Automatic detection of features in ultrasound images of the Eye. In: IMTC, Proceedings of the IEEE, Ottawa, Canada, May 16-19, 2005, vol. 3, pp. 1829–1834 (2005) 12. Hasanien, A.E.: Classification and feature selection of breast cancer data based on decsion tree algorithm. International Journal of Studies in Informatics and Control Journal 12(1), 33–39 (2003) 13. Hassanien, A.E.: Fuzzy-rough hybrid scheme for breast cancer detection. Image and computer vision journal 25(2), 172–183 (2007) 14. Basheer, I.A., Hajmeer, M.: Artificial neural networks: fundamentals, computing, design, and Application. Journal of Microbiological Methods 43, 3–31 (2000)


293

15. Haykin, S.: Neural Networks: A Comprehensive Foundation. IEEE Press, Los Alamitos (1994) 16. Pal, S.K., Polkowski, S.K., Skowron, A. (eds.): Rough-Neuro Computing: Techniques for Computing with Words. Springer, Berlin (2002) 17. Pawlak, Z.: Rough Sets. Int. J. Computer and Information Sci. 11, 341–356 (1982) 18. Grzymala-Busse, J., Pawlak, Z., Slowinski, R., Ziarko, W.: Rough Sets. Communications of the ACM 38(11), 1–12 (1999) 19. El-dahshan, E., Redi, A., Hassanien, A.E., Xiao, K.: Accurate Detection of Prostate Boundary in Ultrasound Images Using Biologically inspired Spiking Neural Network. In: International Symposium on Intelligent Siganl Processing and Communication Systems Proceeding, Xiamen, China, November 28December 1, pp. 333–336 (2007) 20. Hassanien, A.E.: Pulse coupled Neural Network for Detection of Masses in Digital Mammogram. Neural Network World Journal 2(6), 129–141 (2006) 21. Eckhorn, R., Bauer, R., Jordan, W., Brosch, M., Kruse, W., Munk, M., Reitboeck, H.J.: Coherent oscillations: A mechanism of feature linking in the visual cortex? Biol. Cybern. 60, 121–130 (1988) 22. Eckhorn, R., Reitboeck, H.J., Arndt, M.: Feature Linking via Synchronization among Distributed Assemblies: Simulations of Results from Cat Visual Cortex. Neural Comp. 2, 293–307 (1990) 23. Eckhorn, R.: Neural mechanisms from visual cortex suggest basic circuits for linking field models. IEEE Trans. Neural Networks 10, 464–479 (1999) 24. Pavlin, C.J., Harasiewicz, K., Foster, F.S.: Ultrasound biomicroscopy of anterior segment structures in normal and glaucomatous eyes. Am. J. Ophthalmol. 113, 381–389 (1992) 25. Hodge, A.C., Fenstera, A., Downey, D.B., Ladak, H.M.: Prostate boundary segmentation from ultrasound images using 2D active shape models: Optimisation and extension to 3D. Computer Methods and Programs in Biomedicine 8(4), 99–113 (2006) 26. Gohdo, T., Tsumura, T., Iijima, H., Kashiwagi, K., Tsukahara, S.: Ultrasound biomicroscopic study of ciliary body thickness in eyes with narrow angles. American Journal of Ophthamology 129(3), 342–346 (2000) 27. Qizhong, Z.: An Approach to Rough Set Decomposition of Incomplete Information Systems. In: 2nd IEEE Conference on Industrial Electronics and Applications, ICIEA 2007, May 23-25, 2007, pp. 2455–2460 (2007) 28. Setiono, R.: Generating concise and accurate classification rules for breast cancer diagnosis. Artificial Intelligence in Medicine 18(3), 205–219 (2000) 29. Bazan, J., Nguyen, H.S., Nguyen, S.H., Synak, P., Wr´ oblewski, J.: Rough Set Algorithms in Classification Problem. In: Polkowski, L., Tsumoto, S., Lin, T.Y. (eds.) Rough Set Methods and Applications, pp. 49–88. Physica Verlag (2000) 30. Ning, S., Xiaohua, H., Ziarko, W., Cercone, N.: A Generalized Rough Sets Model. In: Proceedings of the 3rd Pacific Rim International Conference on Artificial Intelligence, vol. 431, pp. 437–443. Int. Acad. Publishers, Beijing (1994) 31. Sbeity, Z., Dorairaj, S.K., Reddy, S., Tello, C., Liebmann, J.M., Ritch, R.: Ultrasound biomicroscopy of zonular anatomy in clinically unilateral exfoliation syndrome. Acta Ophthalmol. 86(5), 565–568 (2008) 32. Dorairaj, S.K., Tello, C., Liebmann, J.M., Ritch, R.: Narrow Angles and Angle Closure: Anatomic Reasons for Earlier Closure of the Superior Portion of the Iridocorneal Angle. Acta Ophthalmol. 125, 734–739 (2007)

An Overview of Fuzzy C-Means Based Image Clustering Algorithms Huiyu Zhou1 and Gerald Schaefer2 1

2

School of Engineering and Design Brunel University Uxbridge, U.K [email protected] Department of Computer Science Loughborough University Loughborough, U.K [email protected]

Summary. Clustering is an important step in many imaging applications with a variety of image clustering techniques having been introduced in the literature. In this chapter we provide an overview of several fuzzy c-means based image clustering concepts and their applications. In particular, we summarise the conventional fuzzy c-means (FCM) approaches as well as a number of its derivatives that aim at either speeding up the clustering process or at providing improved or more robust clustering performance.

1 Introduction Image clustering is widely performed in a variety of applications like computer vision, robotics, medical imaging and information retrieval. It can be seen as a process of grouping an image into non-overlapping homogeneous regions that hold consistent characteristics such as gray level, colour or texture. Fuzzy c-means (FCM) is one of the most popular methods for image clustering [3]. Compared to hard thresholding clustering methods, FCM is capable of reducing the uncertainty of pixels belonging to one class and therefore in general providing improved clustering outcomes. In addition, FCM enables multiple classes with varying degrees of membership to be continuously updated [4]. As an unsupervised method, FCM does not require a priori labeling of some patterns to categorise others or infer the cluster structure of the whole data. Apart from the original fuzzy c-means algorithm [3], a number of FCM derivatives have been introduced in the literature. These either target at algorithmic speedup (e.g., fast FCM with random sampling [7] and fast generalized FCM [30], or improvement of clustering performance with respect to noise or artefacts [4] (e.g. probabilistic clustering [24], fuzzy noise clustering [13], LP norm clustering [19]). A.-E. Hassanien et al. (Eds.): Foundations of Comput. Intel. Vol. 2, SCI 202, pp. 295–310. c Springer-Verlag Berlin Heidelberg 2009 springerlink.com

296

H. Zhou and G. Schaefer

Sato and Sato [29] presented a fuzzy clustering algorithm for interactive fuzzy vectors. Hathaway et al. [18] and Pedrycz et al. [28] proposed algorithms that can be used to convert parametric or non-parametric linguistic variables into generalised coordinates before performing fuzzy c-means clustering. Yang and Ko [35] proposed fuzzy c-numbers clustering procedures for segmenting data while Yang and Liu [36] extended this to high-dimensional fuzzy vectors. Takata et al. [31] proposed a clustering method for data with uncertainties using the Hausdorff distance. They also suggested fuzzy clustering procedures for data with uncertainties using minimum and maximum distances based on L1 metric [32]. Auephanwiriyakul and Keller [2] presented a linguistic version of the fuzzy c-means method, based on the extension principle and the decomposition theorem. Hung and Yang [21] suggested fuzzy c-numbers clustering algorithm for LR-type fuzzy number based on exponential-type distance measure. Novel clustering techniques for handling both symbolic and fuzzy data were proposed by Yang et al. [37]. Their fuzzy clustering algorithms, for mixed features of symbolic and fuzzy data, are obtained by modifying the Gowda and Diday’s dissimilarity measure for symbolic data [17] while changing the parametric approach for fuzzy data proposed in [18]. Recently, there is increasing interest in developing fuzzy clustering models for three-way fuzzy data. Coppi and D’Urso presented a fuzzy c-means clustering approach for fuzzy time trajectories that form a geometrical representation of the fuzzy data time array [11].

2 Classical Fuzzy C-Means Clustering Fuzzy c-means is based on the idea of finding cluster centres by iteratively adjusting their positions and evaluation of an objective function which is typically defined as C N E= μkij ||xi − cj ||2 (1) j=1 i=1

μkij

where is the fuzzy membership of sample (or pixel) xi and the cluster identified by its centre cj , and k is a constant that defines the fuzziness of the resulting partitions. E can reach the global minimum when pixels nearby the centroid of corresponding clusters are assigned higher membership values, while lower membership values are assigned to pixels far from the centroid [8]. In here, the membership is proportional to the probability that a pixel belongs to a specific cluster where the probability is only dependent on the distance between the image pixel and each independent cluster centre. The membership functions and the cluster centres are updated by μij = C

m=1

1 ||xj −ci || ||xj −cm ||)2/(k−1)

(2)

An Overview of FCM Based Image Clustering Algorithms

and

N

j=1

ci = N

ukij xj

j=1

ukij

297

(3)

The steps involved in fuzzy c-means image clustering are [3]: Step 1: Initialise the cluster centres ci and let t = 0. Step 2: Initialise the fuzzy partition memberships functions μij according to Equation (2). Step 3: Let t = t + 1 and compute new cluster centres ci using Equation (3). Step 4: Repeat Steps 2 to 3 until convergence. An initial setting for each cluster centre is required and FCM can be shown to converge to a local minimisation solution. The efficiency of FCM has been investigated in [20]. To effectively address the inefficiency of the algorithm several variants of the fuzzy c-means algorithm have been introduced.

3 Fast FCM Clustering with Random Sampling To combat the computational complexity of FCM, Cheng et al. [7] proposed a multistage random sampling strategy. This method has a lower number of feature vectors and also needs fewer iterations to converge. The basic idea is to randomly sample and obtain a small subset of the dataset in order to approximate the cluster centres of the full dataset. This approximation is then used to reduce the number of iterations. The random sampling FCM algorithm (RSFCM) consists of two phases. First, a multistage iterative process of a modified FCM is performed. Phase 2 is then a standard FCM with the cluster centres approximated by the final cluster centres from Phase 1. Phase 1 Let XΔ% be a subset whose number of subsamples is Δ% of the N samples contained in the full dataset X and denote the number of stages as n. 1 and 2 are parameters used as stopping criteria. After the following steps the dataset (denoted as X(ns ∗Δ%) ) will include N ∗ Δ% samples: Step 1: Select X(Δ%) from the set of the original feature vectors matrix (z = 1). Step 2: Initialise the fuzzy memberships functions μij using Equation (2) with X(z∗Δ%) . Step 3: Compute the stopping condition = 1 -z∗((1 -2 )/ns ) and let j = 0 Step 4: Set j = j + 1 Step 5: Compute the cluster centres c(z∗Δ%) using Equation (3). Step 6: Compute μ(z∗Δ%) using Equation (2). Step 7: If ||μj(z∗Δ%) − μj−1 (z∗Δ%) || ≥ , then go to Step 4. Step 8: If z ≤ ns then select another X(Δ%) and merge it with the current X(z∗Δ%) and set z = z + 1, otherwise move to Phase 2 of the algorithm.

298


Phase 2 Step 1: Initialise μij using the results from Phase 1, i.e. c(ns ∗Δ%) with Equation (3) for the full data set Step 2: Go to Steps 3 of the conventional FCM algorithm and iterate the algorithm stopping criterion 2 is met. It has been shown that this improved algorithm is able to reduce the computation requested in the classical FCM method. Other variants of this multistage random sampling FCM framework have also been developed and can be found e.g. in [14] and [23].

4 Fast Generalized FCM Clustering Ahmed et al. [1] introduced an alternative to the classical FCM by adding a term that enables the labelling of a pixel to be associated with its neighbourhood. As a regulator, the neighbourhood term can change the solution towards piecewise homogeneous labelling. As a further extension of this work, Szil´ agyi et al. [30] reported their EnFCM algorithm to speed up the clustering process for black-and-white images. In order to reduce the computational complexity, a linearity-weighted sum image g is formed from the original image, and the local neighbour average image evaluated as ⎛ ⎞ 1 ⎝ α gm = xm + xj ⎠ (4) 1+α NR j∈Nr

where gm denotes the gray value of the m-th pixel of the image g, xj represents the neighbours of xm , NR is the cardinality of a cluster, Nr represents the set of neighbours falling into a window around xm . The objective function used for clustering image g is defined as J=

qc C

2 γl μm ij (gl − ci )

(5)

i=1 i=1

where qc denotes the number of the gray levels in the image, and γl is the number of the pixels having an intensity equal to l with l = 1, 2, . . . , qc . Thus, qc C γ = N under the constraint that l=1 l i=1 μij = 1 for any l. Finally, we can obtain the following expressions for membership functions and cluster centres [4]:.

and

(gl − si )−2/m−1 μil = C −2/m−1 j=1 (gl − sj )

(6)

qc γl μm il gl si = l=1 qc m γ μ l=1 l il

(7)


299

EnFCM considers a number of pixels with similar intensities as a weight. Thus, this process may accelerate the convergence of searching for global similarity. On the other hand, to avoid image blur during the clustering, which may lead to inaccurate clustering, Cai et al. [4] use a measure Sij , s which incorporates the local spatial relationship Sij and the local gray-level g relationship Sij , and is defined as

s g Sij × Sij , j = i Sij = (8) 0, j=i with s Sij

= exp

− max(|pcj − pci |, |qcj − qci |) λs

and g Sij = exp

−||xi − xj ||2 λg × σg2

(9)

(10)

where (pci , qci ) describe the co-ordinates of the i-th pixel, σg is a global s scale factor of the spread of Sij , and λs and λg represent scaling factors. Sij replaces α in Equation (4). Hence, the newly generated image g is updated as j∈Ni Sij xj gi = (11) Sij and is restricted to [0, 255] due to the denominator. Given a pre-defined number of clusters C and a threshold value > 0, the fast generalised FCM algorithm proceeds in the following steps: Step 1: Initialise the clusters cj . Step 2: Compute the local similarity measures Sij using Equation (8) for all neighbours and windows over the image. Step 3: Compute linearly-weighted summed image g using Equation (11). Step 4: Update the membership partitions using Equation (6). Step 5: Update the cluster centres ci using Equation (7). C Step 6: If i=1 ||ci(old) − ci(new ||2 > go to Step 4. Similar efforts to improve the computational efficiency and robustness have also been reported in [25] and [5].

5 Anisotropic Mean Shift Based FCM Clustering An approach to fuzzy c-means clustering that utilises an anisotropic mean shift algorithm coupled with fuzzy clustering was recently introduced by Zhou et al. [41, 40]. Mean shift based techniques have been shown to be capable of estimating the local density gradients of similar pixels. These gradient estimates are iteratively performed so that all pixels can find similar pixels in

300


the same image [9, 10]. A standard mean shift approach method uses radially symmetric kernels. Unfortunately, the temporal coherence will be reduced in the presence of irregular structures and noise in the image. This reduced coherence may not be properly detected by radially symmetric kernels and thus, an improved mean shift approach, namely anisotropic kernel mean shift [33], provides better performance. In mean shift algorithms the image clusters are iteratively moved along the gradient of the density function before they become stationary. Those points gathering in an outlined area are treated as the members of the same cluster. A kernel density estimate is defined by N 1 ˜ f (x) = K(x − xi ), N

(12)

K(x) = |H|−0.5 K(H −0.5 x),

(13)

i=1

with

where N is the number of samples, and xi stands for a sample from an unknown density function f . K(·) is the d-variate kernel function with compact support satisfying the regularity constraints, and H is a symmetric positive definite d × d bandwidth matrix. Usually, we have K(x) = ke (φ), where ke (φ) is a convex decreasing function, e.g. for a Gaussian kernel ke (φ) = ct e−φ/2

(14)

and for an Epanechnikov kernel, ke (φ) = ct max(1 − φ, 0)

(15)

where ct is a normalising constant. If a single global spherical bandwidth is applied, H = h2 I (I is the identity matrix), then we have f˜(x) =

N 1 x − xi K N hd i=1 h

(16)

Since the kernel can be divided into two different radially symmetric kernels, we have the kernel density estimate as N 1 1 α α α α β f˜(x) = ||(cβi − xβi )/(hβ (Hiα ))||2 α q k (d(ci , xi , Hi ))k β N i=1 h (Hi ) (17) where and α and β denote the spatial and temporal components respectively α α and d(cα i , xi , Hi ) is the Mahalanobis metric, i.e. α−1 α α α α α T d(cα (xi − cα i , xi , Hi ) = (xi − ci ) Hi i ).

(18)


301

Anisotropic mean shift is intended to modulate the kernels during the mean shift procedure. The objective is to keep reducing the Mahalanobis distance so as to group similar samples as much as possible. First, the anisotropic bandwidth matrix Hiα is estimated with the following constraints:

α ke (d(x, xi , Hiα )) < 1

(19) β ke ||(x − xi )/hβ (Hiα )||2 < 1 The bandwidth matrix can be decomposed to Hiα = λV AV T

(20)

where λ is a scalar, V is a matrix of normalised eigenvectors, and A is a diagonal matrix of eigenvalues whose diagonal elements ai satisfy p

ai = 1

(21)

i=1

Fig. 1. Fuzzy clustering for segmentation of skin lesions: original image (upper left), ground truth (upper right), FCM (middle left), RSFCM (middle right), EnFCM (bottom left) and AMSFCM (bottom right)

302


The bandwidth matrix is updated by adding more and more points to the computational list: if these points are similar in intensity or colour, then the Mahalanobis distance will be consistently reduced. Otherwise, if the Mahalanobis distance is increased, these points will not be considered in the computation. Anisotropic mean shift based FCM (AMSFCM) proceeds in the following steps: Step Step Step Step Step

1: 2: 3: 4: 5:

Initialise the cluster centres ci . Let j = 0. Initialise the fuzzy partitions μij using Equation (2). Set j = j + 1 and compute ci using Equation (3) for all clusters. Update μij using Equation (2). For each pixel xi determine anisotropic kernel and related colour radius using Equations (17) and (20). Note that mean shift is applied to the outcome image of FCM. Step 6: Calculate the mean shift vector and then iterate until the mean shift, M + (xi ) − M − (xi ), is less than a pixel considering the previous position and a normalised position change: M + (xi ) = νM − (xi ) + (1 − ν)

β β N − − β α 2 j=1 (xj −M (xi ))||(M (xi )−xj )/(h Hj )|| N β β α − β 2 j=1 ||(M (xi )−xj )/(h Hj )||

with ν = 0.5. Step 7: Merge pixels with similar colour. Step 8: Repeat Steps 3 to 6 until convergence.

In Fig. 1 we show the application of various FCM algorithms on dermoscopic images of skin lesions [40]. It can be seen that fuzzy clustering methods are able to accuractely segment the skin lesion.

6 RCFCM and S-FCM Clustering Wei and Xie [34] addressed the low convergence speed of FCM using a competitive learning approach to developed their RCFCM algorithm. The main idea of their algorithm is to magnify the largest membership degree while suppressing the second largest membership degree. This scheme has been found to be able to accelerate the clustering convergence. The key step of RCFCM is to add the following process after conducting steps 1 to 2 of the classical FCM: Step 2+ : Modify membership degree matrix U (k), where U (k) = {μij }. Considering xj , if the degree of membership of xj belonging to the pth cluster is the largest of the overall clusters with the value μpj . In the meanwhile, the degree of membership of it belonging to q-th cluster is the second largest of the overall clusters with the value μqj . After being modified, the degree of membership of xj belonging to each cluster is (0 ≤ α ≤ 1): μpj = μpj + (1 − α)μqj μsj = αμsj

(22)


303

Although RCFCM has some advantages, some problems still appear. For example, values of μij preserve the ranks, i.e. if dij ≤ drj , then μij ≤ μrj . Thse ranks reflect the relation of the sample belonging to each cluster centre. However, if α has not been properly defined, this can result in a even slower convergence of clustering. To avoid this, Equation 22 can be modified as μpj = 1 − α μqj i=p

μij

= 1 − α + αμpj = αμij , i = p

(23)

as is done in the S-FCM algorithm [15] which also rewards the largest membership but suppresses the others. Interestingly, if α = 0, the proposed algorithm becomes the classical hard c-means (HCM) algorithm, while if α = 1 it takes on the form of FCM. Therefore, the algorithm holds a balanced point between HCM and FCM while the determination of α will dominate the convergence of S-FCM.

7 Spatially Weighted FCM Clustering Yang et al. proposed a global image clustering algorithm called SWFCM [38]. This algorithm is formulated by incorporating spatial information into the classical FCM algorithm. The weight used in the algorithm by k-nearest neighbour classifier is modified so as to improve the thresholding performance. A gray level histogram is used to compute the parameters of the FCM. Considering the neighbouring pixels around the central pixel, the fuzzy membership function can be extended to be μ∗ik = μik pik

(24)

with k = 1,2,...,n where n is the index of each pixel, and pik is the probability of data point k belonging to cluster i. Then, the degrees of membership μ∗ik and the cluster centers ci are updated by (μ∗ik )b = J

pik

2/(r−1) j=1 (dik /djk )

(ci )b+1

n ∗ r k=1 ((μik )b ) xk = n ∗ r k=1 ((μik )b )

(25)

(26)

The key issue here is how to define the auxiliary weight variable pik . k-nearest neighbor (k-NN) algorithm [12] is used 2 xn ∈Nki 1/d (xn , k) pik = (27) 2 xn ∈Nk 1/d (xn , k)

304


where Nk is the data set of the nearest neighboring of central pixel k, and Nki is the subset of Nk referring to the data belonging to class i. Given the potential function of each feature vector, we have K(x, xk ) =

1 1 + α||x − xk ||2

where α is a positive constant. Hence the weight value is defined as 2 xn ∈Nki 1/(1 + αd (xn , ci )) pik = 2 xn ∈Nk 1/(1 + αd (xn , ci ))

(28)

(29)

To prevent SWFCM from getting trapped in a local minima it is initialised with a fast FCM algorithm. Once FCM stops, the SWFCM algorithm continues with the updating membership function. In related work, Cheng et al. [6] introduced the concept of fuzziness into a maximum entropy thresholding technique. Zhao et al. [39] presented a direct solution to the search for fuzzy thresholding parameters by exploiting the relationship between the fuzzy c-partition and the probability partition.

8 Lp Norm FCM Clustering It is well known that the quality of the computed cluster centres ci can be degraded due to the effects of noise or outliers in the data sets. This occurs due to the fact that dij = ||xj − ci ||22 which can lead to cluster prototypes being pulled away from the main distribution of the cluster. Kersten [22] and Miyamoto and [27] independently proposed replacing ||xj − ci ||22 with Agusta s 1 ||xj − ci ||1 = k=1 |xkj − cij | in order to improve robustness against outlying data. Lp norm FCM clustering [19] is based on this observation. The objective function employed is hence formulated as Fm,p (U, c) =

I J

Uijm ||xj − ci ||pp

i=1 j=1

=

I J k0

Uijm |xkj − cij |p , m > 1.

(30)

i=1 j=1 k=1

Thus, the datum-to-prototype dissimilarities can be deduced as dij =

s

|xkj − cij |p , i = 1, ..., I; j = 1, ..., J.

(31)

k=1

To compute the v-variable we can optimise the following independent univariate minimisation:


fij (cij ) =

s

m Uki |xkj − cij |p , i = 1, ..., I; j = 1, ..., J.

305

(32)

k=1

For p > 1, the computed value of to be a numerical approximation csij is taken m to the unique zero of fij = −p k=1 Uki |xkj − cij |p−1 sign(xkj − cij ). Lp norma FCM clustering proceeds in the following steps: Step Step Step Step

1: 2: 3: 4:

Initialise the cluster centres ci . Let l = 1. Initialise the fuzzy partitions μij . Estimate ci using Equations (30), (31) and (32). Repeat the above process until (||U new − U old || < 0.00001).

This approach was later extended to integrate the non-Euclidean relational FCM (NERFCM) proposed in [18]. Here, the relational data is represented as s p R = [Rij ] = [||xi − xj ||p ] = [ |xij − xik |p ]. (33) i=1

To handle this data a terminal partition matrix U ∗ is sought with a newer expression as follows: c∗ = arg min Fm,p (U ∗ , c).

(34)

c

9 Probabilistic FCM and FGcM Clustering The original FCM attempts to minimise the objective function as follows: F (L, U ) =

I J

(μij )m d2ij subject to

i=1 j=1

I

μij = 1 ∀j.

(35)

i=1

where L = (β1 ,...,βI ) is a I-tuple of prototypes, I is the number of classes, J is the total number of feature vectors, and U is the fuzzy c-partition matrix. This objective function can be re-formulated to satisfy specific requirements: Fm (L, U ) =

I J i=1 j=1

(μij )m d2ij +

I i=1

ηi

J

(1 − μij )m

(36)

j=1

where ηi are suitable positive numbers. Minimising the left hand side of Equation (36) is equivalent to investigating the following equation: ij 2 m (βi , μij ) = μm Fm ij dij + ηi (1 − μij )

(37)

Differentiating Equation (37) with respect to μij and setting it to zero leads to μij =

1 1 + (d2ij /ηi )1/(m−1)

(38)

306


In practice, the following expression is used so as to reach a good convergence: J m 2 j=1 μij dij ηi = K J (39) m j=1 μij This makes ηi proportional to the average fuzzy intra-cluster distance of cluster βi . K is set to be 1. Therefore, the following rule is valid: 2 xj ∈( i )α dij ηi = (40) |( i )α | where ( i )α is an appropriate α-cut of i is the average intra-cluster distance for all of the “good” feature vectors. Probabilistic FCM algorithm [24] proceeds in the following steps: Step Step Step Step Step Step Step Step

1: 2: 3: 4: 5: 6: 7: 8:

Initialise the cluster centres ci . Let l = 1. Initialise the fuzzy partitions μij . Estimate ηi using Equation (39). Update the prototypes using U l . Compute U l+1 using Equation (38). Increment l. If ||U l−1 − U l || < goto Step 4 . Re-estimate ηi using Equation (40).

Menard et al. [26] proposed a strategy that was inspired by the work of Frieden [16] in which a unifying principle of physics namely the extreme physical information (EPI) was developed. EPI can provide a mechanism to find the constraint terms and search for an exact solution for the unknown distribution of the measurement scenario. Consider the problem of estimating c. Any fluctuation yi − ci = xi will happen with a probability as Pi (yi /ci ) = Pi (xi ), xi = yi − ci .

(41)

Assuming ri = |y − ci |, then we have the Fisher information according to the EPI approach f [q] = −4

I

dri (dqi /dri )2 , Pi (ri ) = qi2 (ri ),

(42)

i=1

where qi is the i-th component probability amplitude for the fluctuation in the measurement. Then we have a bound information functional J[qi ] which obeys K[qi ] = I[qi ] − F [qi ] (43) The bound information functional is denoted as follows: F [qi ] = 4 dri fi (qi , ri ),

(44)


where

fi (qi , ri ) = qi2k (ri )Bi (ri ),

307

(45)

for some functions Bi (ri ). The proposed FGcM algorithm employs an objective function given by F f cm (U, c; Y ) =

I J i=1 j=1

μqij d2ij +

I J J I 1 1 μqij − γj ( μij − 1) = min . λ(q − 1) i=1 j=1 λ j=1 i=1

(46)

The second term of the above equation defines the Tsallis entropy when I i=1 μij = 1. Once minimised, Equation (46) leads to μij =

1 [1 + λ(q − 1)d2 (yj , c˜i )]−1/(q−1) , ∀j ∈ [1, J ], ∀ ∈ [1, I], q > 1, Zq

(47)

where Zq = Jj=1 [1+λ(q −1)d2(yj , ck )]−1/(q−1) . The prototype update equation can be formed as J q j=1 μij yj c˜i = J . (48) q j=1 μij When q → 1, FGcM has the same objective function and algorithm as that of FCM with regularization approach.

10 Discussion and Conclusions In this chapter we provided an overview of image clustering strategies based on fuzzy c-means (FCM). The conventional FCM approach is similar to the hard c-means algorithm (HCM) in that it arrives at a solution through iterative refinement of cluster prototypes, yet in contrast to HCM it also allows partial membership to clusters which in turn leads to an improved clustering performance. Many variants of FCM image clustering have been introduced in the literature and in this chapter we reviewed some of the more important ones that try to either speed up the clustering process or improve the quality of the resulting clusters. Multistage random sampling FCM starts with random sampling of a small subset of the full image. This step is intended to explore the cluster centres of the entire image. Ideally, this sub-sampling scheme should maximally reduce the number of iterations needed for convergence if the sampled subset has characteristics similar to those of the entire image. However, this requirement cannot be guaranteed especially in complex images. Consequently, the prior sub-sampling may result in an incomplete learning process, and in turn in slow clustering for the entire image. EnFCM or fast generalised FCM techniques add a term to the original clustering function so as to associate a pixel with its neighborhood. To consider the effect of the neighbors, a window needs to be defined beforehand.

308


Determination of the window size is image-dependent and hence this parameter may affect the final outcomes of clustering efficiency and accuracy. Comparatively, mean shift FCM leads to similar clustering outcomes but provides slightly faster clustering. The success of the new scheme is due to the fact that the used anisotropic kernel allows us to dynamically update the state parameters and achieve fast convolution by the anisotropic kernel function. RCFCM and S-FCM add a stage before a classical FCM starts in order to magnify the largest membership degree while suppressing the second largest membership degree. By doing this, the clustering convergence can be further accelerated. In SWFCM, spatial information is taken into account in the classical FCM algorithm. The weight used in the algorithm is determined using a k-nearest neighbour classifier. As a simple gray level histogram is utilised to compute the parameters it provides fast convergence. Lp norm FCM algorithm allows the dissimilarity variables to be changed and hence has better performance in the presence of outliers. Probabilistic FCM algorithms add a regulation term in the objective function, which dynamically updates the entire function during the optimisation of FCM. Using the concept of extreme physical information, FGcM incorporates a mechanism to find the constraint terms and search for an exact solution for the unknown distribution of the measured scenario.

References 1. Ahmed, M., Yamany, S., Mohamed, N., Farag, A., Moriaty, T.: A modified fuzzy c-means algorithm for bias field estimation and segmentation of MRI data. IEEE Trans. Medical Imaging 21, 193–199 (2002) 2. Auephanwiriyakul, S., Keller, J.M.: Analysis and efficient implementation of a linguistic fuzzy c-means. IEEE Trans. Fuzzy Systems, 563–581 (2002) 3. Bezdek, J.: A convergence theorem for the fuzzy isodata clustering algorithms. IEEE Trans. Pattern Analysis and Machine Intelligence 2, 1–8 (1980) 4. Cai, W., Chen, S., Zhang, D.: Fast and robust fuzzy c-means clustering algorithms incorporating local information for image segmentation. Pattern Recognition 40(3), 825–838 (2007) 5. Chen, S.C., Zhang, D.Q.: Robust image segmentation using FCM with spatial constraints based on new kernel-induced distance measure. IEEE Trans. Systems, Man and Cybernetics - Part B: Cybernetics 34, 1907–1916 (2004) 6. Cheng, H.D., Chen, J., Li, J.: Thresholding selection based on fuzzy c-partition entropy approach. Pattern Recognition 31, 857–870 (1998) 7. Cheng, T., Goldgof, D., Hall, L.: Fast fuzzy clustering. Fuzzy Sets and Systems 93, 49–56 (1998) 8. Chuang, K., Tzeng, S., Chen, H., Wu, J., Chen, T.: Fuzzy c-means clustering with spatial information for image segmentation. Computerized Medical Imaging and Graphics 30, 9–15 (2006) 9. Comaniciu, D., Meer, P.: Mean shift analysis and applications. In: 7th Int. Conference on Computer Vision, pp. 1197–1203 (1999)


309

10. Comaniciu, D., Meer, P.: Mean shift: a robust approach toward feature space analysis. IEEE Trans. Pattern Analysis and Machine Intelligence 24, 603–619 (2002) 11. Coppi, R., D’Urso, P.: Three-way fuzzy clustering models for LR fuzzy time trajectories. Computational Statistics & Data Analysis 43, 149–177 (2003) 12. Cover, T.M., Hart, P.E.: Nearest neighboring pattern classification. IEEE Trans. Information Theory 13, 21–27 (1967) 13. Dave, R.N., Krishnapuram, R.: Robust clustering methods: a united view. IEEE Trans. Fuzzy Systems 5, 270–293 (1997) 14. Eschrich, S., Ke, J., Hall, L., Goldgof, D.: Fast accurate fuzzy clustering through data reduction. IEEE Trans. Fuzzy Systems 11, 262–270 (2003) 15. Fan, J.-L., Zhen, W.Z., Xie, W.X.: Suppressed fuzzy c-means clustering algorithm. Pattern Recognition Letters 24, 1607–1612 (2003) 16. Frieden, B.: Physics from Fisher information, A Unification. Cambridge University Press, Cambridge (1999) 17. Gowda, K.C., Diday, E.: Symbolic clustering using a new dissimilarity measure. Pattern Recognition 24, 567–578 (1991) 18. Hathaway, R.J., Bezdek, J.C.: NERF c-means: Non-euclidean relational fuzzy clustering. Pattern Recognition 27, 429–437 (1994) 19. Hathaway, R.J., Bezdek, J.C., Hu, Y.: Generalised fuzzy c-means clustering strategies using lp norm distance. IEEE Trans. Fuzzy Systems 8, 576–582 (2000) 20. Hu, R., Hathaway, L.: On efficiency of optimization in fuzzy c-means. Neural, Parallel and Scientific Computation 10, 141–156 (2002) 21. Hung, W.-L., Yang, M.-S.: Similarity measures of intuitionistic fuzzy sets based on hausdorff distance. Pattern Recognition Letters 25(14), 1603–1611 (2004) 22. Kersten, P.R.: Implementing the fuzzy c-medians clustering algorithm. In: IEEE Conf. Fuzzy Syst., pp. 957–962 (1997) 23. Kolen, J., Hutcheson, T.: Reducing the time complexity of the fuzzy c-means algorithm. IEEE Trans. Fuzzy Systems 10(2), 263–267 (2002) 24. Krishnapuram, R., Keller, J.M.: A probabilistic approach to clustering. IEEE Trans. Fuzzy Systems 1, 98–110 (1993) 25. Leski, J.: Toward a robust fuzzy clustering. Fuzzy Sets and Systems 137, 215– 233 (2003) 26. Menard, M., Courboulay, V., Dardignac, P.-A.: Possibilistic and probabilistic fuzzy clustering: unification within the framework of the non-extensive thermostatistics. Pattern Recognition 36(6), 1325–1342 (2003) 27. Miyamoto, S., Agusta, Y.: An efficient algorithm for l1 fuzzy c-means and its termination. Contr. Cybern. 25, 421–436 (1995) 28. Pedrycz, W., Bezdek, J.C., Hathaway, R.J., Rogers, G.W.: Two nonparametric models for fusing heterogeneous fuzzy data. IEEE Trans. Fuzzy Systems 6, 411–425 (1998) 29. Sato, M., Sato, Y.: Fuzzy clustering model for fuzzy data. In: IEEE Int. Conference on Fuzzy Systems, pp. 2123–2128 (1995) 30. Szilagyi, L., Benyo, Z., Szilagyii, S.M., Adam, H.S.: MR brain image segmentation using an enhanced fuzzy c-means algorithm. In: 25th IEEE Int. Conference on Engineering in Medicine and Biology, vol. 1, pp. 724–726 (2003) 31. Takata, O., Miyamoto, S., Umayahara, K.: Clustering of data with uncertainties using hausdorff distance. In: 2nd IEEE Int. Conference on Intelligence Processing Systems, pp. 67–71 (1998)

310


32. Takata, O., Miyamoto, S., Umayahara, K.: Fuzzy clustering of data with uncertainties using minimum and maximum distances based on l1 metric. In: Joint 9th IFSA World Congress and 20th NAFIPS International Conference, pp. 2511–2516 (2001) 33. Wang, J., Thiesson, B., Xu, Y., Cohen, M.: Image and video segmentation by anisotropic kernel mean shift. In: 8th European Conference on Computer Vision, pp. 238–2492 (2004) 34. Wei, L.M., Xie, W.X.: Rival checked fuzzy c-means algorithm. Acta Electronica Sinica 28(7), 63–66 (2000) 35. Yang, M.-S., Ko, C.-H.: On a class of fuzzy c-numbers clustering procedures for fuzzy data. Fuzzy Sets and Systems 84(1), 49–60 (1996) 36. Yang, M.-S.: Fuzzy clustering procedures for conical fuzzy vector data. Fuzzy Sets and Systems 106(2), 189–200 (1999) 37. Yang, M.S., Hwang, P.Y., Chen, D.H.: Fuzzy clustering algorithms for mixed feature variables. Fuzzy Sets and Systems 141, 301–317 (2004) 38. Yong, Y., Chongxun, Z., Pan, L.: A novel fuzzy c-means clustering algorithm for image thresholding. Measurement Science Review 4, 11–19 (2004) 39. Zhao, M.S., Fu, A.M.N., Yan, H.: A technique of three level thresholding based on probability partition and fuzzy 3-partition. IEEE Trans. Fuzzy Systems 9, 469–479 (2001) 40. Zhou, H., Schaefer, G., Sadka, A., Celebi, M.E.: Anisotropic mean shift based fuzzy c-means segmentation of dermoscopy images. IEEE Journal of Selected Topics in Signal Processing 3(1), 26–34 (2009) 41. Zhou, H., Schaefer, G., Shi, C.: A mean shift based fuzzy c-means algorithm for image segmentation. In: 30th IEEE Int. Conference Engineering in Medicine and Biology, pp. 3091–3094 (2008)

Author Index

Banerjee, Soumya 275 Bouchachia, Abdelhamid

Ceberio, Martine 27, 133 Chrysostomou, Chrysostomos El-Dahshan, El-Sayed A. Gamez, J. Esteban

Jiang, Wenxin

275

53

Hassanien, Aboul Ella Hui, C. 175

Magoˇc, Tanja 133 Modave, Fran¸cois 27, 53, 133

237

275

259

Kosheleva, Olga 53 Kreinovich, Vladik 27, 53, 133

197

Nakamatsu, Kazumi 75 Nguyen, Hung T. 27, 53 Peters, James F. 3 Pitsillides, Andreas 197 Ra´s, Zbigniew W. Radi, Amr 275 Schaefer, Gerald

259

295

Wieczorkowska, Alicja Zeephongsekul, P. 111 Zhou, Huiyu 295

259

Foundations of Computational Intelligence Volume 2: Approximate Reasoning

Case-Based Approximate Reasoning

Artificial Intelligence: Foundations of Computational Agents

Foundations of Computational Intelligence.. Volume 6.. Data Mining

Foundations of Computational Intelligence Volume 3.. Global Optimization

Artificial Intelligence: Foundations of Computational Agents

Foundations Of Computational Intelligence Global Optimization

Handbook of Temporal Reasoning in Artificial Intelligence, Volume 1 (Foundations of Artificial Intelligence)

Computational Intelligence: A Compendium (Studies in Computational Intelligence, Volume 115)

Challenges for Computational Intelligence (Studies in Computational Intelligence, Volume 262)

Sequential approximate multi-objective optimization using computational intelligence

Foundations of Computational Linguistics

Computational Intelligence

Computational Intelligence

Foundations of artificial intelligence 1: Handbook of Temporal Reasoning in Artificial Intelligence

Successful Case-based Reasoning Applications (Studies in Computational Intelligence)

Advanced Computational Intelligence Paradigms in Healthcare-2

Foundations of Differential Geometry, Volume 2

Foundations of quantum mechanics, Volume 2

Foundations of Cryptography: Volume 2, Basic Applications

Modern Computational Intelligence Methods for the Interpretation of Medical Images (Studies in Computational Intelligence, Volume 84)

Foundations of Cryptography: Volume 2, Basic Applications

Innovations in Swarm Intelligence (Studies in Computational Intelligence, Volume 248)

Handbook of Numerical Analysis : Special Volume: Foundations of Computational Mathematics

Handbook of Numerical Analysis : Special Volume: Foundations of Computational Mathematics

Handbook of Numerical Analysis. Special Volume: Foundations of Computational Mathematics

Computational Intelligence Eine Einfuhrung

Foundations of Knowledge Representation and Reasoning

Handbook of temporal reasoning in artificial intelligence

Tractable Reasoning in Artificial Intelligence

Computational Intelligence in Archaeology

Foundations of Computational Intelligence Volume 2: Approximate Reasoning

Case-Based Approximate Reasoning

Artificial Intelligence: Foundations of Computational Agents

Foundations of Computational Intelligence.. Volume 6.. Data Mining

Foundations of Computational Intelligence Volume 3.. Global Optimization

Artificial Intelligence: Foundations of Computational Agents

Foundations Of Computational Intelligence Global Optimization

Handbook of Temporal Reasoning in Artificial Intelligence, Volume 1 (Foundations of Artificial Intelligence)

Computational Intelligence: A Compendium (Studies in Computational Intelligence, Volume 115)

Challenges for Computational Intelligence (Studies in Computational Intelligence, Volume 262)

Sequential approximate multi-objective optimization using computational intelligence

Foundations of Computational Linguistics

Computational Intelligence

Computational Intelligence

Foundations of artificial intelligence 1: Handbook of Temporal Reasoning in Artificial Intelligence

Successful Case-based Reasoning Applications (Studies in Computational Intelligence)

Advanced Computational Intelligence Paradigms in Healthcare-2

Foundations of Differential Geometry, Volume 2

Foundations of quantum mechanics, Volume 2

Foundations of Cryptography: Volume 2, Basic Applications

Modern Computational Intelligence Methods for the Interpretation of Medical Images (Studies in Computational Intelligence, Volume 84)

Foundations of Cryptography: Volume 2, Basic Applications

Innovations in Swarm Intelligence (Studies in Computational Intelligence, Volume 248)

Handbook of Numerical Analysis : Special Volume: Foundations of Computational Mathematics

Handbook of Numerical Analysis : Special Volume: Foundations of Computational Mathematics

Handbook of Numerical Analysis. Special Volume: Foundations of Computational Mathematics

Computational Intelligence Eine Einfuhrung

Foundations of Knowledge Representation and Reasoning

Handbook of temporal reasoning in artificial intelligence

Tractable Reasoning in Artificial Intelligence

Computational Intelligence in Archaeology

Recommend Documents