Machine Learning: ECML 2002: 13th European Conference on Machine Learning, Helsinki, Finland, August 19-23, 2002. Proceedings

Lecture Notes in Artificial Intelligence Subseries of Lecture Notes in Computer Science Edited by J. G. Carbonell and J...

Author: Tapio Elomaa | Heikki Mannila | Hannu Toivonen

45 downloads 862 Views 8MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Lecture Notes in Artificial Intelligence Subseries of Lecture Notes in Computer Science Edited by J. G. Carbonell and J. Siekmann

Lecture Notes in Computer Science Edited by G. Goos, J. Hartmanis, and J. van Leeuwen

2430

3

Berlin Heidelberg New York Barcelona Hong Kong London Milan Paris Tokyo

Tapio Elomaa Heikki Mannila Hannu Toivonen (Eds.)

Machine Learning: ECML 2002 13th European Conference on Machine Learning Helsinki, Finland, August 19-23, 2002 Proceedings

13

Series Editors Jaime G. Carbonell, Carnegie Mellon University, Pittsburgh, PA, USA Jörg Siekmann, University of Saarland, Saarbrücken, Germany Volume Editors Tapio Elomaa Heikki Mannila Hannu Toivonen University of Helsinki, Department of Computer Science P.O. Box 26, 00014 Helsinki, Finland E-mail: {elomaa, heikki.mannila, hannu.toivonen}@cs.helsinki.fi

Cataloging-in-Publication Data applied for Die Deutsche Bibliothek - CIP-Einheitsaufnahme Machine learning : proceedings / ECML 2002, 13th European Conference on Machine Learning, Helsinki, Finland, August 19 - 23, 2002. Tapio Elomaa ... (ed.). - Berlin ; Heidelberg ; New York ; Barcelona ; Hong Kong ; London ; Milan ; Paris ; Tokyo : Springer, 2002 (Lecture notes in computer science ; Vol. 2430 : Lecture notes in artificial intelligence) ISBN 3-540-44036-4

CR Subject Classification (1998): I.2, F.2.2, F.4.1

ISSN 0302-9743 ISBN 3-540-44036-4 Springer-Verlag Berlin Heidelberg New York This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer-Verlag. Violations are liable for prosecution under the German Copyright Law. Springer-Verlag Berlin Heidelberg New York, a member of BertelsmannSpringer Science+Business Media GmbH http://www.springer.de © Springer-Verlag Berlin Heidelberg 2002 Printed in Germany Typesetting: Camera-ready by author, data conversion by DA-TeX Gerd Blumenstein Printed on acid-free paper SPIN: 10873900 06/3142 543210

Preface

We are pleased to present the proceedings of the 13th European Conference on Machine Learning (LNAI 2430) and the 6th European Conference on Principles and Practice of Knowledge Discovery in Databases (LNAI 2431). These two conferences were colocated in Helsinki, Finland during August 19–23, 2002. ECML and PKDD were held together for the second year in a row, following the success of the colocation in Freiburg in 2001. Machine learning and knowledge discovery are two highly related fields and ECML/PKDD is a unique forum to foster their collaboration. The benefit of colocation to both the machine learning and data mining communities is most clearly displayed in the common workshop, tutorial, and invited speaker program. Altogether six workshops and six tutorials were organized on Monday and Tuesday. As invited speakers we had the pleasure to have Erkki Oja (Helsinki Univ. of Technology), Dan Roth (Univ. of Illinois, UrbanaChampaign), Bernhard Sch¨ olkopf (Max Planck Inst. for Biological Cybernetics, T¨ ubingen), and Padhraic Smyth (Univ. of California, Irvine). The main events ran from Tuesday until Friday, comprising 41 ECML technical papers and 39 PKDD papers. In total, 218 manuscripts were submitted to these two conferences: 95 to ECML, 70 to PKDD, and 53 as joint submissions. All papers were assigned at least three reviewers from our international program committees. Out of the 80 accepted papers 31 were first accepted conditionally; the revised manuscripts were accepted only after the conditions set by the reviewers had been met. Our special thanks go to the tutorial chairs Johannes F¨ urnkranz and Myra Spiliopoulou and the workshop chairs Hendrik Blockeel and Jean-Fran¸cois Boulicaut for putting together an exiting combined tutorial and workshop program. Also the challenge chair Petr Berka deserves our sincerest gratitude. All the members of both program committees are thanked for devoting their expertise to the continued success of ECML and PKDD. The organizing committee chaired by Helena Ahonen-Myka worked hard to make the conferences possible. A special mention has to be given to Oskari Heinonen for designing and maintaining the web pages and Ilkka Koskenniemi for maintaining CyberChair, which was developed by Richard van de Stadt. We thank Alfred Hofmann of Springer-Verlag for cooperation in publishing these proceedings. We gratefully acknowledge the financial support of the Academy of Finland and KDNet. We thank all the authors for contributing to what in our mind is a most interesting technical program for ECML and PKDD. We trust that the week in late August was most enjoyable for all members of both research communities. June 2002

Tapio Elomaa Heikki Mannila Hannu Toivonen

ECML/PKDD-2002 Organization

Executive Committee Program Chairs:

Tutorial Chairs:

Workshop Chairs:

Challenge Chair: Organizing Chair: Organizing Committee:

Tapio Elomaa (Univ. of Helsinki) Heikki Mannila (Helsinki Inst. for Information Technology and Helsinki Univ. of Technology) Hannu Toivonen (Nokia Research Center and Univ. of Helsinki) Johannes F¨ urnkranz (Austrian Research Inst. for Artificial Intelligence) Myra Spiliopoulou (Leipzig Graduate School of Management) Hendrik Blockeel (Katholieke Universiteit Leuven) Jean-Fran¸cois Boulicaut (INSA Lyon) Petr Berka (University of Economics, Prague) Helena Ahonen-Myka (Univ. of Helsinki) Oskari Heinonen, Ilkka Koskenniemi, Greger Lindén, Pirjo Moen, Matti Nyk¨ anen, Anna Pienim¨ aki, Ari Rantanen, Juho Rousu, Marko Salmenkivi (Univ. of Helsinki)

ECML Program Committee H. Blockeel, Belgium I. Bratko, Slovenia P. Brazdil, Portugal H. Boström, Sweden W. Burgard, Germany N. Cristianini, USA J. Cussens, UK L. De Raedt, Germany M. Dorigo, Belgium S. Dˇzeroski, Slovenia F. Esposito, Italy P. Flach, UK J. F¨ urnkranz, Austria J. Gama, Portugal J.-G. Ganascia, France T. Hofmann, USA L. Holmström, Finland

A. Hyv¨ arinen, Finland T. Joachims, USA Y. Kodratoff, France I. Kononenko, Slovenia S. Kramer, Germany M. Kubat, USA N. Lavraˇc, Slovenia C. X. Ling, Canada R. López de Màntaras, Spain D. Malerba, Italy S. Matwin, Canada R. Meir, Israel J. del R. Mill´ an, Switzerland K. Morik, Germany H. Motoda, Japan R. Nock, France E. Plaza, Spain

Organization

G. Paliouras, Greece J. Rousu, Finland L. Saitta, Italy T. Scheffer, Germany M. Sebag, France J. Shawe-Taylor, UK A. Siebes, The Netherlands D. Sleeman, UK M. van Someren, The Netherlands P. Stone, USA

H. Tirri, Finland P. Turney, Canada R. Vilalta, USA P. Vit´ anyi, The Netherlands S. Weiss, USA G. Widmer, Austria R. Wirth, Germany S. Wrobel, Germany Y. Yang, USA

PKDD Program Committee H. Ahonen-Myka, Finland E. Baralis, Italy J.-F. Boulicaut, France N. Cercone, Canada B. Crémilleux, France L. De Raedt, Germany L. Dehaspe, Belgium S. Dˇzeroski, Slovenia M. Ester, Canada R. Feldman, Israel P. Flach, UK E. Frank, New Zealand A. Freitas, Brazil J. F¨ urnkranz, Austria H.J. Hamilton, Canada J. Han, Canada R. Hilderman, Canada S.J. Hong, USA S. Kaski, Finland D. Keim, USA J.-U. Kietz, Switzerland R. King, UK M. Klemettinen, Finland W. Kl¨ osgen, Germany Y. Kodratoff, France J.N. Kok, The Netherlands S. Kramer, Germany S. Matwin, Canada

S. Morishita, Japan H. Motoda, Japan G. Nakhaeizadeh, Germany Z.W. Ra´s, USA J. Rauch, Czech Republic G. Ritschard, Switzerland M. Sebag, France F. Sebastiani, Italy M. Sebban, France B. Seeger, Germany A. Siebes, The Netherlands A. Skowron, Poland M. van Someren, The Netherlands M. Spiliopoulou, Germany N. Spyratos, France E. Suzuki, Japan A.-H. Tan, Singapore S. Tsumoto, Japan A. Unwin, Germany J. Wang, USA K. Wang, Canada L. Wehenkel, Belgium D. Wettschereck, Germany G. Widmer, Austria R. Wirth, Germany S. Wrobel, Germany M. Zaki, USA

VII

VIII

Organization

Additional Reviewers

N. Abe F. Aiolli Y. Altun S. de Amo A. Appice E. Armengol T.G. Ault J. Azé M.T. Basile A. Bonarini R. Bouckaert P. Brockhausen M. Brodie W. Buntine J. Carbonell M. Ceci S. Chikkanna-Naik S. Chiusano R. Cicchetti A. Clare M. Degemmis J. Demsar F. De Rosis N. Di Mauro G. Dorffner G. Dounias N. Durand P. Er¨ astö T. Erjavec J. Farrand S. Ferilli P. Floréen J. Franke T. Gaertner P. Gallinari P. Garza A. Giacometti

S. Haustein J. He K.G. Herbert J. Himberg J. Hipp S. Hoche J. Hosking E. H¨ ullermeier P. Juvan M. K¨ aäri¨ ainen D. Kalles V. Karkaletsis A. Karwath K. Kersting J. Kindermann R. Klinkenberg P. Koistinen C. K¨ opf R. Kosala W. Kosters M.-A. Krogel M. Kukar L. Lakhal G. Lebanon S.D. Lee F. Li J.T. Lindgren J. Liu Y. Liu M.-C. Ludl S. Mannor R. Meo N. Meuleau H. Mogg-Schneider R. Natarajan S. Nijssen G. Paaß

L. Pe˜ na Y. Peng J. Petrak V. Phan Luong K. Rajaraman T. Reinartz I. Renz C. Rigotti F. Rioult ˇ M. Robnik-Sikonja M. Roche B. Rosenfeld S. R¨ uping M. Salmenkivi A.K. Seewald H. Shan J. Sinkkonen J. Struyf R. Taouil J. Taylor L. Todorovski T. Urbancic K. Vasko H. Wang Y. Wang M. Wiering S. Wu M.M. Yin F. Zambetta ˇ B. Zenko J. Zhang S. Zhang T. Zhang M. Zlochin B. Zupan

Organization

Tutorials Text Mining and Internet Content Filtering José Mar´ıa G´ omez Hidalgo Formal Concept Analysis Gerd Stumme Web Usage Mining for E-business Applications Myra Spiliopoulou, Bamshad Mobasher, and Bettina Berendt Inductive Databases and Constraint-Based Mining Jean-Fran¸cois Boulicaut and Luc De Raedt An Introduction to Quality Assessment in Data Mining Michalis Vazirgiannis and M. Halkidi Privacy, Security, and Data Mining Chris Clifton

Workshops Integration and Collaboration Aspects of Data Mining, Decision Support and Meta-Learning Marko Bohanec, Dunja Mladenić, and Nada Lavraˇc Visual Data Mining Simeon J. Simoff, Monique Noirhomme-Fraiture, and Michael H. B¨ ohlen Semantic Web Mining Bettina Berendt, Andreas Hotho, and Gerd Stumme Mining Official Data Paula Brito and Donato Malerba Knowledge Discovery in Inductive Databases Mika Klemettinen, Rosa Meo, Fosca Giannotti, and Luc De Raedt Discovery Challenge Workshop Petr Berka, Jan Rauch, and Shusaku Tsumoto

IX

Table of Contents

Contributed Papers Convergent Gradient Ascent in General-Sum Games . . . . . . . . . . . . . . . . . . . . . . . . 1 Bikramjit Banerjee and Jing Peng Revising Engineering Models: Combining Computational Discovery with Knowledge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Stephen D. Bay, Daniel G. Shapiro, and Pat Langley Variational Extensions to EM and Multinomial PCA . . . . . . . . . . . . . . . . . . . . . . . 23 Wray Buntine Learning and Inference for Clause Identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Xavier Carreras, Llu´ıs M` arquez, Vasin Punyakanok, and Dan Roth An Empirical Study of Encoding Schemes and Search Strategies in Discovering Causal Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 Honghua Dai, Gang Li, and Yiqing Tu Variance Optimized Bagging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 Philip Derbeko, Ran El-Yaniv, and Ron Meir How to Make AdaBoost.M1 Work for Weak Base Classifiers by Changing Only One Line of the Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 G¨ unther Eibl and Karl Peter Pfeiffer Sparse Online Greedy Support Vector Regression . . . . . . . . . . . . . . . . . . . . . . . . . . 84 Yaakov Engel, Shie Mannor, and Ron Meir Pairwise Classification as an Ensemble Technique . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Johannes F¨ urnkranz RIONA: A Classifier Combining Rule Induction and k-NN Method with Automated Selection of Optimal Neighbourhood . . . . . . . . . . . . . . . . . . . . . 111 Grzegorz G´ ora and Arkadiusz Wojna Using Hard Classifiers to Estimate Conditional Class Probabilities . . . . . . . . 124 Ole Martin Halck Evidence that Incremental Delta-Bar-Delta Is an Attribute-Efficient Linear Learner . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 Harlan D. Harris Scaling Boosting by Margin-Based Inclusion of Features and Relations . . . . 148 Susanne Hoche and Stefan Wrobel Multiclass Alternating Decision Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 Geoffrey Holmes, Bernhard Pfahringer, Richard Kirkby, Eibe Frank, and Mark Hall

XII

Table of Contents

Possibilistic Induction in Decision-Tree Learning . . . . . . . . . . . . . . . . . . . . . . . . . . 173 Eyke H¨ ullermeier Improved Smoothing for Probabilistic Suffix Trees Seen as Variable Order Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 Christopher Kermorvant and Pierre Dupont Collaborative Learning of Term-Based Concepts for Automatic Query Expansion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 Stefan Klink, Armin Hust, Markus Junker, and Andreas Dengel Learning to Play a Highly Complex Game from Human Expert Games . . . . 207 Tony Kr˚ akenes and Ole Martin Halck Reliable Classifications with Machine Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 Matjaˇz Kukar and Igor Kononenko Robustness Analyses of Instance-Based Collaborative Recommendation . . . 232 Nicholas Kushmerick iBoost: Boosting Using an instance-Based Exponential Weighting Scheme . . . . . . . . . . . . . . . . . . . . . . . . 245 Stephen Kwek and Chau Nguyen Towards a Simple Clustering Criterion Based on Minimum Length Encoding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258 Marcus-Christopher Ludl and Gerhard Widmer Class Probability Estimation and Cost-Sensitive Classification Decisions . . 270 Dragos D. Margineantu On-Line Support Vector Machine Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282 Mario Martin Q-Cut – Dynamic Discovery of Sub-goals in Reinforcement Learning . . . . . . 295 Ishai Menache, Shie Mannor, and Nahum Shimkin A Multistrategy Approach to the Classification of Phases in Business Cycles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307 Katharina Morik and Stefan R¨ uping A Robust Boosting Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319 Richard Nock and Patrice Lefaucheur Case Exchange Strategies in Multiagent Learning . . . . . . . . . . . . . . . . . . . . . . . . . 331 Santiago Onta˜ n´ on and Enric Plaza Inductive Confidence Machines for Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . .345 Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman Macro-Operators in Multirelational Learning: A Search-Space Reduction Technique . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357 Lourdes Pe˜ na Castillo and Stefan Wrobel

Table of Contents

XIII

Propagation of Q-values in Tabular TD(λ). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .369 Philippe Preux Transductive Confidence Machines for Pattern Recognition . . . . . . . . . . . . . . . 381 Kostas Proedrou, Ilia Nouretdinov, Volodya Vovk, and Alex Gammerman Characterizing Markov Decision Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391 Bohdana Ratitch and Doina Precup Phase Transitions and Stochastic Local Search in k-Term DNF Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405 Ulrich R¨ uckert, Stefan Kramer, and Luc De Raedt Discriminative Clustering: Optimal Contingency Tables by Learning Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418 Janne Sinkkonen, Samuel Kaski, and Janne Nikkil¨ a Boosting Density Function Estimators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431 Franck Thollard, Marc Sebban, and Philippe Ezequel Ranking with Predictive Clustering Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444 Ljupˇco Todorovski, Hendrik Blockeel, and Saˇso Dˇzeroski Support Vector Machines for Polycategorical Classification . . . . . . . . . . . . . . . . 456 Ioannis Tsochantaridis and Thomas Hofmann Learning Classification with Both Labeled and Unlabeled Data . . . . . . . . . . . 468 Jean-Noël Vittaut, Massih-Reza Amini, and Patrick Gallinari An Information Geometric Perspective on Active Learning . . . . . . . . . . . . . . . . 480 Chen-Hsiang Yeang Stacking with an Extended Set of Meta-level Attributes and MLR . . . . . . . . 493 ˇ Bernard Zenko and Saˇso Dˇzeroski

Invited Papers Finding Hidden Factors Using Independent Component Analysis . . . . . . . . . . 505 Erkki Oja Reasoning with Classifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 506 Dan Roth A Kernel Approach for Learning from almost Orthogonal Patterns . . . . . . . . 511 Bernhard Sch¨ olkopf, Jason Weston, Eleazar Eskin, Christina Leslie, and William Stafford Noble Learning with Mixture Models: Concepts and Applications . . . . . . . . . . . . . . . .529 Padhraic Smyth Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .531

Convergent Gradient Ascent in General-Sum Games Bikramjit Banerjee and Jing Peng Electrical Engineering and Computer Science Department, Tulane University New Orleans, LA 70118, USA {banerjee,jp}@eecs.tulane.edu http://www.eecs.tulane.edu/Peng

Abstract. In this work we look at the recent results in policy gradient learning in a general-sum game scenario, in the form of two algorithms, IGA and WoLF-IGA. We address the drawbacks in convergence properties of these algorithms, and propose a more accurate version of WoLF-IGA that is guaranteed to converge to Nash Equilibrium policies in self-play (or against an IGA learner). We also present a control theoretic interpretation of variable learning rate which not only justifies WoLF-IGA, but also shows it to achieve fastest convergence under some constraints. Finally we derive optimal learning rates for fastest convergence in practical simulations.

1

Introduction

Game theory has been a driving impetus for modeling concurrent reinforcement learning problems. With booming e-commerce (notwithstanding the recent slump), the day is not far when automated buyers and sellers will control the electronic economy. There are other potential applications like disaster relief by robots(in potentially hazardous environments, especially after September 11), automated and robotic control of applications (viz. ranging from households to Mars exploration), etc where coordination among multiple agents will hold the key. This makes the focus on multiagent learning research extremely timely and justified. Several algorithms for multiagent learning have been proposed[5],[4],[3],[1], mostly guaranteed to converge to equilibrium in the limit. Bowling and Veloso note in [2], that none of these methods simultaneously satisfies rationality and convergence, two of the desirable criteria for any multiagent learning algorithm. A recent work [8] demonstrated that policy gradient ascent (which they called “Infinitesimal Gradient Ascent”or IGA) could achieve convergence to either Nash Equilibrium policies or Nash Equilibrium payoffs (when the policies don’t converge) in self-play. This algorithm was rational [2] but not convergent to Equilibrium policies in all general-sum games. Subsequently, it was modified with a variable learning rate [2] but the resulting algorithm (WoLF-IGA) might still not converge in some general-sum games. The rationale behind WoLF (Win or T. Elomaa et al. (Eds.): ECML, LNAI 2430, pp. 1–9, 2002. c Springer-Verlag Berlin Heidelberg 2002

2

Bikramjit Banerjee and Jing Peng

Learn Fast) was to allow the opponent to adapt to the learner’s policy by learning slowly (i.e. changing its policy slowly) when the learner is “winning”, but learn fast when it is not “winning”. However, they [2] used an approximate criterion to determine when the learner was “winning”, and consequently could fail to converge in some general-sum games. The motivation of the present work is to fill this gap to produce the first multiagent learning algorithm that is both rational and universally convergent. We then proceed to analyse why learning at a variable rate is essential for convergence in policy gradient learning, and show that WoLF-IGA is the best we can do under some constraints. Finally we address the question of how should we choose the learning rates so that we can achieve fastest convergence.

2

Definitions

Here we provide definitions of key concepts for our work. We refer to Ai as the set of possible actions available to the ith agent. Definition 1 A bimatrix game is given by a pair of matrices, (M1 , M2 ), (each of size |A1 | × |A2 | for a two-agent game) where the payoff of the ith agent for the joint action (a1 , a2 ) is given by the entry Mk (a1 , a2 ), ∀(a1 , a2 ) ∈ A1 × A2 , k = 1, 2. A constant-sum game (also called competitive games) is a special bimatrix game where M1 (a1 , a2 ) + M2 (a1 , a2 ) = c, ∀(a1 , a2 ) ∈ A1 × A2 , where c is a constant. If c = 0, then it is also called a zero-sum game. Definition 2 A mixed-strategy Nash Equilibrium for a bimatrix game (M1 , M2 ) is a pair of probability vectors (π1∗ , π2∗ ) such that π1∗T M1 π2∗ ≥ π1T M1 π2∗ ∀π1 ∈ P D(A1 ). π1∗T M2 π2∗ ≥ π1∗T M2 π2 ∀π2 ∈ P D(A2 ). where P D(Ai ) is the set of probability-distributions over the ith agent’s action space. No player in this game has any incentive for unilateral deviation from the Nash equilibrium strategy, given the other’s strategy. There always exists at least one such equilibrium profile for an arbitrary finite bimatrix game [6].

3

Policy Gradient Ascent in Bimatrix Games

The basic idea of such an algorithm is to iteratively update an agent’s strategy based on the consequent improvement in the agent’s expected payoff. When both the agent’s fail to improve their strategies any further (which may never happen), they must have converged to some Nash Equilibrium of the game [8]. The

Convergent Gradient Ascent in General-Sum Games

3

simplified domain for studying this problem is a two-agent, two action scenario, with the following payoff matrices r r c c R = 11 12 and C = 11 12 . r21 r22 c21 c22 Let α and β denote the probablities of the two agents selecting the first actions from their respective sets of available actions. Then, the expected payoffs of the two agents, (Vr for the row agent, and Vc for the column agent) are given by Vr (α, β) = r11 (αβ) + r22 ((1 − α)(1 − β)) + r12 (α(1 − β)) + r21 ((1 − α)β) Vc (α, β) = c11 (αβ) + c22 ((1 − α)(1 − β)) + c12 (α(1 − β)) + c21 ((1 − α)β) Then given a strategy pair (α, β) (constrained to lie in the unit square), and letting u = (r11 + r22 )− (r21 + r12 ) and u = (c11 + c22 )− (c12 + c21 ), the gradients are given by ∂Vr (αk , βk ) = βu − (r22 − r12 ) (1) ∂α ∂Vc (αk , βk ) = αu − (c22 − c21 ) (2) ∂β and the strategy pair can be updated as αk+1 = αk + η

∂Vr (αk , βk ) ∂α

βk+1 = βk + η

∂Vc (αk , βk ) ∂β

(3)

The new gradients generated by the above rules are constrained to lie in the unit square by taking their projections on the boundary whenever they cross out. For η → 0, the algorithm is called infinitesimal gradient ascent or IGA. It is known from game theory [7], that the algorithm may never converge in (α, β), but their expected payoffs have been proved to always converge to that of some Nash pair[8].

4

WoLF-IGA

Using equations 1,2,3 and η → 0, we get the unconstrained dynamics of the strategy pair given by the differential equations ∂α −(r22 − r12 ) 0 u α ∂t + . (4) = ∂β −(c22 − c21 ) β u 0 ∂t It has been proved [8] that the points of zero gradients (projected) are Nash Equilibria. However, the algorithm may not converge to such a point in case 0 u the matrix U = has imaginary eigenvalues and the center (point of zero u 0

4

Bikramjit Banerjee and Jing Peng

gradient) lies within the unit square. Consequently, the algorithm was modified with a variable learning rate[2] to converge to Nash pair even in this remaining subcase. The notion of variable learning rate changes the update rules in 3 to αk+1 = αk + ηlkr

∂Vr (αk , βk ) ∂α

βk+1 = βk + ηlkc

∂Vc (αk , βk ) ∂β

(5)

where lkr,c ∈ {lmin , lmax }. Since the proof of convergence of IGA for the other subcases depends only on the sign of the gradients, the above learning rules have the same convergence properties as long as lmin , lmax > 0. Moreover, for lmin < lmax , the algorithm can be made to converge to Nash pair in the remaining subcase[2], by choosing l when Vr (αt , βt ) ≥ Vr (αe , βt ) lr (t) = min lmax otherwise l when Vc (αt , βt ) ≥ Vc (αt , β e ) lc (t) = min (6) lmax otherwise where (αe , β e ) are some Nash pair. The unconstrained dynamics of the system now follows the differential equations r ∂α α −l (t)(r22 − r12 ) 0 lr (t)u ∂t + . (7) = c ∂β l (t)u 0 −lc(t)(c22 − c21 ) β ∂t The algorithm given by equations 5,6 is called the WoLF-IGA. When Vr (αt , βt ) ≥ Vr (αe , βt ), the row agent is said to be winning, in the sense that it would prefer its current strategy to its Nash strategy against the opponent’s current strategy. However, in order to find out whether an agent is winning, Bowling et.al.[2] used an approximate criterion for winning, defined by Vr (αt , βt ) ≥ Vr (αe , β e ), since Vr (αe , βt ) cannot be directly computed without knowing αe . As they note, this criterion fails to guarantee convergence in some general-sum games. In the next section we derive an alternative criterion, that can be easily computed and guarantees convergence.

5

Accurate Criterion for Winning

We are concerned with the subcase where U has purely imaginary eigenvalues and the center is within the unit square. The differential equation 4 has following solution for α(t) for the unconstrained dynamics[8] √ √ α(t) = B u cos( uu t + φ) + αe


5

where B and φ are dependent on the intitial α, β. This also describes the constrained motion of the row agent when they have come down to an ellipse fully contained within the unit square. We note that ∂ 2 α(t) = −|uu |(α − αe ) ∂t2 From [2], ∂Vr (αt , βt ) ∂α 1 ∂ 2 α(t) ∂Vr (αt , βt ) =− |uu| ∂t2 ∂α

Vr (αt , βt ) − Vr (αe , βt ) = (αt − αe )

2

∂Vr (αt ,βt ) < 0. Thus for the Hence we have Vr (αt , βt ) ≥ Vr (αe , β e ) when ∂ ∂tα(t) 2 ∂α 2 iterative updates, if ∆k = αk − αk−1 and ∆k = ∆k − ∆k−1 , then the row agent is winning if ∆k ∆2k < 0. This is not only easy to compute, but because this accurately estimates Vr (αt , βt ) − Vr (αe , βt ), the new method is now guaranteed to converge in all general-sum games. There is a similar criterion of winning for the column agent. We note that the same criterion can be extended to the system r in equation 7, since ∂l∂t(t) = 0 and lr (t)lc (t) = constant within each quadrant, as is evident from figure 1. Figure 2 shows more clearly, how this criterion works in this case. We demonstrate the working of our criterion on the bimatrix game R = 03 32 ,C = . This game is directly adopted from [2], where they used it 12 01 as a counterexample for their approximate criterion. The only Nash Equilibrium is (0.5, 0.5), and the starting point is (α, β) = (0.1, 0.9). Figure 3 shows our algorithm converging to the equilibrium while their’s failing. The experiment was run for 10, 000 iterations, and the accurate version converged in 1369 iterations. The choices for various parameters in both cases were thus: lmax = 1.0, lmin = 0.08, and precision1 required was = 0.01. We now turn to looking at learning with a variable rate from a control theory perspective.

6

Variable Learning Rate

Observe that when uu < 0, the equation 4 specifies a conserved energy oscillation system. For the constrained system, the motion may reach an edge of the unit square but sooner or later it will come down to an ellipse contained completely in the unit square. Since energy is conserved, convergence is improbable. The only way to force it to converge is to apply a force of attenuation proportional to its velocity, and in a direction opposing its velocity. For real world free oscillations e.g. in case of a swinging pendulum, friction with air provides such 1

(α,β) (α,β) i.e. if the stopping condition is | ∂Vr∂α | < and | ∂Vc∂β | 0 and lmax is finite, then H(t) is also given by the bottom curve in figure 1. This involves a short force of infinite magnitude exactly at the points where (α, β) cross quadrants in the phase plane. At all other points the force is zero. Given that we are constrained to have lr and lc constant within each quadrant (to maintain a piecewise elliptical trajectory for (α, β)), the attenuation within r any quadrant is necessarily zero, since dldt /lr = 0 inside any quadrant. This 2

This force was absent in the second order differential equation that could be created from equation 4


7

β >0

>0

2> 0

2< 0

α

decStrength[currentDec] then currentDec = v if decStrength[v] |Class(v)| |Class(currentDec)| D[k] = currentDec return D Ignoring the consistency checking in the function getClassif icationV ector(·, ·) we obtain the k nearest neighbours algorithm with selection of the optimal k

RIONA: A Classifier Combining Rule Induction and k-NN Method

117

(ONN ). An experimental comparison of RIONA and ONN is presented in the next section.

5

Experimental Study

Table 1 presents experimental results for 24 data sets from UCI repository [2]. For data that are split into a training and a testing set the experiments were performed for joined data. The accuracy for C5.0, DeEPs and DeEPsNN are taken from the paper [8]. The remaining algorithms were tested on a 800MHz PentiumIII PC, with 512M bytes of RAM. The algorithm RIA is time expensive so it was tested only for smaller data sets. The results were obtained by performing 10-fold cross-validation 10 times for each data set. All implemented algorithms: RIONA, RIA, ONN, 3-NN and RIONA(B) were tested with exactly the same folds and the significance of difference between algorithms was estimated using one-tailed paired t test.1 SVDM metric and the optimal neighbourhood were computed from a training set independently for each run in a cross-validation test. The total average accuracy was computed over all data sets except breast, bupa-liver and primary (for RIA it was computed only over the data sets that are given the accuracy). For all data sets the presented results were obtained for the metric described in Section 3.2 and N ormN Strength measure for conflict resolving (see Section 4). Although during the preliminary experiments we tried other types of a metric, no one appeared better then the presented one in terms of accuracy on a range of problem domains. We also tried to omit normalisation factor in the measure N ormN Strength what gave almost identical results. The optimal size of a neighbourhood was searched during the process of learning on the basis of the training examples. From the time complexity perspective it was important to limit searching for the optimal k to a small fixed range of possible values from 1 to kmax in such a way that sorting and consistency checking of kmax nearest neighbours were efficient. Since the values kmax optimal in this sense are the values close to the square root of the training set size (see Section 5.2) we set kmax = 200 (it is close to the square root of the size of the largest domains). In the next subsection we examine the significance of this setting. In Table 1 one can see that significant differences in accuracy between RIONA and ONN (k-NN with selection of the optimal neighbourhood) occurred mostly for smaller data sets (breast, bupa-liver, chess, primary, solar-flare and yeast ). The only difference between RIONA and ONN is the operation of consistency checking. In order to explain the similarity of results we checked what part of the k-neighbourhood for the optimal k is eliminated by the operation of consistency-checking and found that only for the domains breast, primary 1

The result of a single cross-validation test was the accuracy averaged over all 10 folds and the final average accuracy and the confidence level for difference between RIONA and the corresponding algorithm were computed from 10 repeats of the cross-validation test (for census-income and shuttle only 4 repeats).

118

Grzegorz G´ ora and Arkadiusz Wojna

Table 1. The average optimal k, the average accuracy (%) and the standard deviation for RIONA with the optimal k-best neighbourhood and the average accuracy (%) for the other systems: RIA, ONN, 3-NN, RIONA with the optimal B(tst, R) neighbourhood, C5.0, DeEPs and DeEPsNN. The superscripts denote the confidence levels: 5 is 99.9%, 4 is 99%, 3 is 97.5%, 2 is 95%, 1 is 90%, and 0 is below 90%. Plus indicates that the average accuracy of an algorithm is higher than in RIONA and minus otherwise Domain (size, attr, classes)

kopt

australian (690, 14, 2)

41,2

86,1±0,4 65,0−5 85,7−2 85,0−4

RIONA

RIA

breast (277, 9, 2)

77,9

73,4±1,0

73,00

breast-wis (683, 9, 2)

3,0

97,0±0,3 89,7−5 97,00

bupa-liver (345, 6, 2)

40,6 66,6±1,7 63,0−5 64,1−4

census (45222, 16, 2)

42,1

83,8±0,0

-

chess (3196, 36, 2)

11,9

98,0±0,1

-

73,90

−5

ONN

3-NN

RIONA(B)

C5.0

85,7−2

85,9

84,9

68,6−5

73,60

-

-

-

97,10

96,1−5

95,4

96,4

96,3

88,4

66,40

-

-

-

84,1+5 82,0−5

83,9+5

85,8

85,9

85,9

96,9−5 97,0−5 −1

66,00

DeEPs DeEPsNN

97,5−5

99,4

97,8

97,8

−5

73,1−4

71,3

74,4

74,4 68,0

german (1000, 20, 2)

29,2 74,5±0,5 70,1

glass (214, 9, 6)

2,1

70,7±1,9 39,5−5 70,70 71,9+1

63,9−5

70,0

58,5

heart (270, 13, 2)

19,4

83,2±1,0 62,8−5 83,10

81,3−5

83,40

77,1

81,1

81,1

iris (150, 4, 3)

37,1

94,6±0,6 90,5−5 94,40

95,3+4

94,70

94,0

96,0

96,0

letter (20000, 16, 26)

3,8

95,8±0,1

95,80

95,80

94,0−5

88,1

93,6

95,5

lymph (148, 18, 4)

1,4

85,4±1,3 76,4−5 86,3+1 84,4−2

81,4−5

74,9

75,4

mushroom (8124, 22, 2)

1,0 100,0±0,0

-

100,00 100,00

nursery (12960, 8, 5)

-

74,1

72,1

100,00 100,0 100,0

84,1 100,0

43,3 99,3±0,0

-

99,30 98,1−5

99,2−4

97,1

99,0

99,0

pendigits (10992, 16, 10) 1,2

99,4±0,0

-

99,40

99,40

97,4−5

96,7

98,2

98,8

pima (768, 8, 2)

34,3

74,7±0,9 65,2−5 74,40

72,2−5

72,7−5

73,0

76,8

73,2

primary (336, 15, 21)

75,9

31,7±0,8 32,4+1 40,3+5 33,5+4

31,60

-

-

-

satimage (6435, 36, 6)

3,7

91,3±0,1

91,30 91,4+2

87,7−5

86,7

88,5

90,8

segment (2310, 19, 7)

1,7

97,4±0,1 45,3−5 97,5+2 97,3−2

92,1−5

97,3

95,0

96,6

shuttle (58000, 9, 7)

1,3

99,9±0,0

99,90

99,8−5

99,6

97,0

99,7

solar-flare (1066, 10, 8)

70,9

81,2±0,3 81,4+1 82,7+5 78,1−5

81,7+5

82,7

83,5

83,5

splice (3186, 60, 3)

17,3

93,9±0,2

93,90

94,00

94,6+5

94,2

69,7

69,7

wine (178, 13, 3)

10,1 97,2±0,6 40,1−5 97,20

96,90

94,5−5

93,3

95,6

95,5

yeast (1484, 8, 10)

23,0 59,8±0,6 45,9−5 58,1−5 54,9−5

59,1−4

56,1

59,8

54,6

87,3

86,6

86,1

87,1

Total Average

88,7±0,4

-

64,3

99,90

88,7

87,8

and solar-flare the fraction of eliminated nearest neighbours was significant. For other domains the number of consistent objects from the optimal neighbourhood in RIONA algorithm is close to the number of all objects from the optimal neighbourhood of k-NN algorithm. Therefore the differences in classification accuracy are small. These observations suggest that the operation of consistency checking in RIONA is not very significant and it should be considered to be more restrictive. On the other hand, the accuracy of RIONA and ONN is comparable or better than well-known classifiers, in particular, their accuracy is generally better than the accuracy of RIA and 3-NN. It suggests the conclusion that RIONA and ONN may replace successfully both the rule-based algorithm using all minimal rules and a k-NN with a fixed k. It also proves that using a properly selected subset


119

of rules in rule-based systems gives better results than using all minimal rules. The range of tested data sets indicates that the presented algorithms work well for domains with both numerical and symbolic attributes. In particular, it works well for numerical attributes without preprocessing. 5.1

Further Study

In this section we describe more experiments and conclusions that can help us to understand important aspects of RIONA. First, we performed the experiments that helped us to compare two types of a neighbourhood: thr radial B(tst, R) and the k-best S(tst, k). For each data set we estimated the optimal value of the radius R and the optimal value of k from a training set and compared classification accuracy for both types of a neighbourhood. Looking at the third and the seventh columns in Table 1 one can see that the accuracy of the algorithm for the neighbourhood B(tst, R) is significantly worse than S(tst, k) on 14 domains (with the confidence level 4, -5) and significantly better on 3 domains (with the confidence level +4, +5). Therefore in further experiments we focused our attention on the neighbourhood S(tst, k). The setting kmax = 200 preserved the efficiency of RIONA but the interesting question was how significantly this setting influenced the classification results. Please note that the maximal possible value k is just the size of a training set. In order to answer this question the following experiment was performed: for the smaller sets (less than 4000 objects) the classification accuracy was measured for all possible values of k and for the greater sets the maximal value k was set to kmax = 500 (for the set nursery we made the exception kmax = 1000). The classification accuracy was measured for the leave-one-out method applied to the whole sets. Figures 1, 2 present the dependence of classification accuracy on the value of k for exemplary domains. For most data sets we observed that while increasing k beyond a certain small value the classification accuracy is falling down (see Figure 1). In particular, while comparing the third and the fourth column in Table 1, one can see that

Fig. 1. Accuracy for german

Fig. 2. Accuracy for census-income

120


for most data sets the results for the total neighbourhood are significantly worse than the results for the neighbourhood found by the algorithm RIONA. For the remaining data sets (breast, census-income, nursery, primary, solar-flare) the accuracy becomes stable beyond a certain value k (see Figure 2). For the former group we examined the neighbourhood size (the value of k) for which the maximum accuracy was obtained. In the latter case we examined both the value of k beyond which the accuracy remains stable and the fluctuations in accuracy while increasing k. For most domains the optimal value of k appeared to be much less than 200. On the other hand, for the domains where the optimal k was greater (australian, census-income and nursery) the loss in accuracy related to this setting was insignificant: it remained within the range of 0,15%. Moreover, the accuracy became stable for values of k also much lower than 200. Therefore we could conclude that the setting kmax = 200 preserved good time complexity properties and did not change the results significantly for tested data sets. For data sets split originally into a training and a testing set (splice, satimage, pendigits, letter, census-income, shuttle) we performed the experiments to compare the accuracy for two cases: when the value k was estimated either from a training set or from a test set (the optimal k). Experiments showed that for pendigits accuracy obtained by RIONA differs by about half percent from the accuracy with an optimal number k and for the other domains the difference remains in the range of 0.2%. It means that the used algorithm finds almost optimal number k in terms of obtained accuracy. Analogical experiments were done for the neighbourhood B(tst, R) and we observed that after the value R exceeded a constant Rmax (where Rmax was relatively small in comparison to the maximal possible value of R) the accuracy either became worse or did not improve significantly. This suggests the similar conclusion, i.e. the best accuracy is obtained for a small radius. 5.2

Time Complexity of RIONA

First, the learning algorithm performs two phases for each training object. In the first phase it selects kmax nearest objects among n = |trnSet| objects. On average it is done in the linear time. In the second phase the algorithm sorts all kmax 2 selected objects and checks consistency among them. It takes O(kmax ). Finally, for the whole training set the algorithm computes leave-one-out accuracy for each 1 ≤ k ≤ kmax , which takes O(nkmax ). Summing up, the average complexity of 2 the learning algorithm is O(n(n + kmax )). In practice the component O(n2 ) is dominant. Testing is analogical to learning. The classification algorithm finds kopt nearest examples and then checks consistency among them. Since kopt ≤ kmax , the 2 ) for a single test object and the total average comcomplexity is O(n + kmax 2 plexity of the testing algorithm is O(m(n + kmax )) where m is a number of test objects. In Table 2 one can see that for all the presented data sets the average time of classification for a single object is less than 0.6 s. Moreover, for larger data sets it is comparable with a single object test time in the algorithm ONN and is much shorter than a single test object time in the algorithm RIA.


121

Table 2. Single object test time (in seconds) for RIONA, RIA and ONN Domain

tRION A tRIA tON N Domain

tRION A tRIA tON N

australian

0,026

0,087 0,022 breast

0,016

0,021 0,014

breast-wis

0,032

0,063 0,017 bupa-liver

0,009

0,016 0,006

census

0,572

> 5, 0 0,568 chess

0,130

0,891 0,126

german

0,047

0,188 0,042 glass

0,010

0,012 0,006

heart

0,019

0,024 0,014 iris

0,003

0,006 0,003

letter

0,236

> 5, 0 0,224 lymph

0,017

0,019 0,014

mushroom

0,223

> 5, 0 0,219 nursery

0,169

> 5, 0 0,167

pendigits

0,133

> 5, 0 0,130 pima

0,013

0,055 0,010

primary-tumor

0,018

0,028 0,018 satimage

0,174

> 5, 0 0,169

segment

0,046

0,557 0,042 shuttle

0,378

> 5, 0 0,376

solar-flare

0,025

0,082 0,023 splice

0,405

3,194 0,393

wine

0,010

0,891 0,007 yeast

0,017

0,104 0,014

In case when the number of test objects is approximately equal to the number of training objects, taking into account both the learning and the classification phase, the average time complexity of RIONA is in practise O(n2 ), while the average time complexity of RIA is O(n3 ) what is quite a significant acceleration.

6

Conclusions and Future Research

The research reported in the paper attempts to bring together the features of rule induction and instance-based learning in a single algorithm. As the empirical results indicate the presented algorithm obtained the accuracy comparable to the well-known systems such as: 3-NN, C5.0, DeEPs and DeEPsNN. The experiments show that the choice of a metric is very important for classification accuracy of the algorithm. The combination of the normalised Manhattan metric for numerical attributes and SVDM metric for symbolic attributes proved to be very successful. It did not require discretisation for numerical attributes. We have compared two types of a neighbourhood: the k-nearest neighbours S(tst, k) and the ball B(tst, R). The former type of a neighbourhood gave generally better results, although the latter seemed more natural. This may suggest that the topology of the space induced by the used metric is rather complex. We found that the appropriate choice of the neighbourhood size is also an important factor for classification accuracy. It appeared that for all domain problems the optimal accuracy is obtained for a small neighbourhood (a small number of nearest neighbours k in S or a small radius R in B neighbourhood). This leads us to the conclusion that generally it is enough to consider only a small neighbourhood instead of the maximal neighbourhood related to the whole training set. This is interesting from the classification perspective, because it suggests that usually only a small number of training examples is relevant for accurate classification. It also illustrates the empirical fact that while using rule-based classifiers one can obtain better results by rejecting some rules instead of using

122


all minimal rules like the algorithm RIA does. We propose an approach to use only the rules that are built on the basis of a neighbourhood of the test case. The fact mentioned above is also the key idea that allowed us to make the original algorithm RIA efficient without loss in classification accuracy. In practice the complexity of learning and classification is only squarely and linearly dependent on the size of a learning sample respectively. Although a great effort was put into accelerating the algorithm, we think that further acceleration is possible, for instance by more specialised data structures and an approximate choice of nearest examples (see e.g. [10]). The facts that RIONA and ONN algorithms have similar classification accuracy and the fraction of objects eliminated by the consistency checking operation is very small indicate that this operation has rather small influence on the accuracy of the algorithm. It suggests that the k-NN component remains a dominant element of RIONA and shows that either the construction of local rules should be more general or the operation of consistency checking should be more restrictive. In RIONA the selection of the optimal value of k is performed globally. One possible extension of this approach is to apply a local method to searching for the appropriate value of k (see e.g. [12]). The interesting topic is the dependence of the average number of training examples on the distance to a test case. Empirically it was noticed that the dependence was close to linear, what seemed surprising to us.

Acknowledgements The authors are very grateful to professor Andrzej Skowron for his useful remarks on this presentation. This work was supported by the grants 8 T11C 009 19 and 8 T11C 025 19 from the Polish National Committee for Scientific Research.

References 1. Bazan, J. G. (1998). Discovery of decision rules by matching new objects against data tables. In: L. Polkowski, A. Skowron (eds.), Proceedings of the First International Conference on Rough Sets and Current Trends in Computing (RSCTC-98), pages 521-528, Warsaw, Poland. 112, 113, 114 2. Blake, C. L., Merz, C. J. (1998). UCI Repository of machine learning databases [http://www.ics.uci.edu/˜mlearn/MLRepository.html], Department of Information and Computer Science, Irvine, CA: University of California. 117 3. Cost, S. and Salzberg, S. (1993). A weighted nearest neighbor algorithm for learning with symbolic features. Machine Learning, 10, pages 57-78. 111 4. Domingos, P. (1996). Unifying instance-based and rule-based induction. Machine Learning, 24(2), pages 141-168. 111, 112, 113, 114 5. Duda, R. O. and Hart, P. E. (1973). Pattern classification and scene analysis. New York, NY: Wiley. 111 6. Golding, A. R., Rosenbloom, P. S. (1991). Improving rule-based systems through case-based reasoning. Proceedings of AAAI-91, pages 22-27, Anaheim, CA. 111


123

7. Li, J., Ramamohanarao, K. and Dong, G. (2001). Combining the strength of pattern frequency and distance for classification. The Fifth Pacific-Asia Conference On Knowledge Discovery and Data Mining, pages 455-466, Hong Kong. 111, 112 8. Li, J., Dong, G., Ramamohanaro, K. and Wong, L. (2001). DeEPs: A new instancebased discovery and classification system. [http://sdmc.krdl.org.sg:8080/˜limsoon/ limsoonpapers.html], School of Computing, National University of Singapore. 117 9. Michalski, R. S., Mozetic, I., Hong, J. and Lavrac, N. (1986) The Multi-Purpose Incremental Learning System AQ15 and its Testing to Three Medical Domains. Proceedings of AAAI-86, pages 1041-1045, San Mateo: Morgan Kaufmann. 111 10. Mitchell T. M. (1997). Machine learning. Portland: McGraw-Hill. 111, 122 11. Schaffer, C. (1994). A conservation law for generalisation performance. Proceedings of the Twelfth International Conference on Machine Learning, pages 259-265, New Brunswick, NJ: Morgan Kaufmann. 111 12. Wettschereck, D. (1994). A study of Distance-Based Machine Learning Algorithms. Doctor of Philosophy dissertation in Computer Science, Oregon State University. 113, 122

Using Hard Classifiers to Estimate Conditional Class Probabilities Ole Martin Halck Norwegian Defence Research Establishment (FFI) P.O. Box 25, NO-2027 Kjeller, Norway [email protected]

Abstract. In many classification problems, it is desirable to have estimates of conditional class probabilities rather than just h“ ard” class predictions. Many algorithms specifically designed for this purpose exist; here, we present a way in which hard classification algorithms may be applied to this problem without modification. The main idea is that by stochastically changing the class labels in the training data in a simple way, a classification algorithm may be used for estimating any contour of the conditional class probability function. The method has been tested on a toy problem and a problem with real-world data; both experiments yielded encouraging results.

1

Introduction

Classification is one of the most studied problems in machine learning research. In the simplest case, the task is to use training data to infer a classifier function c : X → {0,1}, where X is the instance or input space from which data points are taken. The usual measure of the quality of a classifier is the proportion of correct classifications it yields when given data not present in the training set. In the following, a classifier that outputs this kind of binary-valued predictions will be called a hard classifier. Hard classifiers are clearly most useful when there is reason to believe that there is a deterministic relationship between each data points’ position in input space and its class. In many problems this is not the case; this may be because all relevant information is not encoded in the input representation, or it may be due to real randomness in the problem domain. In such cases, it is desirable to have an indication of the certainty that a data point belongs to a given class. Learning machines that estimate actual probabilities of class membership of input data are particularly useful in this respect, for instance in problems where the costs of misclassification of positive and negative examples are different.. Soft two-class classification algorithms of this kind are essentially regression algorithms where a Bernoulli probability model is assumed. Many methods have been devised for the estimation of conditional class probability functions, ranging from classical logistic regression, via neural networks T. Elomaa et al. (Eds.): ECML, LNAI 2430, pp. 124-134, 2002.  Springer-Verlag Berlin Heidelberg 2002

Using Hard Classifiers to Estimate Conditional Class Probabilities

125

and Bayesian methods (e.g. [1]), to recent advances in the field of kernel methods and support vector machines (SVMs) (e.g. [2,3,4,5]). In the latter area, we shall return to the work of Platt [2], who proposed fitting a logistic function to the unthresholded output of an SVM classifier, and of Wahba [3,4], who modified the cost function of a kernel method to obtain probabilistic outputs. These previous approaches have in common that they require implementation (or modification) of methods specifically for conditional class probability estimation. In this paper, we present a novel approach where hard classifiers are used for addressing this task. In our method, only the training data need to be modified – in a simple way – while existing (hard) classification software can be used as-is. Our approach is motivated and presented in Section 2 and subjected to experimental evaluation in Section 3. Section 4 discusses some theoretical and practical issues relating to this method, while Section 5 concludes the paper.

2

Problem and Algorithm Formulation

In this section, we formulate the problem of estimating conditional class probabilities, and relate it to the pure classification problem. This serves as a motivation for our algorithm, which is subsequently described. The presentation is kept at a somewhat informal level; a more thorough examination of the simplifications and assumptions made is given in Section 4. 2.1 Conditional Class Probabilities and Hard Classifiers

In the problem we study here, we assume that each data point x is drawn from an input space X according to the probability distribution p(x) and assigned to class y = 1 with probability p( y = 1 x) and to class y = 0 otherwise. The task is to

estimate the true conditional class probability p( y = 1 x) as a function of x, based on training data {(xi , yi )}iL=1. What is the relation between hard classification and this task? The goal in hard classification, assuming equal misclassification costs, is usually to obtain a function c : X → {0,1} that minimizes the expected classification error on unseen data. For a given data point x, the expected error is minimized if the equivalence c(x) = 1 ⇔ p( y = 1 x) > 12 holds. Thus, the classification boundary of a hard classifier can be seen as an estimate of the surface in input space where p( y = 1 x) = 12 . 2.2 Motivation for Our Algorithm The main idea of our algorithm is motivated by the simple relationship described above. There, the boundary p( y = 1 x) = 12 is considered; clearly, if we can find the

126

Ole Martin Halck

boundaries p( y = 1 x) = q for other values of q as well, we are able to estimate p( y = 1 x) for a new example x, by checking its place in the instance space in relation to the probability contours for the various values of q. The idea of our algorithm is to s“ hift” these contours by modifying the training data, so that the surface at which p( y = 1 x) = q in the original data corresponds to the surface p ( y = 1 x) = p ( y = 1 x) =

1 2 1 2

for the new data set. A hard classifier estimating the boundary can then be trained on this data set, and thus give an estimate of the

contour p( y = 1 x) = q in the original data set. Specifically, consider the (unknown) surface in input space where p( y = 1 x) has a given constant value q < 12 . Loosely stated, we may then expect that in our training set, a larger proportion than q of the data points on one side of this surface belong to class 1, while this is true of less than a proportion q of the points on the other side.

a

b

p = 0.75

p = 0.25

p = 0.5

p = 0.75

p = 0.25

p = 0.5

Fig. 1. Illustration of the idea of the algorithm. Training data are shown by dots (positive examples) and crosses (negative examples). The grey lines show contours of the true probability function; the dashed lines illustrate how a classifier might draw its decision boundary. a) Original training data; the decision boundary is an estimate of the 0.5 contour. b) Negative examples have been flipped to positives with a probability of 1/3; the decision boundary now estimates the 0.25 contour

Now, assume that we create a new data set from the training set, where x is unchanged for each data point (x, y ), while the class y is set to 1 with probability s = ( 12 − q ) (1 − q ) regardless of its actual class, and otherwise left unchanged. In a region of input space where the original data set contains a proportion r of positive examples, we may now expect that a proportion of approximately r + (1 − r ) s = (r + 1 − 2q) 2(1 − q) of the modified data points are positive. This


quantity clearly grows monotonously with r, and is equal to

1 2

127

if and only if r = q.

Thus, the decision boundary of a hard classifier trained on this modified data set may serve as an estimate of the p( y = 1 x) = q contour surface in input space. Figure 1 illustrates this relationship in the case when q = 14 . For q > 12 , a similar operation, where the class labels are set to 0 with a suitable probability, may of course be performed. In this way, we may estimate any contour surface 0 < p ( y = 1 x) < 1, and by building a collection of estimates of this kind we are able to approximate p( y = 1 x) by a discrete function having the resolution we require. The following section describes this method in more detail, along with practical solutions to some problems that arise.

2.3 Description of the Algorithm Given a set D = {di }iL=1 of training data, where di = (xi , yi ), the first steps in using hard classifiers to form an estimate of the conditional class probability function p( y = 1 x) are as follows: 1.

Choose the coarseness of the estimating function. For simplicity, we divide the K 1 interval (0,1) into K equally-sized parts at the points {qk }k =1 , where qk = k K for k = 0,1,..., K . 2. For each k from 1 to K − 1, estimate the contour surface p( y = 1 x) = qk : • Make a new data set D ( k ) = {(x(i k ) , yi( k ) )}iL=1 from D as follows: For each i from 1 to L, set xi( k ) ← xi ; if qk < 12 , set yi( k ) ← 1 with probability ( 12 − qk ) (1 − qk ) and yi( k ) ← yi otherwise; if qk > 12 , set yi( k ) ← 0 with probability (qk − 12 ) qk and yi( k ) ← yi otherwise. • Train a hard classifier ck on the data set D ( k ) . Seen in isolation, the interpretation of a single classifier ck classifying a data point x as positive is that p( y = 1 x) is estimated to be larger than qk . A seemingly obvious way of obtaining a full estimate for p( y = 1 x) is therefore to find k such that ci (x) = 1 for i ≤ k and ci (x) = 0 for i > k , and use (qk + qk +1 ) 2 as our estimate. Unfortunately, due to the stochasticity of the algorithm, it may (and does) happen that a classifier ck classifies a point as negative, while another classifier ck ′> k does not. To address this problem, we use a simple remedy that reduces to the procedure above when the outputs of the classifiers are indeed consistent with each other. The final step in the estimation of the conditional class probability function is then: 3.

Estimate p( y = 1 x) by

128

Ole Martin Halck

1 pˆ HC ( y = 1 x) =  K This function clearly ranges from

1 2K

K −1



k =1



1

∑ ck (x)  + 2K .

to 1 − 21K in increments of

(1) 1 K

, and constitutes

the output of our algorithm.

2.4 Related Work The idea of using modified data sets in training is not in itself new. In the field of ensemble methods, machine-learning algorithms are run repeatedly on different data sets resampled from the original one, and the votes of the resulting classifiers are used as a basis for classification. The proportions of votes could also of course be regarded as estimates of class probabilities, but this probabilistic interpretation is not necessarily well-founded. These methods differ from our approach in that only the selection of chosen data points varies; the class labels themselves are not changed. To our knowledge, the only other algorithm that changes class labels of the training examples is MetaCost [6], which operates in a slightly different context, that of costsensitive classification. In MetaCost, an ensemble of classifiers is trained using resampled versions of the training set. The training examples are then relabelled with their optimal classifications, given the chosen misclassification costs and votes from the single classifiers. Any classification algorithm can then be trained on this modified set to provide cost-sensitive classification. Thus, another similarity between MetaCost and the algorithm presented here is that both are able to use any hard classification method as a b“ lack box”. We shall return to the link between class probability estimation and cost-sensitive classification in connection with the experiments described in Section 3.2.

3

Experiments

In this section, we present results from testing our algorithm on two problems. First, we apply it to a simple illustrative toy problem; the second test employs a data set from the UCI machine learning repository and compares the results achieved with results reported in previous research.

3.1 Toy Problem Experiment As an illustration of our approach, we defined a problem with input space

( − 12 , 12 ) × ( − 12 , 12 )

and class probability function

(

(

p( y = 1 x) = 1 + exp 9 x

2

))

−1

.

A contour plot of this function is shown in Figure 2a. A training set D of L = 250 data points x were drawn with uniform probability over X and labelled with y values in {0,1} according to the probability function p( y = 1 x) – see Figure 2b.


129

As a basis for comparison, we first estimated p( y = 1 x) by forming a distanceweighted average function using a simple radial basis function (RBF) algorithm, with a Gaussian kernel function K ( xi − x ) =

 x −x2  exp  − i 2    2πσ 2σ   1

(2)

around each data point (xi , yi ). This yields the estimate L

pˆ RBF ( y = 1 | x) = ∑ yi K ( xi − x ) i =1

L

∑ K ( xi − x ) .

(3)

i =1

Based on some initial experiments, we set σ = 0.15; Figure 2c shows the resulting contour plot of pˆ RBF ( y = 1 x). Next, we estimated the conditional class probability function using only hard classifiers in combination with the algorithm we have described. We used the classification algorithm that results from thresholding pˆ RBF ( y = 1 x) at 12 . When run on the original data set D, the result is clearly a hard classifier with a decision boundary following the 0.5 contour in Figure 2c. We partitioned the interval (0,1) into K = 50 parts, so that qk = k / 50 for k = 0,1,...,50, and ran the algorithm as described in Section 2.3. That is, for each qk , k = 1, 2,..., 49, we created a modified data set, trained a hard classifier on this

modified set, and used Equation (1) as an estimate pˆ HC ( y = 1 x) of the conditional class probabilities. Figure 2d shows a contour plot of the results. A comparison to Figure 2c shows that our algorithm yields similar results, apart from being less smooth – this latter property is not surprising, given the algorithm’s discrete and stochastic nature. A more quantitative assessment of the performance of the two algorithms can be gained from estimating the expected negative loglikelihood, according to the probability estimate, of a new data point drawn from p(x) and p( y = 1 x). This quantity was estimated using a uniform grid of 51× 51 points over the input space in the following way: − E  LL ( pˆ )  = −

1 2

51

∑ ∑ ∑ p ( y = c x = ( 50i , 50 ) ) lnˆ p ( y = c x = ( 50i , 50 ) ) . 25

25

1

j

j

i =−25 j =−25 c = 0

Table 1. Expected negative log-likelihood of a new data point according to each model

pˆ

− E  LL ( pˆ ) 

p

0.460

pˆ RBF

0.481

pˆ HC

0.482

(4)

130

Ole Martin Halck

Fig. 2. Experiments with the toy problem. a) Contours of the true conditional class probability function. b) Randomly generated training data; dots represent positive and crosses negative examples. c) RBF regression estimate of the probability function. d) Estimate using modified data sets and hard RBF classifiers

The results for pˆ RBF and pˆ HC , as well as for the true function p, are given in Table 1; this confirms that the two algorithms have similar performance. 3.2 Experiment on the UCI Adult Benchmark As a more challenging test for the algorithm, we used the Adult data set from the UCI machine learning repository [7]. Each data point x here consists of fourteen values (six numeric and eight categorical) taken from a census form for a household; the class label indicates whether the households’ yearly income was greater or smaller than USD 50,000. This data set, comprising 32,561 training examples and 16,281 test examples, was used in order to enable direct comparison to the work of Platt [2], in which he presents a method where the outputs of a support vector machine (SVM) are mapped to probabilities by fitting a logistic function. Platts’ paper compares the negative log-likelihoods of the test data (held out during training) given the conditional class probability estimates resulting from this approach and from a regularized-likelihood (RL) kernel method [3,4]. We evaluate our algorithm by


131

comparing its performance to these reported results, using the same hold-out set and criterion. For classification, we downloaded the decision tree classification program CRUISE version 1.09 [8] over the Internet. The algorithm used in CRUISE is described in [9]. The motivations for choosing this software package were that it was a) readily available in executable form for the Windows NT platform, and b) free. The point of requiring an executable classification program was that the experiments would, if successful, support our claim that software for hard classification may be used for conditional class probability estimation without modification. Again, we set K = 50, and for each k made a modified version of the training data set as described above. CRUISE was then run on each of these sets, where, for simplicity, the default values were used for all parameters except one. The exception was the number of cross-validation folds that CRUISE uses for pruning after building the full decision tree; this parameter was set to 2 rather than the default of 10 in order to reduce running time. The performance was evaluated by calculating the negative log-likelihood for the hold-out test data set T according to the resulting estimate of the probability function: − LL ( pˆ ) = −

∑

( xi , yi )∈T

ln ˆ p ( y = yi xi ).

(5)

Table 2 shows the results, averaged over five runs, alongside those reported in [2], and also gives the equivalent, but more intuitive, geometric mean of the predicted likelihoods of the test examples:  LL( pˆ )  L ( pˆ ) = exp =  ∏ ˆ p ( y = yi xi )   ( x , y )∈T  T  i i 

1 T

.

(6)

Table 2. Negative log-likelihoods and equivalent geometric means of predicted likelihood for three methods, using the test data in the UCI Adult data set

Algorithm SVM + logistic RL kernel method Hard classifiers

− LL ( pˆ )

L ( pˆ )

5323

0.721

5288

0.723

5394

0.718

In this experiment, our algorithm shows slightly poorer performance than the other two methods. However, those methods were specifically designed for this kind of problem, and also employed some preprocessing of the input data representation [2]. We simply applied a ready-made hard classification algorithm – with no data preprocessing and hardly any parameter tuning1 – to stochastically modified data sets. 1

The one exception – decreasing the number of cross-validation folds used by CRUISE for tree pruning – should, if anything, probably worsen the performance of the method.

132

Ole Martin Halck

Seen in this light, we consider our results to be remarkably close to those previously obtained. We have mentioned cost-sensitive classification as one context where estimates of class probabilities are useful. This relationship can also be seen the other way – a set of cost-sensitive classifiers can be used for estimating class probabilities. To see this, consider the case when the cost of classifying a negative example as positive is 1, while the cost of misclassifying positive examples is k. Classifying an example as positive then means that the expected cost of this choice is less than the expected cost of a negative classification, that is, p( y = 0 x) ⋅1 is less than p( y = 1 x) ⋅ k , or p( y = 1 x) > 1 (1 − k ). Thus, the contour for the probability level p( y = 1 x) = q can be estimated by running a cost-sensitive classifier (again as a black box), setting the cost of misclassifying positive examples to k = (1 − q) q. Since our chosen hard classifier supports cost-sensitive classification, we also compared our algorithm with the results achieved when estimating the same 49 probability levels by this method, using the same way of estimating the final probability estimates. The result for the cost-sensitive estimation was − LL( pˆ ) = 5716, which is clearly inferior to the results achieved by our algorithm.

4

Discussion

In the informal exposition in Section 2, we considered the decision boundary of a hard classifier as an estimate of the contour in input space where p( y = 1 x) = 12 . In fact, this view is the basis of the motivation for our algorithm. However, although it is clearly always optimal that a given single data point x is classified as positive if and only if p( y = 1 x) ≥ 12 , the assumption that a classifier will show this behaviour over the whole input space does not necessarily hold, even given unlimited training data. The reason for this lies in the inductive bias of any practical classification algorithm – a given algorithm is not able to output an arbitrary classification function c : X → {0,1}, but has a limited hypothesis space from which to choose its classifier. This means that the globally best classifier in this hypothesis space, according to classification accuracy, may not classify according to p everywhere. Platt [2] shows, for example, that the decision boundary of a support vector machine trained on the UCI data set used here does not estimate the p = 12 contour well. This means that although we do not need to do any modification to a classification algorithm in order to apply it the way we have described, it is important that we know something about the relationship between its classification properties and class probabilities. This point of view sheds some light on the good results obtained here using the CRUISE software – like most other classification algorithms based on decision trees, CRUISE assigns to each of its leaf nodes the class to which the majority of the nodes’ examples belongs. When we consider that the set of tests leading to a given leaf node in effect describes a region in input space, it becomes


133

clear that this way of assigning class labels makes the resulting decision boundary approximate the p = 12 contour. A natural objection to our approach is that it may seem unappealing to introduce more noise into an already noisy data set by modifying the training examples. Our answer to this is that the notion of n“ oise” is really not the correct one in this probabilistic setting. Unlike in many regression problems, where the observations of the function to be learnt consist of the true values with an added noise component, the data in our setting are simply the natural realisations of an underlying Bernoulli probability function. Thus, flipping the labels in a consistent way should be seen as merely shifting the probability levels of the function, rather than introducing more noise. These considerations also give a hint about which problems the algorithm is likely to solve well. Being based on shifting the probability levels in the instance space, it thus implicitly assumes that such a probability-based model is natural for the problem at hand. Consequently, the method should perform best if the problem is indeed truly probabilistic in nature. The value of our algorithm lies in the fact that it makes classification algorithms applicable to a new class of problems. This approach may thus be useful if a classification algorithm suited to the problem at hand is readily available, but a conditional probability estimation algorithm is not. If the hypothesis representation of the chosen classification algorithm is easily understandable to humans – as is the case for decision trees, for example – this method also has another advantage, namely that it yields descriptions of the regions of input space where the probability of the positive class is greater than a given level. The main practical disadvantage of our method is that it is somewhat expensive in terms of runtime – for each probability level, a new data set must be created and the classification algorithm run on this set. On the other hand, the user is free to choose an appropriate trade-off between runtime efficiency and the resolution of the estimating function, by selecting the number of probability levels to estimate. Another disadvantage is that it does not naturally generalize to multi-class problems. Of course, the process can be repeated for each class, but it is then unlikely that the estimated probabilities for the classes sum to one. In the special case where the classes have a natural order, however, the process can be run repeatedly by considering the upper n classes as positive examples, with n ranging from 1 to the number of classes minus one, each run yielding a probability function estimate pn. The probability levels for class m can then be estimated by considering the difference between the probability functions pm and pm–1 [10]. Even in this case, though, consistent estimates are not guaranteed, due to the stochasticity of the algorithm – pm – pm–1 may erroneously be negative in parts of the instance space.

5

Conclusion

We have described a method for obtaining probabilistic estimates of class membership in classification problems, using only hard classifiers. The algorithm works by generating a succession of data sets from the original one. In each of these sets, a proportion of the examples have their class labels flipped in a way that allows the decision boundary of a hard classifier, trained on the modified data, to be

134

Ole Martin Halck

interpreted as an estimate of a given probability contour. By collecting the resulting set of hard classifiers, we may form an estimate of the true conditional class probability function. The algorithm has been tested on a toy problem and a problem from the UCI data set repository, with encouraging results.

References 1.

MacKay, D. J. C.: The evidence framework applied to classification networks. Neural Computation 4 (1992) 720–736. 2. Platt, J. C.: Probabilities for SV machines. In: Smola, A.J., Bartlett, P., Schölkopf, B., Schuurmans, D. (eds.): Advances in Large Margin Classifiers, MIT Press (2000) 61–74. 3. Wahba, G.: Multivariate function and operator estimation, based on smoothing splines and reproducing kernels. In: Casdagli, M., Eubank, S. (eds.): Nonlinear Modeling and Forecasting, SFI Studies in the Sciences of Complexity, Proc. Vol. XII, Addison-Wesley (1992) 95–112. 4. Wahba, G.: Support vector machines, reproducing kernel Hilbert spaces and the randomized GACV. In: Schölkopf, B., Burges, C. J. C., Smola, A. J. (eds.): Advances in Kernel Methods – Support Vector Learning, MIT Press (1999) 69–88. 5. Tipping, M.: Sparse Bayesian learning and the relevance vector machine. Journal of Machine Learning Research 1 (2001) 211–244. 6. Domingos, P.: MetaCost: A general method for making classifiers cost-sensitive. In: Proceedings of the 5th International Conference on Knowledge Discovery and Data Mining (KDD-99), ACM Press (1999) 155–164. 7. Blake, C. L., Merz, C. J.: UCI Repository of machine learning databases. Irvine, CA: University of California, Department of Information and Computer Science (1998). URL: http://www.ics.uci.edu/~mlearn/MLRepository.html. 8. Loh, W.-Y.: CRUISE v 1.09 web page. URL: http://www.stat.wisc.edu/~loh/ cruise.html. 9. Kim, H., Loh, W.-Y.: Classification trees with unbiased multiway splits. Journal of the American Statistical Association 96 (2001) 589–604. 10. Frank, E., Hall, M.: A simple approach to ordinal classification. In: De Raedt, L., Flach, P. (eds.): Proceedings of the 12th European Conference on Machine Learning (ECML 2001). Lecture Notes in Artificial Intelligence vol. 2167, Springer (2001) 1451– 56.

Evidence that Incremental Delta-Bar-Delta Is an Attribute-Efficient Linear Learner Harlan D. Harris University of Illinois at Urbana-Champaign, Department of Computer Science MC-258, Urbana, IL 61801 USA [email protected]

Abstract. The Winnow class of on-line linear learning algorithms [10,11] was designed to be attribute-efficient. When learning with many irrelevant attributes, Winnow makes a number of errors that is only logarithmic in the number of total attributes, compared to the Perceptron algorithm, which makes a nearly linear number of errors. This paper presents data that argues that the Incremental Delta-Bar-Delta (IDBD) second-order gradient-descent algorithm [14] is attribute-efficient, performs similarly to Winnow on tasks with many irrelevant attributes, and also does better than Winnow on a task where Winnow does poorly. Preliminary analysis supports this empirical claim by showing that IDBD, like Winnow and other attribute-efficient algorithms, and unlike the Perceptron algorithm, has weights that can grow exponentially quickly. By virtue of its more flexible approach to weight updates, however, IDBD may be a more practically useful learning algorithm than Winnow.

1

Introduction

Linear learning algorithms make predictions by computing linear functions of their inputs. Since linear learners aren’t capable of representing or learning nonlinear concepts, practical use often requires either layering multiple linear learners and using a backpropagation algorithm, or generating an expanded feature space. Expanded feature spaces typically involve generating combinations of the original features, resulting in very large numbers of attributes irrelevant to the concept to be learned. The two algorithms most studied for use with this approach have been the Perceptron algorithm and Winnow [10,8,12,7]. Some learning domains such as computer vision and natural language processing naturally provide very large feature spaces with many irrelevant attributes, even without an expanded feature space. In this paper, I provide evidence that the Incremental Delta-Bar-Delta (IDBD) algorithm [14] combines the attribute-efficient properties of Winnow with additional robustness and flexibility, and is particularly useful for learning when many attributes may be irrelevant. In the on-line learning framework, the learner repeatedly performs a prediction task and receives a supervised training signal. Examples x are selected from an instance space, and are labeled by a concept c in a concept space. For each trial, the learner is given the example, predicts the example’s label, e.g. T. Elomaa et al. (Eds.): ECML, LNAI 2430, pp. 135–147, 2002. c Springer-Verlag Berlin Heidelberg 2002

136

Harlan D. Harris

p ∈ {−1, 1}, and then receives the true label, e.g. ∈ {−1, 1}, where = c(x). The goal of an on-line learner is to minimize the total number of mistakes ( = p) made in the prediction task while learning concept c. For a linear learner, the concept space is linear functions, as represented by a linear threshold unit, or Perceptron. Given an input vector x, of width n, a Perceptron computes the function p = sign(w·x), where w is the weight vector of the Perceptron. Since the hyperplane defined by the weight vector alone always includes the zero, Perceptrons frequently include a fixed or trainable bias weight. In order to learn non-linearly-separable functions, a common approach is to generate conjunctions of the inputs, either explicitly [12], or using kernel functions [2]. Although arbitrary DNF expressions can then be represented, linear functions being adequate to represent disjunctions of these conjunctions, several issues remain. There are an exponential number of conjunctions of the n input features, requiring exponential time and space to process. Use of kernel functions present other problems, particularly in domains where the input space is naturally large and mostly irrelevant [7]. In perhaps the most important problem for on-line learning, the number of examples needed to learn may increase linearly with the size of the expanded input space. An attribute-efficient algorithm is one in which the number of required examples increases only logarithmically in the number of irrelevant features. In an engineering setting, these expanded feature space approaches have been used most commonly in natural language processing (e.g., [12]), in which the presence or absence of particular words or word combinations leads to very large numbers of sparse, irrelevant features. Valiant [15] motivates this type of expanded-basis learning by reviewing biological systems, which are able to learn quickly in settings with very large numbers of interconnected neurons. For example, in the cerebellum, the neuronal architecture closely resembles the expanded feature space and linear learner paradigm [5,13].

2 2.1

Algorithms Perceptron Learning Rule

The Perceptron learning rule trains a linear threshold unit (Perceptron) by adding a fraction of each mis-predicted input vector to the weight vector. wi ← wi + η( − p)xi ,

(1)

where η ∈ (0, 1) is a learning rate parameter. The Perceptron rule performs incremental gradient descent in weight space. Variations include the Least-MeanSquare (LMS) rule (for non-thresholded linear units), and backpropagation (for multi-layer networks). 2.2

Winnow

The Winnow algorithms were introduced by Littlestone [10,11] as linear learners that separate (winnow) relevant from irrelevant attributes. Winnow was designed

Evidence that Incremental Delta-Bar-Delta

137

to efficiently learn monotone (non-negative) disjunctions and r-of-k threshold functions, and many of its proofs and applications have been in those domains. There are a number of variations of Winnow, optimized for different domains and with varying notation. The Winnow2 version of the algorithm, shown below, is typically initialized with all weights set to 1 or n1 . The threshold is often set to n, giving a ratio of threshold to initial weights of n : 1 or n2 : 1.  xi  if p = −1 ∧ = 1, wi α x i wi ← wi (1/α) (2) if p = 1 ∧ = −1,   otherwise, wi where xi ∈ {0, 1}. The Balanced Winnow variation learns non-monotone functions. Winnow works by forcing weights on irrelevant features towards zero exponentially quickly for false positives, and by raising weights on relevant features exponentially quickly for false negatives. The use of a multiplicative rather than additive update rule means that fewer changes are needed for convergence. The mistake bounds for the Winnow and Perceptron algorithms have been closely compared [8]. When learning monotone disjunctions of r literals, Winnow makes no more than O(r log n) mistakes, where n is the total number of attributes, and is thus attribute-efficient. In contrast, when learning the same class of functions, the Perceptron learning rule makes a nearly linear (in n) number of mistakes. If n is much larger than r, then Winnow performs much better, while if n is not much larger then r, the Perceptron algorithm does better, particularly if few features are active (non-zero) at any one time. 2.3

IDBD

Sutton’s IDBD (Incremental Delta-Bar-Delta) algorithm [14] is a variation on Jacobs’ Delta-Bar-Delta algorithm [6]. Delta-Bar-Delta, in turn, is a variation on the LMS learning rule (and also on backpropagation) that includes heuristics designed to speed up learning, most notably the inclusion of per-weight learning rates. The incremental version, IDBD, is additionally suitable for on-line learning and for learning non-stationary concepts. (Unlike DBD, IDBD is restricted to learning weights for linear units, and no backpropagation variant has apparently been derived. Note also that IDBD was originally derived to learn linear functions, with real-valued inputs and non-thresholded outputs, but I am using it here to learn weights for a linear-threshold unit and Boolean inputs.) As shown below, the particular way that IDBD accelerates learning seems to put it in the same category of attribute-efficient learning algorithms as Winnow. IDBD uses an additive weight update rule with a per-unit modifiable learning rate, rather than a shared, fixed learning rate as in the Perceptron rule. The algorithm can be seen to be performing gradient descent in learning-rate space, as well as in weight-space [14]. Intuitively, if a weight repeatedly changes in the same direction, then the learning rate is increased, since this input’s weight appears not to have converged (it is either too high or too low). If the weight

138

Harlan D. Harris

changes appear to be random, then the learning rate is decreased, since this input’s weight appears to have converged, and the weight is oscillating around the correct value. The update rules are as follows: βi = βi + θhi xi ( − p) βi

(3)

ηi = e wi = wi + ηi xi ( − p)

(4) (5)

hi = hi [1 − ηi x2i ]+ + ηi xi ( − p)

(6)

Note the per-input learning rate, ηi , in equation 5. Also, observe that β is updated using the Perceptron rule, using θ as a learning rate, with the addition of hi , which represents the recent summed history of the weight’s changes. The [·]+ notation is equivalent to max(·, 0). An interesting aspect of IDBD is that its update rules are defined recursively; β depends on h, which depends on β via η. Sutton [14] gives experimental results showing that IDBD has lower cumulative error on a simple non-stationary tracking task (where the concept periodically changes) than does the LMS rule, is able to distinguish relevant from irrelevant inputs, and weights those inputs approximately optimally. However, he does not address the issue of attribute-efficiency, and no further theoretical or empirical examination of IDBD has been performed to my knowledge.

3

Experiments

In this section I present new experimental evidence that compares IDBD to Winnow and the Perceptron algorithm. Experiment 1 tests these algorithms on the l-of-m-of-n threshold function, and shows that IDBD usually performs comparably to Winnow, and much better than the Perceptron learning rule, when learning disjunctive concepts with many irrelevant attributes. Experiment 2 systematically varies the complexity parameter n, with results suggesting that IDBD’s mistake bounds grow logarithmically, like Winnow, and unlike the Perceptron algorithm. Experiment 2b uses the same domain to illustrate IDBD’s low need for parameter tuning. Experiment 3 then looks at learning of random linear functions with few irrelevant attributes, and shows that IDBD performs well even in circumstances when Winnow does poorly, which reinforces the idea that IDBD is not merely a reimplementation of Winnow. (In an earlier experiment, IDBD was shown to be useful for learning complex DNF concepts with incomplete expanded feature spaces [5].) For all results reported below, 10 replications were performed with different random number seeds used to generate the data, and the results were averaged. Initial weights for all three algorithms were set to n1 , and the thresholds were n2 .


139

1000 Perceptron Winnow IDBD

Cumulative Errors

800

600

400

200

0

1

10

25 50 75 Number of Required Active Attributes (l)

100

Fig. 1. Cumulative errors after 10,000 examples in an l-of-100-of-1000 learning task. Mean of ten replications, with 95% confidence intervals. Winnow performs best when l = 1 (disjunction) or l = 100 (conjunction), but IDBD performs similarly or better for intermediate values of l

3.1

Experiment 1: Comparing IDBD to Winnow with Irrelevant Attributes

This experiment was designed to compare IDBD to Winnow and the Perceptron algorithm on a task covered by the theoretical results of Kivinen et al. [8]. When learning a concept where the Perceptron rule should make a nearly linear (in irrelevant attributes) number of errors, and Winnow should make a logarithmic (in irrelevant attributes) number of errors, how does IDBD empirically perform? To test this, the algorithms were compared on l-of-m-of-n Boolean concepts. These concepts are defined so that if l or more of the first m attributes are set to 1, the example is positive, and if fewer than l of the first m attributes are set to 1, the example is negative. Data was generated as follows. Each input vector was of width n. The first m attributes were considered “relevant,” while the remaining n − m attributes were “irrelevant,” and were set to 1 with a probability of 0.25. The examples were half positive and half negative. For positive examples, a number r in the interval [l, m] was chosen, while for negative examples, r was in the interval [0, l − 1]. Then, r attributes in the first m were randomly selected and were set to 1, with the other relevant attributes set to 0. 10,000 examples were generated and presented to the algorithms. Based on pilot experiments, the learning rates of the Perceptron rule (η) and IDBD (θ) were set to 0.1, and the learning rate of Winnow (α) was set to 1.1.

140

Harlan D. Harris

Table 1. Error rate in final 1000 examples of an l-of-100-of-1000 learning task. Mean of ten replications Perceptron Winnow IDBD

1 6.8 1.6 4.1

10 5.0 1.0 2.7

25 3.8 1.1 2.1

50 3.0 1.4 1.7

75 2.9 2.6 2.1

100 4.8 0.0 3.2

Moderate changes to these rates do not qualitatively change the results (see also Experiment 2b). Each algorithm was run with various values of l, with m = 100 and n = 1000. The number of errors after presentation of 10,000 examples is shown in Figure 1. The error rate for the last 1000 examples is shown in Table 3.1. IDBD performed uniformly much better than Perceptron, with fewer cumulative errors and lower final error rates. Compared to Winnow, IDBD usually made a similar number of cumulative errors, but often with a somewhat higher residual error rate. 3.2

Experiment 2: Showing Attribute-Efficient Learning by IDBD

The theoretical results predict that as n grows, the Perceptron algorithm should make errors that increase nearly linearly, while Winnow’s mistakes should increase only logarithmically. If IDBD is attribute-efficient, its results should be similar to Winnow’s. To test this, l and m were set to 10 and 100, respectively, and n was varied between 100 and 1000. Instead of running for a fixed number of examples, each trial was terminated when 200 examples in a row were classified correctly (i.e., the concept had been effectively learned). The learning rates for the Perceptron rule (η), IDBD (θ), and Winnow (α) were set to 0.8, 0.1, and 1.4, respectively (see below). This test shows precisely how well each algorithm scales up with irrelevant attributes, and should illustrate the theoretical predictions discussed above. The results are shown in Figure 2. The Perceptron algorithm, as expected, did much more poorly than Winnow, and its mistake measures increased sharply with n. Winnow made only slightly more errors as n increased, and the curve appears logarithmic. Most interestingly, IDBD shares Winnow’s property of making few additional errors as n increases. Like Winnow, IDBD’s attribute-efficiency curve seems logarithmic, and is qualitatively different from the Perceptron algorithm. These results strongly suggest that IDBD shares the irrelevant-attribute-efficiency that Winnow is know for.


141

750

Cumulative Errors

Perceptron Winnow IDBD 500

250

0

200

400 600 800 Total Attributes (100 Relevant)

1000

Fig. 2. Cumulative errors at convergence (200 correct predictions in a row) for a 10-of-100-of-n learning task. Mean of ten replications, with 95% confidence intervals. IDBD and Winnow show attribute efficiency, unlike the Perceptron algorithm

3.3

Experiment 2b: Showing IDBD Is Relatively Insensitive to Its Parameters

To more carefully explore the strengths and weaknesses of these algorithms, and to replicate one of Sutton’s conclusions about IDBD [14], the same domain was used for an exploration of the learning rate parameter space. As before, the target concepts are 10-of-100-of-n threshold functions. The learning rates tested were 0.001, 0.01, 0.05, 0.1, 0.2, 0.4, and 0.8. (For Winnow, α was set to 1 plus the above numbers.) The variable n took the values 100, 500, and 1000. As before, 10 replications were performed, and the cumulative errors before convergence (200 correct predictions in a row) were counted. The results are shown in Table 3.3. Clearly, the IDBD algorithm was relatively insensitive to small values of the learning rate, as compared with Winnow and the Perceptron algorithm. However, it was susceptible to convergence failure when the learning rate was very high. Again, note the sharp increase in errors between n = 500 and n = 1000 for the Perceptron algorithm, compared to the small increases for Winnow and IDBD.

142

3.4

Harlan D. Harris

Experiment 3: Showing IDBD Does Well when Winnow Does Poorly

The same theoretical results that predict that Winnow should learn with fewer mistakes when irrelevant attributes are predominant also predict that the Perceptron learning rule should learn with fewer mistakes when all (or nearly all) attributes are relevant. In addition, since Winnow is optimized for disjunctions, it’s reasonable to expect that it will do relatively poorly when learning arbitrary weights. By virtue of its fixed, multiplicative weight update rule, Winnow may find it difficult to set weights with the precision needed for arbitrary, smallmargin concepts. Weights can oscillate around a target value, their step size too large to approach it. To test this intuition, the three algorithms were compared on randomly-generated linear threshold functions. These functions are of the ˜ · x, where the elements of w ˜ were randomly generated, indepenform c(x) = w dent real-valued numbers. The data was generated as follows. For each concept, target weights were selected from a uniform distribution over the interval [0, 10]. Then, 20,000 examples of each concept were generated by randomly setting Boolean inputs with probability 0.5, multiplying by the target concept’s weights, and comparing the result to a threshold equal to the expected value, 10 ∗ n4 . Approximately half of the resulting examples were thus positive, and half were negative. Note that weights near 10 are maximally relevant, while those near zero are essentially irrelevant. Unlike the previous experiments, there is a continuum of relevance, with nearly all weights being somewhat relevant to the target concept. In this experiment, we used learning rates of 0.01 for the Perceptron rule, 0.2 for IDBD, and 1.01 for Winnow, based on informal pilot experiments. The results, for n = 200 (other values of n were similar), can be seen in Figure 3. As expected, the Perceptron algorithm performed better than Winnow.

Table 2. Cumulative errors to convergence on a 10-of-100-of-n task, varying learning rates. Bold represent the parameter that results in the lowest error count at n = 1000. NC means that the algorithm did not always converge even after 100,000 examples n 100 500 1000 Winnow 100 500 1000 IDBD 100 500 1000 Alg. Percep.

.001 11645 12495 14608 15145 20442 22762 380 1185 1374

.01 1671 2889 4800 1667 2126 2390 365 910 1097

.05 382 1215 1415 358 445 502 289 421 553

.1 216 823 1018 190 240 261 223 358 352

.2 115 434 635 115 153 185 171 211 NC

.4 63 413 562 107 132 164 118 NC NC

.8 37 330 550 192 278 480 180 NC NC


143

The IDBD algorithm showed performance similar to, and slightly but significantly better than, that of the Perceptron learning rule for this domain.

4

Discussion

These empirical results suggest that IDBD is an attribute-efficient learner, with a logarithmic attribute-efficiency curve. IDBD makes mistakes at rates only somewhat higher than Winnow when irrelevant attributes are plentiful, but significantly lower than Winnow when all attributes are relevant. By being both attribute-efficient and flexible, IDBD should be particularly useful when the number of irrelevant attributes is unknown prior to learning. For example, robotic multi-modal sensory systems, computational linguistic applications [12], and neurological modeling applications [5,15] often naturally have extensive irrelevant attributes and expanded feature spaces. Other linear learning systems with attribute-efficient properties have been mentioned in the literature. The p-norm family of Generalized Perceptron1 algorithms [4] is able to approximate the Perceptron and Winnow algorithms by choices of the p parameter. For a sufficiently large p, the p-norm algorithm is known to be attribute-efficient. The ALMAp algorithm [3] combines a p-normlike computation with a decaying learning rate, a specifiable margin, and nor1

The Generalized Perceptron defines Quasi-Additive algorithms defined by a vector z, updated exactly like the Perceptron, and a function f (z) = w specifying the weight vector as a function of z. The Perceptron, Balanced Winnow, and p-norm algorithms may be defined by f (z) = z, sinh(z), and sign(z)p|z|p−1 , respectively.

3000 Perceptron IDBD Winnow

Cumulative Errors

2500 2000 1500 1000 500 0

0

5000

10000 Examples

15000

20000

Fig. 3. Cumulative errors on 200-attribute random linear threshold concepts. Average of ten replications, with 95% confidence intervals

144

Harlan D. Harris

malized weight vectors. Although it has not been shown to be attribute-efficient, it seems likely to be so. IDBD has similarities with these two algorithms, but differences and advantages as well. Like p-norm and Winnow, and unlike ALMAp , IDBD’s weights are unbounded, and by having increasing learning rates2 , can grow exponentially fast. Like ALMAp , but unlike p-norm and Winnow, IDBD uses adjustable learning rates which decrease as the learner converges. However, IDBD has per-weight learning rates, while ALMAp has only a single dynamic learning rate. A practical advantage in real-world learning situations is that IDBD has only a single parameter, the learning rate θ, compared to p-norm with two and ALMAp with three. (IDBD has more variables requiring initial conditions, however.) IDBD is relatively insensitive to the settings of its learning rate parameter [14], allowing IDBD to be used with more confidence given less knowledge of the target concept than other algorithms. A final advantage is that IDBD was designed to be capable of learning drifting concepts, while other attribute-efficient learners may deal poorly with concept change. (But see [1] for a variation on Winnow which does learn drifting concepts well.) 4.1

Analytical Support

Several analytical approaches can be used to complement the empirically-supported assertion that IDBD is attribute-efficient. In this section, I investigate the relative speed of weight changes in various algorithms, and argue that exponential weight increases are sufficient for attribute-efficiency. Then, I show that a variation of IDBD is very similar to the Generalized Perceptron formulation of an algorithm that is known to be attribute-efficient. IDBD’s attribute-efficiency seems to be due to the algorithm’s ability to increase weights exponentially fast on relevant attributes, relative to the other weights. With exponentially fast increases, relevant weights outweigh irrelevant weights quickly enough that a number of errors only logarithmic in the number of irrelevant weights need be made [10]. Consider a simple case of the weight on a relevant Boolean input, where the learner makes repeated false negative predictions. That is, for simplicity, let x = 1 and − p = 1 (actually = 2, but we can use 1 by doubling the learning rate). How does w change over time? n For the Perceptron algorithm in this case, w ← w + η, and w(n) = i=1 η = O(n). The Perceptron algorithm only increases weights linearly quickly, and it is not attribute-efficient. For Winnow under these assumptions, w ← αw, and w(n) = w(0)αn = O(αn ). Clearly, Winnow increases weights exponentially quickly, and it is attribute-efficient. Before examining IDBD, let’s first examine a simple second-order gradientdescent relative, in which η = β and h = 1. Then, (using T instead of θ so as not to be confused with the θ(·) of asymptotic notation) we have β ← β + T 2

Winnow can be rewritten as an additive learning rule with a per-weight learning rate that is equal to a function of the weight itself.


145

Table 3. IDBD-h algorithm prediction label update 1 -1 if xi = 1, wi ← wi − ezi , demotion zi ← zi − η promotion -1 1 if xi = 1, zi ← zi + η wi ← wi + ezi ,

n n and ← w + β. Therefore, β(n) = i=1 T = O(n), and w(n) = i=1 β(n) = n w n 2 i=1 j=1 T = O(n ). This algorithm does not grow weights exponentially, and thus could not be expected to show the logarithmic attribute-efficient behavior of Winnow3 . Adding η = eβ back to IDBD, but keeping h = 1, we now have that w ← w + eβ , and w(n) = ni=1 eO(n) = O(en ). This algorithm, without the decaying sum of recent weight changes, can increase weights exponentially, and thus should be attribute-efficient. It is rather similar to Winnow, in that the effective learning rate is related to all previous weight changes. Re-adding the h of equation 6 to complete IDBD results in an attributeefficient algorithm that modulates the exponential in the exponentially-increasing learning rate by the extent to which the recent weight changes have been in the same direction. The result is more flexible than Winnow, yet still appears attribute-efficient. With some modifications, IDBD can be shown to be similar to the Weighted Majority algorithm [11,9]. To see this, first modify IDBD by separating the promotion (false negative) and demotion (false positive) cases, as in the original presentation of Winnow [10]. Then, let hi = 1 as above, and assume that xi ∈ {0, 1}. We then have the algorithm shown in Table 3, which I’ll call IDBD-h. Note that by reversing the order of wi and zi updates in demotions, that a sequence of promotions and demotions can be re-ordered arbitrarily and will result in the same final weight vectors. Assuming the initial values of z and w are zero, the weight vector w can be zi /η iη η written as a function of the current value of z: wi = i=1 e = eηe−1 (ezi − 1). Note that the ratio is a constant, and thus can be shifted to the bias weight, if their is one, or ignored otherwise. We therefore have a Generalized Perceptron with f (z) = ez − 1. This is very similar to the Weighted Majority algorithm [9], which can be defined as a G.P. with f (z) = ez (plus simple transformations of the input representation and learning rate)[4]. Clearly, the IDBD-h algorithm should have mistake bounds that are similar to those of the attribute-efficient (see [11]) Weighted Majority. Unfortunately, direct analysis of IDBD-h has so far failed, as the methods for finding mistake bounds for Generalized Perceptron can’t be easily applied to the f (z) above, and other approaches have yet to be successful. 3

√ It’s interesting to consider whether this algorithm might have O( n) mistake bounds. In fact, work in progress shows that a closely related algorithm does.

146

Harlan D. Harris

However, this analysis on a simplified version of IDBD further provides evidence for the notion of IDBD’s attribute-efficiency. 4.2

Conclusions

This paper has provided an empirical basis for believing that the IDBD algorithm has attribute-efficient properties. In addition, it was shown that IDBD has strengths relative to more traditional attribute-efficient learners such as Winnow, and that analytical approaches to establishing IDBD’s attribute efficiency show promise. Future work will analytically explore the space of attribute-efficient linear learners, including IDBD and related algorithms. By more clearly identifying the relationships between attribute-efficient algorithms, and by defining the functional features of each, it will become easier to explain experimental results, and to apply these algorithms to real-world problems.

Acknowledgments I wish to thank Dan Roth, Gary Dell, Jerry DeJong, Sylvian Ray, Jesse Reichler, Dav Zimak, Ashutosh Garg, and several anonymous reviewers for their helpful suggestions on this and earlier versions of this paper. This work was in part supported by NSF grant SBR-98-73450 and NIH grant DC-00191.

References 1. P. Auer and M. K. Warmuth. Tracking the best disjunction. Machine Learning, 32:127–150, 1998. 144 2. N. Christiani and J. Shawe-Taylor. An Introduction to Support Vector Machines. Cambridge University Press, 2000. 136 3. Claudio Gentile. A new approximate maximial margin classification algorithm. Journal of Machine Learning Research, 2:213–242, December 2001. 143 4. Adam J. Grove, Nick Littlestone, and Dale Schurrmans. General convegence results for linear discriminant updates. Machine Learning, 43(3):173–210, 2001. 143, 145 5. Harlan D. Harris and Jesse A. Reichler. Learning in the cerebellum with sparse conjunctions and linear separator algorithms. In Proceedings of the International Joint Conference on Neural Networks 2001, 2001. 136, 138, 143 6. Robert A. Jacobs. Increased rates of convergence through learning rate adaptation. Neural Networks, 1:295–307, 1988. 137 7. Roni Khardon, Dan Roth, and Rocco Servedio. Efficiency versus convergence of boolean kernels for on-line learning algorithms. In Proceedings of Neural Information Processing Systems 2001, 2001. 135, 136 8. J. Kivinen, M. K. Warmuth, and P. Auer. The Perceptron algorithm versus Winnow: Linear versus logarithmic mistake bounds when few input variables are relevant. Artificial Intelligence, 97:325–343, 1997. 135, 137, 139 9. N. Littlestone and M. K. Warmuth. The weighted majority algorithm. Information and Computation, 108:212–261, 1994. 145 10. Nick Littlestone. Learning quickly when irrelevant attributes abound: A new linearthreshold algorithm. Machine Learning, 2:285–318, 1988. 135, 136, 144, 145


147

11. Nick Littlestone. Mistake bounds and logarithmic linear-threshold learning algorithms. PhD thesis, University of California, Santa Cruz, Technical Report UCSCCRL-89-11, March 1989. 135, 136, 145 12. Dan Roth. Learning to resolve natural language ambiguities: A unified approach. In Proceedings of AAAI-98, 15th Conference of the American Association for Artificial Intelligence, pages 806–813, 1998. 135, 136, 143 13. Nicolas Schweighofer and Michael A. Arbib. A model of cerebellar metaplasticity. Learning and Memory, 4:421–428, 1998. 136 14. Richard S. Sutton. Adapting bias by gradient descent: An incremental version of Delta-Bar-Delta. In Proceedings of the Tenth National Conference on Artificial Intelligence, pages 171–176. MIT Press, 1992. 135, 137, 138, 141, 144 15. Leslie G. Valiant. Projection learning. Machine Learning, 37:115–130, 1999. 136, 143

Scaling Boosting by Margin-Based Inclusion of Features and Relations Susanne Hoche and Stefan Wrobel Otto-von-Guericke University, Magdeburg, Germany {hoche,wrobel}@iws.cs.uni-magdeburg.de

Abstract. Boosting is well known to increase the accuracy of propositional and multi-relational classification learners. However, the base learner’s efficiency vitally determines boosting’s efficiency since the complexity of the underlying learner is amplified by iterated calls of the learner in the boosting framework. The idea of restricting the learner to smaller feature subsets in order to increase efficiency is widely used. Surprisingly, little attention has been paid so far to exploiting characteristics of boosting itself to include features based on the current learning progress. In this paper, we show that the dynamics inherent to boosting offer ideal means to maximize the efficiency of the learning process. We describe how to utilize the training examples’ margins - which are known to be maximized by boosting - to reduce learning times without a deterioration of the learning quality. We suggest to stepwise include features in the learning process in response to a slowdown in the improvement of the margins. Experimental results show that this approach significantly reduces the learning time while maintaining or even improving the predictive accuracy of the underlying fully equipped learner.

1

Introduction

Boosting is a method for enhancing learning algorithms by basing predictions on a group of specialized hypotheses. Instead of searching for one highly accurate prediction rule covering a given set of training examples, an ensemble of rules is constructed by repeatedly calling a base learner with a changing distribution of weights for the training examples. Each rule in the ensemble might cover only a small subset of the examples, and all predictions are combined into one accurate joint prediction. Boosting is a popular technique for increasing the accuracy of classification learners and has been developed into practical algorithms that have demonstrated superior performance on a broad range of application problems in both propositional and multi-relational domains [3,19,17,5,7]. However, the iterative nature of boosting implies an amplification of the underlying learner’s complexity. Boosting’s efficiency is vitally determined by the base learner’s efficiency. A standard approach to deal with the issue of efficiency in the presence of large feature sets would be to use a feature selection method in an a priori fashion, and then run boosting with the small selected feature subset. However, deciding a priori on the number of features to be included in T. Elomaa et al. (Eds.): ECML, LNAI 2430, pp. 148–160, 2002. c Springer-Verlag Berlin Heidelberg 2002

Scaling Boosting by Margin-Based Inclusion of Features and Relations

149

the learning process might lead to inferior results since it is often difficult to decide just how many features to include. If too many features are included the learner is unnecessary slow, if too few features are included the learning result might not be sufficiently accurate. Instead, in this paper we suggest to actively determine the right balance between speed and accuracy of a learner based on its learning progress. We propose to monitor the learning success in terms of the development of the training examples’ mean margins - which are known to be maximized by boosting and to present step-by-step promising features to the learner whenever the improvement of the margins drops below a certain threshold. The margins’ improvement is measured by the ratio of the mean margins’ gradients averaged over several iterations and the current gradient of the training examples’ mean margins. This ratio increases from one iteration to the next as long as the margins increase significantly. As soon as the ratio starts to decrease, an estimate of the slowdown in the margins’ improvements is determined. This estimate predicts the expected decrease of the ratio and is used to determine when to provide a new feature to the learner. Whenever the actual decrease of the ratio is exceeding the predicted decrease by a certain factor, a new feature is included in the learning process. To this end, all features present in the training examples are initially sorted according to their mutual information [25,14] with the examples’ class, and new features are provided to the learner in a demand-driven fashion, starting with the top two features and the relations in which they occur. The evaluation of our approach on various domains shows that our approach significantly reduces learning times while maintaining or even improving predictive accuracy. Although our learner is multi-relational, our experiments indicate that the results apply equally well to boosting in propositional domains. This paper is organized as follows. In section 2, we review boosting. In section 3, we present our approach to include features and relations into the learning process in a demand-driven fashion. Our experimental evaluation of the approach is described and discussed in section 4. In section 5, we discuss related work and conclude in section 6 with some pointers to future work.

2

Boosting

Boosting is a method for improving the predictive accuracy of a learning system by means of combining a set of base classifiers constructed by a base learner into one single hypothesis [22,20,19]. The idea is to “boost” a weak learner performing slightly better than random guessing into an arbitrarily accurate learner by repeatedly calling the learner on varying probability distributions over the training instances. The probability distribution models the weight associated with each training instance and indicates the influence of an instance when building a base classifier. Initially, all instances have equal influence on the construction of a base hypothesis, i.e. the probability distribution is uniform. In each iterative call of the learner, a base hypothesis is learned with a prediction confidence for each example. The weights of misclassified instances are increased and those of cor-

150

Susanne Hoche and Stefan Wrobel

rectly classified instances are decreased according to the confidence of the learned base hypothesis. Thus, correctly classified instances have less and misclassified instances have more influence on the construction of the base hypothesis in the next iteration. That way, in each new round of boosting the learner is confronted with a modified learning task and forced to focus on the examples which have not yet been correctly classified. Finally, all base hypotheses learned are combined into one strong hypothesis. An instance x is classified by the strong hypothesis by adding up the prediction confidence of each base hypothesis covering x, and classifying x according to the sign of this sum. 2.1

Constrained Confidence-Rated Boosting

In this paper, we employ a specific form of constrained boosting, C 2 RIB, Constrained Confidence-Rated ILP-Boosting, which we introduced in [7] and which forms the basis of the work presented here (C 2 RIB D ). In Tables 1 and 2, we give a concise description of the proposed algorithm. Components of the base algorithm C 2 RIB in Table 1 are denoted by ’◦’. For a definition of the functions in the following explanation, the reader is referred to Table 1. In C 2 RIB, the training instances are randomly split into two sets used for specialization and pruning of clauses, respectively. Starting with the target predicate, the refinement operator ρ of the relational learner iteratively refines the clause C maximizing the objective function z˜ until either a clause C is found with hitherto maximal z˜(C ) that covers only positive examples, or z˜ can not be further maximized. The resulting clause is subject to overfitting on the training data, and thus immediately considered for pruning. The generated hypothesis is compared to the so called default hypothesis, just comprising the target predicate and satisfying all examples. Whichever of these two hypotheses maximizes the objective function z is chosen as the base classifier of the current iteration and its prediction confidence is used to update the probability distribution for the next iteration.

3

Margin-Based Inclusion of Features and Relations

The objective of our work presented here is to accelerate the learning process of the boosted ILP-learner C 2 RIB without a deterioration of its prediction accuracy. The idea is to equip the learner at all times with the right amount of power needed to “successfully” perform the learning task, i.e. to start the learner with a few features and relations to be considered for refinement, monitor the learning results and include additional features and relations into the learning process by demand. For this purpose, we exploit the dynamics inherent to boosting, namely that it a) is known to maximize training examples’ margins, b) is based on combining classifiers specialized on certain fractions of the instance space, and c) works by repeatedly calling a weak learner. Table 2 and the sections of Table 1 marked with ’•’ give a concise description of the algorithm which is detailed in the following. References to Table 2 will be indicated by “T2. ”.


151

Table 1. C 2 RIB D Algorithm Let N denote the number of training instances ei = (xi , yi ) ∈ E = E + ∪E − , and yi = 1 for ei ∈ E + and yi = −1 for ei ∈ E − . Let p be the target predicate of arity a(p) to be learned, T the total number of iterations of the weak learner, and D a probability distribution over E with Dit the probability of ei in the t-th iteration. For a clause C and a set S ⊆ E, let w+ , w− be weight functions defined as in 1. and 2. further down this page, and c(C, S) C’s prediction confidence on S defined according to 3. • Let F be the set of features sorted in descending order with respect to their mutual information with the examples’ class computed according to equation (2), and F the set of features known to the learner, initially comprising the top two features of F. ◦ Set Di1 := N1 for 1 ≤ i ≤ N ◦ For t = 1 . . . T Dit ≈ 23 ◦ Split E randomly into G and P according to Dt s.t. (xi ,yi )∈G ◦ C := p(X1 , · · · , Xa(p) ) ◦ Z˜ := 0 ◦ While w− (C, G) > 0 z (C )}, where z˜ is defined as in 4. ◦ Let C := argmaxC ∈ρ(C) {˜ ˜ ◦ Let Z := z˜(C ) ◦ If Z˜ − Z˜ ≤ 0 exit loop ◦ Else C := C , Z˜ := Z˜ ◦ P runes(C) := {p(X1 , · · · , Xa(p) ) ← B | C = p(X1 , · · · , Xa(p) ) ← BB } ◦ Remove from P runes(C) all clauses C where c(C , E) ≤ 0 ◦ If P runes(C) = ∅ let Ct := p(X1 , · · · , Xa(p) ) ◦ Else ◦ C := argminC ∈P runes(C) {loss(C )}, with loss(C ) defined as in 5. ◦ Let Ct := argmaxC ∈{C ,p(X1 ,···,Xa(p) )} {(z(C )}, with z defined as in 6. ◦ ht : X → is the function c(Ct , E) if e = (x, y) ∈ E is covered by Ct ht (x) = 0 else Dit

◦ Update the probability distribution: Dit :=

e(yi ·ht (xi ))

t

D , Dit+1 := i i

Dit

,1 ≤

i≤N =F • If t > 2 and F • Let Ht := {h1 , · · · , ht }, with base classifier hk of iteration 1 ≤ k ≤ t • F = CheckLearningP rogress(Ht, t, E, N, F, F , ) as detailed in Table 2

◦ Construct the strong hypothesis H(x) := sign

3.1

Ct :(x,y) covered by Ct

c(Ct , E)

Margins in the Framework of Confidence-Rated Boosting

In this approach, we monitor the learning success by observing the training examples’ mean margins. The margin of an example ei = (xi , yi ) under an ensemble Ht of classifiers is a real-valued number margin(Ht , ei ) ∈ [−1, 1] indicating

152


Table 1. continue

Function Definitions

Dt 1. w+ (C, S) =def. (xi ,1)∈S covered by C i t 2. w− (C, S) =def. D (xi ,−1)∈S covered by C i 1 3. c(C, S) =def.

1 2

ln

w+ (C,S)+ 2N

1 w− (C,S)+ 2N

.

4. z˜(C) =def. w+ (C, G) − w− (C, G). 5. loss(C) =def. (1 − (w+ (C, P) + w− (C, P))) + w+ (C, P) · e(−c(C,G)) + w− (C, P) · e(c(C,G)) 6. z(C) =def.

w+ (C, E) −

2

w− (C, E)

the amount of disagreement of the classifiers in Ht with respect to ei ’s class. For the binary case we deal with here, we can define the margin of ei under Ht as the difference between the sum of the absolute weights of those base classifiers in Ht predicting for ei its correct class yi , and the sum of the absolute weights of those base classifiers in Ht predicting for ei the incorrect class y = yi [24,6]. We define the weight w(hk , ei ) of a base classifier hk with respect to an example ei = (xi , yi ) as its prediction confidence (as defined in Table 1, 3.) if hk covers ei , and 0 otherwise. We define, following [6], the margin of ei under ensemble Ht = {h1 , · · · , ht } of t classifiers hk with weights w(hk , ei ) as |w(hk , ei )| − |w(hk , ei )| . (1) margin(Ht , ei ) = hk ∈Ht :hk (xi )=yi

hk ∈Ht :hk (xi )=yi

We normalize the prediction confidences of the base classifiers such that the absolute values of the confidences of allbase classifiers sum to 1. Consequently, hk ∈Ht :hk (xi )=yi |w(hk , ei )| ∈ [0, 1], hk ∈Ht :hk (xi )=yi |w(hk , ei )| ∈ [0, 1], and margin(Ht , ei ) ∈ [−1, 1] for all ei ∈ E and ensembles Ht . Large positive margins (close to +1) indicate “confident” correct classification, and small negative margins (close to −1) indicate “confident” incorrect classification. Boosting is known to be especially effective at increasing the margins of the training examples [24,6]. It forces the focus on misclassified instances by increasing their probabilities. Misclassified examples show small or even negative margins. Consequently, the learner is forced to search for base hypotheses which correctly classify these hard examples and thus increase their margins. Since the margins are increasing in the course of iterated calls to the base learner, the gradient of the mean margins can be assumed to be positive and be employed to monitor the quality of the learning process. The repeated calls of the base learner in the boosting framework allow for a stepwise inclusion of features in the course of iterations. If the learning curve indicates that the learner’s current instrumentation is not sufficient any longer,


153

Table 2. CheckLearningProgress CheckLearningP rogress(Ht, t, E, N, F, F ) returns F

n

1. Compute for E the examples’ average margin AMt = N1 margin(Ht, ei ) i=1 according to equation (1) 2. Let gradient(t) be the slope of the line determined by the least square fit to the AMk in k, 1 ≤ k ≤ t Tl 1 gradient(t − j) if t > Tl j=1 3. Compute trend(t) := T1l t−1 gradient(j) if t ≤ Tl , t−2 j=2 where Tl denotes the number of iterations over which the gradients are averaged trend(t) 4. Compute ratio(t) := gradient(t) 5. If t > 3: (a) If ratio(t−1) exhibits a local maximum, estimate the slowdown in the margins’ improvement in the form of predict(x) := a ln(1x ) , where a, b are chosen such b that predict(2) = ratio(t − 1) and predict(3) = ratio(t); of f set := t − 3 1 (b) If a, b have already been determined, compute predict(t) := a t−of f set ln(

b

)

(c) Else predict(t) := ratio(t) (d) If predict(t) > α, select the first element F of F, i.e. the feature with the next ratio(t) greatest mutual information with the training examples’ class; F := F ∪ {F } (e) Else F := F

learning can be continued in the next iteration with an enhanced equipment. Initially, we provide our learner with the target relation to be learned together with two features with the greatest mutual information [25,14] with the examples’ class and the relations in which these features occur. In each iteration of boosting, the learning success is monitored in terms of the development of the training examples’ mean margins. To this end, we define the gradient gradient(t) of an iteration t as the slope of the line determined by the least square fit to the average margins in each single iteration 1 to t (T2.1, T2.2). We then average the gradients over the last Tl iterations as to smooth temporary fluctuations in the margins’ development (T2.3), and compute the ratio of the averaged previous gradients and the gradient of the current iteration (T2.4). The margins’ improvement is measured by this ratio which increases from one iteration to the next as long as the margins increase significantly. As soon as the ratio starts to decrease, an estimate for the slowdown in the margins’ improvements is determined (T2.5a). This estimate predicts the expected decrease of the ratio and is used to determine when a new feature has to be presented to the learner. The estimate is chosen to be an inverse-logarithm. Whenever the actual decrease of the ratio exceeds the predicted decrease by a certain threshold, a new feature is included into the learning process (T2.5d).

154

3.2


Mutual Information between Features

Initially, all features in the given training examples are sorted according to their mutual information [25,14] with the examples’ class. The mutual information M I(F1 , F2 ) between two features F1 , F2 is defined as the difference between the entropy of F1 and the entropy of F1 given F2 [27], i.e. as the amount of information about the possible values (f11 , f12 , ...) of feature F1 that is obtained when the value f ∈ {f21 , f22 , ...} of feature F2 is known. To compute the mutual information between class C and feature Fj , we estimate the probability distributions of C and Fj from the training data, ignoring missing values, as follows: – The probability p(C = c) of any training example being of class c is estimated c | as the fraction |E |E| of training examples from E belonging to c. – The probability p(Fj = fi ) that the nominal feature Fj takes value fi is |F =f | estimated as the fraction j|E| i of training examples for which feature Fj takes value fi . – The joint probability p(C = c) ∧ (Fj = fi ) is derived from the probabilities of the two single events. The mutual information between a feature Fj and the class C of an example can then be defined as M I(C, Fj ) = E(C) − E(C|Fj ) =

mj k

p(C = c, Fj = fi ) ln

i=1 c=1

p(C = c, Fj = fi ) p(C = c)p(Fj = f i)

(2)

with k possible classes and mj possible values of feature Fj . For features Fj with continuous values, we estimate the probability distribution by discretizing the values of Fj with an entropy based method [4] and using the resulting interval [d1 , ..., dmi ] to estimate the probability of Fj taking a value in the interval Ii := [di , di+1 ), 1 ≤ i < mi − 1, and Imi−1 := [dmi −1 , dmi ] respectively, |Fj ∈Ii | of training examples for which feature Fj takes a value as the fraction |E| in Ii , 1 ≤ i ≤ mi − 1. Note that this way of sorting features according to their mutual information with respect to classification assumes independence of the features and may thus result in inferior performance in domains with highly correlated features.

4

Empirical Evaluation

To evaluate our approach, we performed experiments on data sets differing in the number of features and the total number of examples. We determine prediction accuracy and learning time for each dataset for both the base case C 2 RIB and for C 2 RIB D described in this paper, and compare the results to those of other systems (Tables 3 to 5). For C 2 RIB D , we also indicate the average number of features included in the learning process. In all experiments, 1) the


155

Table 3. Accuracy, standard deviation and learning time in minutes for SLIPPER [3], C 2 RIB and C 2 RIB D on five propositional domains SLIPPER Domain

Ex. Fea- Train / Acc tures

breast-wisc

699

horse-colic

368

hypothyroid 3163 mushroom 8124 splicejunction

3190

Test

StdD

9 10CV 95.8 n/a 23 10CV 85.0 n/a 25 10CV 99.3 n/a 22 10CV 99.8 n/a 60 10CV 94.1

[3] C2 RIB Time Acc StdD

n/a 96.1 ±1.5 n/a 81.0 ±8.4 n/a 95.2 ±0.69 n/a 99.3 ±3.0 n/a 53.13 ±3.0

C2 RIBD Time Acc

Time

StdD

9 95.4 ±1.7 3.6 83.7 ±5.7 39.1 96.6 ±2.9 144 99.6 ±0.16 289 88 ±4.7

sel. Feat.

5.1

5.8

0.9

2

20.8

11.8

71.4

4.8

13.6

4.4

base learner is invoked T = 100 times, 2) the gradients of the examples’ mean margins are averaged over the last Tl = 10 iterations and, 3) the threshold α is set to 1.01 (see 5d in Table 2). The value 1.01 has been empirically determined on the domain of Mutagenicity [26], and has not been modified for subsequent experiments on the other domains in order to ensure proper cross validation results. We chose three different types of domains in order to get an assessment of our learner 1) on propositional tasks, and 2) on general knowledge and data mining tasks and 3) on ILP benchmark and classic Machine Learning problems. The first set of experiments comprises five propositional domains from the UCI-repository [16]. We compare our approach to the propositional constrained confidence-rated booster SLIPPER [3] which served as a basis for C 2 RIB. Predictive accuracies are estimated by 10-fold-cross validation.1 As can be seen from Table 3, C 2 RIB performs in four domains on par with or slightly weaker than SLIPPER. C 2 RIB D reduces C 2 RIB’s learning time2 up to one order of magnitude with a superior predictive accuracy in four domains, and without a significant deterioration of predictive accuracy in the one domain where only few features are present. C 2 RIB shows a poor performance on the splice-junction dataset, most likely due to the great number of features. However, C 2 RIB D clearly outperforms C 2 RIB both in accuracy and learning time. The second set of experiments was conducted on datasets subject of the data mining competitions PKDD Discovery Challenge 2000 [1] (classification of loans, where Task AC is based on all loans, and Task A only on the closed loans from 1

2

However, in [3], single training- and test set splits are used for hypothyroid, mushroom and splice-junction. Learning times for SLIPPER are not known to us.

156


Task AC), and KDD Cup 2001, Task2 [2] (prediction of gene functions). The predictive accuracy is estimated by 10-fold-cross validation, and the results are compared to Progol [15] and RELAGGS [13], a transformation-based approach to ILP, combined with SVMlight and C4.5rules, respectively, run on the propositionalized data. For Task AC, Progol was run for 2 days, and discontinued without any results. Prediction accuracies of C 2 RIB and C 2 RIB D are, for Task AC, notedly lower than the ones obtained by RELAGGS/C4.5rules, however still in the range of standard deviation of the accuracies obtained by RELAGGS/SVMlight , as holds for Task A. However, learning times of C 2 RIB and C 2 RIB D are lower than the ones of the other systems. For Task AC, C 2 RIB D speeds up C 2 RIB’s learning time by factor 2. For Task A, C 2 RIB D seems to be penalized for sorting the features in the presence of few examples. For the gene function

Table 4. Accuracy, standard deviation and learning time for Progol [15], RELAGGS [13], C 2 RIB and C 2 RIB D on some data mining competition domains Progol RELAGGS RELAGGS C2 RIB C2 RIBD Domain

Ex.

SVMlight

C4.5rules

Acc

Acc

Acc

Acc

sel.

StdD

StdD

StdD

StdD

Feat.

Time

Time

Time

Time

Time

n/a n/a 2 days 45.7 ±10.5 hrs 92.2 24 min

90.8 ±3.2 23 min 88.0 ±5.3 10 min 92.2 ≈ 2 min

94.1 ±3.2 23 min 88.0 ±6.5 10 min n/a n/a

88.9 ±3.4 20 min 86.3 ±6.1 3.6 min 91.1 53 min

88.9 ±3.4 9.5 min 86.7 ±6.6 4.2 min 91.5 27 min

Fea- Acc

Train / tures StdD Test PKDD DS 2000, AC PKDD DS

2000, A KDD Cup 2001, Task2

682 10CV

24

234 10CV

24

1243 862/381

49

10

10.2

5

prediction task, C 2 RIB and C 2 RIB D were ran on the original KDD Cup 2001 training-test-data partition and the results were compared to Progol3 and RELAGGS/SVMlight .4,5 Again, learning time is reduced by factor 2 in the demand-driven approach C 2 RIB D . It slightly improves C 2 RIB’s predictive accuracy which is on par with the other systems’ accuracies. Finally, we evaluated our approach on the two ILP benchmark problems Mutagenicity [26] (prediction of mutagenic activity of 188 molecules (description B4 )) and QSARs, Quantitative Structure Activity Relationships, [9,10] (prediction of a greater-activity relationship between pairs of compounds based on 3 4 5

L. Pe˜ na Castillo, unpublished, 2002 M.-A. Krogel, unpublished, 2002 RELAGGS won Task of KDD Cup 2001.


157

Table 5. Accuracy, standard deviation and learning time in minutes for C 2 RIB and C 2 RIB D in comparison to other systems on two ILP benchmark and one artificial domain Fors

Progol

C2 RIB C2 RIBD

Acc

Acc

Acc

Acc

sel.

StdD

StdD

StdD

StdD

Feat.

Time

Time

Time

Time

82.0 [26] 89.0 [8] 88.0 [26] 88.0 ±3.0 ±6.0 ±2.0 ±3.4 n/a n/a 307 7 82.9 n/a 79.8 83.4 ±2.7 n/a ±3.7 ±2.9 0.7 n/a 372 91 n/a n/a 77.78 [18] 83.3 n/a n/a ±6.43 ±0 n/a n/a 1.15 0.44

88.8 ±5.2 1.53 83.3 ±1.9 70 89.6 ±8.6 0.1

FOIL Domain

Ex. Fea- Acc Train/ tures StdD Test

Time

188 10CV

18

QSARs

2788 5CV

12

Eastbound

55 / 6

9

Mutagenicity

Trains

6

11.8

6.25

their structure), and on the artificial problem of Eastbound Trains6 proposed by Ryszard Michalski (prediction of trains’ directions based on their properties). For the two ILP domains, predictive accuracy is estimated by 10- and 5-foldcross validation, respectively, and results are compared to FOIL [21], Fors [8] and Progol. For the Eastbound Trains, the data is split into one training and test set partition, and the results are averaged over 8 iterations of the experiment. Predictive accuracy of C 2 RIB is higher than or on par with the one of the other learners. C 2 RIB D significantly outperforms C 2 RIB both in terms of predictive accuracy and learning time in two of the three domains, indicating that our approach seems to be superior in classical, highly structured ILP domains.

5

Related Work

The idea of selecting smaller feature subsets and shifting the bias to a more expressive representation language is common in multi-relational learning. The work probably most related to our work is [12], where AdaBoost [22] is combined with molfea, an inductive database for the domain of biochemistry [11]. In [12], AdaBoost is employed to identify particularly difficult examples for which molfea constructs new special purpose structural features. AdaBoost re-weighting episodes and molfea feature construction episodes are alternated. In each iteration, a new feature constructed by molfea is presented to a propositional learner, the examples are re-weighted in accordance to the base classifier learned by it, and a new feature is constructed by molfea based on the modified weights. In contrast, our approach actively decides when to include new features 6

The examples were generated with the Random Train Generator available at http://www-users-cs-york.ac.uk/∼stephen/progol.html

158


from the list of ranked existing features with the central goal of including new features only when absolutely necessary in order to be maximally efficient. This means that in principle the two approaches could be easily combined, for example by calling a generator of new features whenever the list of existing features has been exhausted. [23] propose a wrapper model utilizing boosting for feature selection. In their approach, alternative feature subsets are assessed based on the underlying booster’s optimization criterion. The feature subset optimal according to this criterion is then presented as a whole to a learner. In contrast, we use a criterion of mutual information once before we start the boosting process to establish a feature ranking, and utilize the characteristics of our boosted learner to actively decide when to include a new feature. However, it would be interesting to combine both approaches.

6

Conclusion

In this paper, we have proposed an approach to boosting a weak relational learner which starts off with a minimal set of features and relations and is - by demand stepwise strengthened. Our work is based on C 2 RIB [7], a fast weak ILP-learner in a constrained confidence-rated boosting framework. The quality of the current learning results is measured in terms of the gradient of the training examples’ mean margins, and the learner is strengthened whenever the learning curve drops under a certain threshold. To that purpose, features occurring in the training examples are sorted according to their mutual information with the examples’ class and by and by provided to the learner together with the relation in which they occur. We showed that learning times are significantly reduced while the predictive accuracy is comparable to those of other learning systems and, in the majority of cases, superiour to those of the “fully equipped” learner C 2 RIB. These results are encouraging, especially since all experiments were conducted without optimizing parameters. One question for further work is whether one could expect to even gain a higher predictive accuracy by repeatedly evaluating the features’ ordering and taking into account the examples’ weights under the current probability distribution. In each iteration, the learner is presented a different training set, emphasizing the hard examples more and more. A stronger influence of so far misclassified examples on the feature ranking could support the induction of correct classifiers for those examples that are particularly difficult to learn. Another question for further research is whether it is possible to determine automatically for every domain a) an optimal threshold to which the deviation of the current from the expected decrease of the ratio of the average and the current gradient should be compared and b) the number of iterations over which the gradients should be averaged. It is also part of the future work to investigate other approaches to feature selection, and make use of the accelerated learning time to incorporate more standard elements of “full-blown” ILP-learners and to determine the right balance between speed and accuracy of the learning system.


159

This work was partially supported by DFG (German Science Foundation), project FOR345/1-1TP6. We would like to thank L. Pe˜ na Castillo and M. Krogel for providing their results on the KDD Cup 2001, L. Pe˜ na Castillo for reviewing previous versions of this paper, and J. Kaduk for many inspiring discussions.

References 1. P. Berka. Guide to the financial Data Set. In: A. Siebes and P. Berka, editors, PKDD2000 Discovery Challenge, 2000. 155 2. J. Cheng, C. Hatzis, H. Hayashi, M.-A. Krogel, Sh. Morishita, D. Page, and J. Sese. KDD Cup 2001 Report. ISIGKDD Explorations, 3(2):47-64, 2002. 156 3. W. Cohen and Y. Singer. A Simple, Fast, and Effective Rule Learner. Proc. of 16th National Conference on Artificial Intelligence, 1999. 148, 155 4. U. M. Fayyad, and K. B. Irani. Multi-Interval Discretization of Continuous-Valued Attributes for Classification Learning. Proc. of 13th Int. Joint Conf. on AI, 1993. 154 5. Y. Freund, and R. E. Schapire. Experiments with a New Boosting Algorithm. Proc. of 13th International Conference on Machine Learning, 1996. 148 6. A. J. Grove, and D. Schuurmans. Boosting in the limit: Maximizing the margin of learned ensembles. Proc. of 15th National Conf. on AI, 1998. 152 7. S. Hoche, and S. Wrobel. Relational Learning Using Constrained Confidence-Rated Boosting. Proc. 11th Int. Conf. on Inductive Logic Programming (ILP), 2001. 148, 150, 158 8. A. Karalic. First Order Regression. PhD thesis, University of Ljubljana, Faculty of Computer Science, Ljubljana, Slovenia, 1995. 157 9. R. D. King, S. Muggleton, R. A. Lewis, and M. J. E. Sternberg. Drug design by machine learning: The use of inductive logic programming to model the structure activity relationships of trimethoprim analogues binding to dihydrofolate reductase. Proc. of the National Academy of Sciences of the USA 89(23):11322-11326, 1992. 156 10. R. D. King, A. Srinivasan, and M. Sternberg. Relating chemical activity to structure: An examination of ILP successes. New Generation Computing, Special issue on Inductive Logic Programming 13(3-4):411-434, 1995. 156 11. S. Kramer, and L. De Raedt. Feature construction with version spaces for biochemical applications. Proc. of the 18th ICML, 2001. 157 12. S. Kramer. Demand-driven Construction of Structural Features in ILP. Proc. 11th Int. Conf. on Inductive Logic Programming (ILP), 2001. 157 13. M.-A. Krogel , and S. Wrobel. Transformation-Based Learning Using Multirelational Aggregation. Proc. 11th Int. Conf. on Inductive Logic Programming (ILP), 2001. 156 14. W. J. McGill. Multivariate information transmission. IRE Trans. Inf. Theory, 1995. 149, 153, 154 15. S. Muggleton. Inverse Entailment and Progol. New Gen. Computing, 13, 1995. 156 16. P. M. Murphy, and D. W. Aha. UCI repository of machine learning databases. University of California-Irvine, Department of Information and Computer Science, 1994. http://www1.ics.uci.edu/ mlearn/MLRepository.html 155 17. D. Opitz, and R. Maclin. Popular Ensemble Method: An Empirical Study. Journal of Artificial Intelligence Research 11, pages 169-198, 1999. 148

160


18. L. Pe˜ na Castillo, S. Wrobel. On the Stability of Example-Driven Learning Systems: a Case Study in Multirelational Learning. Proceedings of MICAI 2002, 2002. 157 19. J. R. Quinlan. Bagging, boosting, and C4.5. Proc. of 14th Nat. Conf. on AI, 1996. 148, 149 20. J. R. Quinlan. Boosting First-Order Learning. Algorithmic Learning Theory, 1996. 149 21. J. R. Quinlan and R. M. Cameron-Jones. FOIL: A Midterm Report. In P. Brazdil, editor, Proc. of the 6th European Conference on Machine Learning, 667: 3-20, 1993. 157 22. R. E. Schapire. Theoretical views of boosting and applications. Proceedings of the 10th International Conference on Algorithmic Learning Theory, 1999. 149, 157 23. M. Sebban, and R. Nock. Contribution of Boosting in Wrapper Models. In: J. M. Zytkow, and J. Rauch, eds, Proc. of the PKDD’99, 1999. 158 24. R. E. Schapire, Y. Freund, P.Bartlett, and W. S. Lee. Boosting the Margin: A New Explanation for the Effectiveness of Voting Methods. The Annals of Statistics, 26(5):1651-1686, 1998. 152 25. C. E. Shannon. A mathematical theory of communication. Bell. Syst. Techn. J., 27:379-423, 1948. 149, 153, 154 26. A. Srinivasan, S. Muggleton, M. J. E. Sternberg, and R. D. King. Theories for mutagenicity: A study in first-order and feature-based induction. Artificial Intelligence, 1996. 155, 156, 157 27. D. Wettschereck. A Study of Distance-based Machine Learning Algorithms. PhD thesis, Oregon State University,Computer Science Department, Corvallis, USA 1994. 154

Multiclass Alternating Decision Trees Geoffrey Holmes, Bernhard Pfahringer, Richard Kirkby, Eibe Frank, and Mark Hall Department of Computer Science, University of Waikato Hamilton, New Zealand {geoff,bernhard,rkirkby,eibe,mhall}@cs.waikato.ac.nz

Abstract. The alternating decision tree (ADTree) is a successful classification technique that combines decision trees with the predictive accuracy of boosting into a set of interpretable classification rules. The original formulation of the tree induction algorithm restricted attention to binary classification problems. This paper empirically evaluates several wrapper methods for extending the algorithm to the multiclass case by splitting the problem into several two-class problems. Seeking a more natural solution we then adapt the multiclass LogitBoost and AdaBoost.MH procedures to induce alternating decision trees directly. Experimental results confirm that these procedures are comparable with wrapper methods that are based on the original ADTree formulation in accuracy, while inducing much smaller trees.

1

Introduction

Boosting is now a well established procedure for improving the performance of classification algorithms. AdaBoost [8] is the most commonly used boosting procedure, but others have gained prominence [3,10]. Like many classification algorithms, most boosting procedures are formulated for the binary classification setting. Schapire and Singer generalize AdaBoost to the multiclass setting producing several alternative procedures of which the best (empirically) is AdaBoost.MH [14]. This version of AdaBoost covers the multilabel setting where an instance can have more than one class label as well as the multiclass setting where an instance can have a single class label taken from a set of (more than two) labels. Alternating decision trees are induced using a real-valued formulation of AdaBoost [14]. At each boosting iteration three nodes are added to the tree. A splitter node that attempts to split sets of instances into pure subsets and two prediction nodes, one for each of the splitter node’s subsets. The position of this new splitter node is determined by examining all predictor nodes choosing the position resulting in the globally best improvement of the purity score. Essentially, an ADTree is an AND/OR graph. Knowledge contained in the tree is distributed as multiple paths must be traversed to form predictions. Instances that satisfy multiple splitter nodes have the values of prediction nodes that they reach summed to form an overall prediction value. A positive sum represents one class and a negative sum the other in the two-class setting. The result T. Elomaa et al. (Eds.): ECML, LNAI 2430, pp. 161–172, 2002. c Springer-Verlag Berlin Heidelberg 2002

162

Geoffrey Holmes et al.

is a single interpretable tree with predictive capabilities that rival a committee of boosted C5.0 trees [7]. An additional attractive feature of ADTrees, one that is not possible with conventional boosting procedures, is their ability to be merged together. This is a particularly useful attribute in the context of multiclass problems as they are often re-formulated in the two-class setting using one or more classes against the others. In such a setting ADTrees can be combined into a single classifier. In their original exposition on ADTrees, Freund and Mason [7] note that because alternating trees can be defined as a sum of simple base rules it is a simple matter to apply any boosting algorithm to the problem of inducing ADTrees. For the multiclass setting one possible candidate is AdaBoost.MH. In this paper we also explore and compare two other solutions. The first is to adapt the original two-class ADTree algorithm to the multiclass setting using a variety of wrapper methods. The second is to use the multiclass LogitBoost [10] procedure as the underlying boosting algorithm. This algorithm is a natural choice as it is directly applicable to multiclass problems. The paper is organized as follows. In Section 2 we review ADTrees and the LogitBoost procedure. Section 3 describes our attempts to cast ADTrees to the multiclass setting. Section 4 describes the new algorithm that induces ADTrees using LogitBoost. Section 5 contains experimental results that compare both the LogitBoost and AdaBoost.MH methods with the best of the adaptations of the original algorithm on some benchmark datasets. Section 6 summarizes the contributions made in this paper.

2

Background

In this section we first summarize the original algorithm for inducing ADTrees. As Freund and Mason [7] argue that any boosting method is applicable to ADTree induction, it is natural to suppose that AdaBoost.MH would provide a good setting for the multiclass extension (given that AdaBoost works so well in the two-class setting). A similar argument can be made for an alternative framework based on LogitBoost, and this is discussed in the final part of this section. 2.1

ADTrees

Alternating decision trees provide a mechanism for combining the weak hypotheses generated during boosting into a single interpretable representation. Keeping faith with the original implementation, we use inequality conditions that compare a single feature with a constant as the weak hypotheses generated during each boosting iteration. In [7] some typographical errors and omissions make the algorithm difficult to implement so we include below a more complete description of our implementation. At each boosting iteration t the algorithm maintains two sets, a set of preconditions and a set of rules, denoted Pt and Rt , respectively. A further set C of weak hypotheses is generated at each boosting iteration.

Multiclass Alternating Decision Trees

163

1. Initialize Set the weights wi,t associated with each training instance to 1. Set the first rule R1 to have a precondition and condition which are both true. CalW+ (c) culate the prediction value for this rule as a = 12 ln W where W+ (c), W− (c) − (c) are the total weights of the positive and negative instances that satisfy condition c in the training data. The initial value of c is simply True. 2. Pre-adjustment Reweight the training instances using the formula wi,1 = wi,0 e−ayt (for two-class problems, the value of yt is either +1 or -1). 3. Repeat for t = 1, 2, . . ., T (a). Generate the set C of weak hypotheses using the weights associated with each training instance wi,t (b). For each base precondition c1 ∈ Pt and each condition c2 ∈ C calculate Zt (c1 , c2 ) = 2

p

W+ (c1 ∧ c2 )W− (c1 ∧ c2 ) +

!

p

W+ (c1 ∧ ¬c2 )W− (c1 ∧ ¬c2 ) + W (¬c1 )

(c). Select c1 , c2 which minimize Zt (c1 , c2 ) and set Rt+1 to be Rt with the addition of the rule rt whose precondition is c1 , condition is c2 and two prediction values are: a=

1 W+ (c1 ∧ c2 ) + ln , 2 W− (c1 ∧ c2 ) +

b=

1 W+ (c1 ∧ ¬c2 ) + ln 2 W− (c1 ∧ ¬c2 ) +

(d). Set Pt+1 to be Pt with the addition of c1 ∧ c2 and c1 ∧ ¬c2 . (e). Update the weights of each training example according to the equation wi,t+1 = wi,t e−rt (xi )yt 4. Output the classification rule that is the sign of the sum of all the base rules in RT +1 : T class(x) = sign rt (x) t=1

In terms of parameter settings for implementations described in this paper, we set the value of to 1, and vary the value of T for stopping the induction in fixed increments (namely, 10, 20, 50 and 100). Determining an optimal setting for T is still an open research question. 2.2

LogitBoost

As mentioned above, the underlying learning algorithm for ADTrees is AdaBoost. Friedman et al. [10] analyze AdaBoost from a statistical perspective

164


and find that it can be viewed as a stage-wise estimation procedure for fitting an additive logistic regression model according to an exponential loss function. This finding enables them to derive a stage-wise boosting procedure, implementing an adaptive Newton algorithm, that optimizes the (more standard) binomial likelihood instead of the exponential loss function used in AdaBoost. They call this algorithm LogitBoost. They also describe a generalized version of LogitBoost that optimizes the multinomial likelihood. This algorithm is directly applicable to multiclass problems. Compared to AdaBoost.MH (see Section 3.2), the general form of LogitBoost (which we call LT1PC later) has the advantage that it can be wrapped around any numeric predictor without any modifications. AdaBoost.MH, on the other hand, requires serious modification to the weak learner so that it can produce a separate prediction for each class value and also deal with class specific weights.

3

Multiclass ADTrees

When extending any algorithm from binary to multiclass classification there are two options. The simplest approach is to transform the multiclass problem into several binary classification problems. This general approach can be applied to any classification algorithm, resulting in a set of voting models. Typically, this approach leads to a large number of models. Alternatively, we can attempt to induce a single tree capable of predicting each of the class labels directly. 3.1

Multiclass as Multiple Two-Class Problems

Transforming ADTrees to map multiple class labels to two classes can be approached in several ways. As ADTrees can be merged, the resulting multiclass model can be a single tree derived from the set of two-class voting trees. A standard method [6] is to treat a subset of class labels as class A, and the set of remaining labels as class B, thus reducing the problem to two classes from which a model can be built. This is then repeated for different subsets and the models vote towards the class labels they represent. Provided there is sufficient class representation and separation between the subsets, the vote tallies for individual class labels can be collected to form a reasonable prediction. We experimented with a number of subset generation schemes: 1-against-1 [9,1]: generate a tree for every pair of classes, where subset A contains only the first class and subset B contains only the second. An advantage of this approach is that each tree need only be trained with a subset of the data, resulting in faster learning [11]. 1-against-rest: one tree per class, where subset A contains the class, and subset B contains the remaining classes. random: randomly generate a unique subset, creating twice as many trees as there are classes. Random codes have good error-correcting properties [13]. exhaustive: every unique subset possible.


165

Note that the exhaustive method is not computationally practical as class numbers increase (in our experiments this occurs when there are more than 16 class labels). 3.2

Direct Induction

The AdaBoost.MH algorithm is almost identical to AdaBoost. The major difference is that instead of generating weak hypotheses ht that map the input space X to either a discrete set [−1, +1] or by extension R, the weak hypotheses map X × Y to R, where Y is a finite set of class labels. It would appear that the correct interpretation of AdaBoost.MH is not immediately obvious, for example, Friedmann et al [10] interpret the method as a variant of 1-against-rest and build a distinct classifier per class. Many of the criticisms of AdaBoost.MH in [10] are based on this mis-interpretation. Our results suggest that AdaBoost.MH and LogitBoost actually share much in common in terms of both predictive performance and computational complexity. In fact, AdaBoost.MH constructs a single tree per iteration. To construct an ADTree using AdaBoost.MH we need to change predictor nodes to handle a vector of predictions (one per class) and splitter nodes to compute a Z value per class label. At each iteration the test that minimises the sum of Z scores over all class labels is added to the tree. To perform prediction using this tree we sum all contributions at each predictor node that is satisfied by the example, to form a prediction vector containing a single prediction per class. We choose the maximum value from this vector as the single output class.

4

LADTree Algorithm

We follow Friedmann et al [10] in defining the multiclass context. Namely, that ∗ for an instance i and a J class problem, there are J responses yij each taking values in {−1, 1}. The predicted values, or indicator responses, are represented by the vector Fj (x) which is the sum of the responses of all the ensemble classifiers on instance x over the J classes. The class probability estimate is computed from a generalization of the two-class symmetric logistic transformation to be: J eFj (x) , Fk (x) = 0 pj (x) = J Fk (x) k=1 e k=1

(1)

The LogitBoost algorithm can be fused with the induction of ADTrees in two ways, which will be explained in the following subsections. In the first, more conservative approach called LT1PC we grow separate trees for each class in parallel. In the second approach called LT, only one tree is grown predicting all class probabilities simultaneously.

166

4.1


LT1PC: Inducing One Tree per Class

The LADTree learning algorithm applies the logistic boosting algorithm in order to induce an alternating decision tree. As with the original algorithm, a single attribute test is chosen as the splitter node for the tree at each iteration. Stored with each training instance is a working response and weights on a per-class basis. The aim is to fit the working response to the mean value of the instances, in a particular subset, by minimising the least-squares value between them. When choosing tests to add to the tree we look for the maximum gain, that is, the greatest drop in the least squares calculation. Note, in the algorithm below the fmj (x) vector is equivalent to the single prediction weight of a predictor node in the original ADTree algorithm. The algorithm is as follows: 1. Initialize Create a root node with Fj (x) = 0 and Pj (x) =

1 J ∀j

2. Repeat for m = 1, 2, . . . , T : (a) Repeat for j = 1, . . . , J : (i) Compute working responses and weights in the jth class y ∗ −pij y ∗ −pij wij = ijzij zij = pijij(1−pij ) (ii) Add the single test to the tree that best fits fmj (x) by a weighted least-squares fit of zij to xi with weights wij (b) Add prediction nodes to the tree by setting 1 J fmj (x) ← j−1 k=1 fmk (x)), and J (fmj (x) − J Fj (x) ← Fij (x) + fmj (x) (c) Update pj (x) via Equation 1 above 3. Output Output the classifier argmaxj Fj (x) With this algorithm, trees for the different classes are grown in parallel. Once all of the trees have been built, it is then possible to merge them into a final model. If the structure of the trees is such that few tests are common, the merged tree will mostly contain subtrees affecting only one class. The size of the tree cannot outgrow the combined size of the individual trees. The merging operation involves searching for identical tests on the same level of the tree. If such tests exist then the test and its subtrees can be merged into one. The additive nature of the trees means that the prediction values for the same class can be added together when merged. 4.2

LT: Directly Inducing a Single Tree

We can make a simple adjustment to this algorithm within Step 2 by moving Step (a)(ii) out to become Step (b). We then obtain a single directly induced tree, as follows:


167

2. Repeat for m = 1, 2, . . . , T : (a) Repeat for j = 1, . . . , J : (i) Compute working responses and weights in the jth class y ∗ −pij y ∗ −pij zij = pijij(1−pij ) wij = ijzij (b) Add the single test to the tree that best fits fmj (x) by a weighted least-squares fit of zij to xi with weights wij (c) Add prediction nodes to the tree by setting J 1 fmj (x) ← j−1 k=1 fmk (x)), and J (fmj (x) − J Fj (x) ← Fij (x) + fmj (x) (d) Update pj (x) via Equation 1 above The major difference to LT1PC is that in LT we attempt to simultaneously minimise the weighted mean squared error across all classes when finding the best weak hypothesis for the model.

5

Experimental Results

The datasets and their properties are listed in Table 1. The first set of ten datasets are used to compare ADTrees with LT as an algorithm for solving two-class problems. The remainder are used in multiclass experiments, ordered incrementally from the smallest number of classes (3) to the largest (26). Most of the datasets are from the UCI repository [2], with the exception of half-letter. Half-letter is a modified version of letter, where only half of the class labels (A-M) are present. In the case of the multiclass datasets, on the first nine having less than eight classes, accuracy estimates were obtained by averaging the results from 10 separate runs of stratified 10-fold cross-validation. In other words, each scheme was applied 100 times to generate an estimate for a particular dataset. For these datasets, we speak of two results as being “significantly different” if the difference is statistically significant at the 5% level according to a paired two-sided t-test, each pair of data points consisting of the estimates obtained in one ten-fold cross-validation run for the two learning schemes being compared. On the datasets with more than eight classes, a single train and test split was used. Statistical significance was measured by the McNemar [5] test. NA (for not available) in the results table signifies that the learning scheme did not finish training. If learning could not complete within the time period of a week then it was terminated and marked NA. It is not surprising that the exhaustive method did not finish above 16 classes when one considers the number of permutations required. Due to the presence of these unfinished experiments the averages for all methods listed in this table exclude the last four datasets. Thus a fair comparison is possible. Table 2 shows that LT is comparable to ADTree over ten boosting iterations on two-class datasets. There is little change to this result when raising the number of boosting iterations to 100.

168


Table 1. Datasets and their characteristics Dataset Classes Instances (train/test) Attributes Numeric Nominal breast-wisc 2 699 9 9 0 cleveland 2 303 13 6 7 credit 2 690 15 6 9 hepatitis 2 155 19 6 13 ionosphere 2 351 34 34 0 labor 2 57 16 8 8 promoters 2 106 57 0 57 sick-euthyroid 2 3163 25 7 18 sonar 2 208 60 60 0 vote 2 435 16 0 16 iris 3 150 4 4 0 balance-scale 3 625 4 4 0 hypothyroid 4 3772 29 7 22 anneal 6 898 38 6 32 zoo 7 101 17 1 16 autos 7 205 25 15 10 glass 7 214 9 9 0 segment 7 2310 19 19 0 ecoli 8 336 7 7 0 led7 10 1000/500 7 0 7 optdigits 10 3823/1797 64 64 0 pendigits 10 7494/3498 16 16 0 vowel 11 582/462 12 10 2 half-letter 13 8000/1940 16 16 0 arrhythmia 16 302/150 279 206 73 soybean 19 307/176 35 0 35 primary-tumor 22 226/113 17 0 17 audiology 24 200/26 69 0 69 letter 26 16000/4000 16 16 0

Given the large number of options for solving multiclass problems using ADTrees we designed the following experiments to provide useful comparisons. First, we determine the best multiclass ADTree method by treating the induction as a two-class problem (Table 3). Second, we compare this method with the AdaBoost.MH and the two LADTree methods described in the last section. Generally, it is difficult to compare all of these methods fairly in terms of the number of trees produced. For example, the 1-against-1 method produces J(J−1) trees, 1-against-rest J, random 2 ∗ J, and exhaustive 2J−1 trees. LT1PC 2 produces J trees while AdaBoost.MH and LT induce a single tree. Thus, it can be the case that the number of trees is greater than the number of boosting iterations, for example, the average number of trees produced by 1-against-1 over all nineteen multiclass datasets is 79. Unless otherwise stated, in all tables we compare methods against a fixed number of boosting iterations (10).


169

Table 2. Two-class problems: ADTree vs. LT dataset ADTree(10) LT(10) breast-wisc 95.61 95.65 cleveland 81.72 80.36 credit 84.86 85.04 hepatitis 79.78 77.65 ionosphere 90.49 89.72 labor 84.67 87.5 promoters 86.8 87.3 sick-euthyroid 97.71 97.85 sonar 76.65 74.12 vote 96.5 96.18 +, − statistically significant difference

−

+ + − −

Table 3 allows us to compare the results of each method on an overall basis through the average and on a pair-wise basis through the significance tests. Note that the significance tests are all performed with respect to the first column in the table. On both scales the exhaustive method is the best. As the exhaustive method is not practical for large class datasets we chose the 1-against-1

Table 3. Wrapping two-class ADTree results dataset 1vs1 1vsRest Random Exhaustive iris 95.13 95.33 95.33 95.33 balance-scale 83.94 85.06 + 85.06 + 85.06 + hypothyroid 99.61 99.63 99.64 99.64 anneal 99.01 98.96 99.05 99.19 + zoo 90.38 93.45 + 95.05 + 95.94 + autos 78.48 77.51 77.98 79.99 + glass 75.90 74.33 − 73.79 − 76.76 segment 96.74 95.94 − 95.91 − 96.62 ecoli 83.31 83.96 84.69 + 85.95 + led7 75.40 74.40 76.40 75.60 optdigits 92.49 90.26 − 92.21 − 93.82 pendigits 94.11 91.48 − 86.16 − 89.54 − vowel 47.40 41.13 − 48.48 + 50.65 half-letter 88.71 80.77 − 76.13 − 80.98 − arrhythmia 68.00 66.00 − 66.00 68.00 soybean 89.36 89.10 89.36 NA primary-tumor 46.90 43.36 − 46.90 NA audiology 76.92 80.77 84.62 NA letter 85.98 70.63 − 65.20 − NA average 84.57 83.21 83.46 84.87 +, − statistically significant difference to 1vs1

170


method to compare against LADTrees, as this method is very similar in overall performance. Table 4 compares the “winner” of Table 3 (1-against-1) to AdaBoost.MH and both versions of LADTrees of various sizes. It demonstrates the improvements that can be made by increasing the number of boosting iterations for the single tree methods LT and AdaBoost.MH as they generate tree sizes closer to the number generated by 1-against-1. The 1-against-1 method defeats each of the small tree methods at 10 boosting iterations. But when the number of iterations is increased to 100 tests each, we notice a dramatically different picture: all methods are outperforming the 1against-1 method. Consider the 100 iteration case: 1-against-1 is boosted 10 times but produces J(J−1) trees, which represents an average tree size of 790 (tests). LT and Ad2 aBoost.MH outperform this method on average after 100 iterations (i.e. using trees with 100 tests). Table 4 shows that LT(100) outperforms most of the early datasets (class sizes 3-13) but struggles against two of the later datasets. For soybean 1-against-1 uses a tree of size 1710, and for primary-tumor it uses a tree of size 2310. Perhaps the most remarkable result is for half-letter where 1against-1 using 780 tests has an accuracy of 88.71% whereas LT(100) achieves 92.16% using only 100 tests. Clearly, both on an overall average and on a per dataset basis, AdaBoost.MH and LT are comparable methods. There are no obvious performance differences between these methods at 10 and 100 iterations. Table 4 also compares the two logistic methods. Due to the number of trees used by LT1PC it outperforms LT both on average and on pairwise tests. But these differences seem to disappear as the number of iterations increases: at 10 boosting iterations LT1PC wins on 11 datasets and has 4 losses; at 100 boosting iterations LT1PC has only 4 significant wins and 3 losses.

6

Conclusions

This paper has presented new algorithms for inducing alternating decision trees in the multiclass setting. Treating the multiclass problem as a number of binary classification problems and using the two-class ADTree method produces accurate results from large numbers of trees. Although ADTrees can be merged, the size of the combined tree prohibits its use as a practical method, especially if interpretable models are a requirement. Using AdaBoost.MH for multiclass problems was thought to be problematic. The theoretical objections to this method presented in [10] appear to be based on a mis-interpretation of AdaBoost.MH. Our experimental results demonstrate that this method is competitive with LogitBoost in the multiclass setting, at least for ADTrees. Two new algorithms, LT1PC and LT, for inducing ADTrees using LogitBoost are presented. One method induces a single tree per class, the other a single tree, optimised across all classes. In experimental results comparing these methods to


171

Table 4. LADTree and AdaBoost.MH results dataset 1PC(10) LT(10) MH(10) 1PC(100) LT(100) iris 95.07 94.20 − 94.93 95.13 95.13 balance-scale 88.80 + 84.50 84.21 86.53 + 90.40 hypothyroid 99.49 − 99.59 99.57 − 99.55 − 99.62 anneal 99.44 + 98.50 − 97.41 − 99.62 + 99.66 zoo 92.95 + 94.34 + 94.55 + 92.35 + 94.53 autos 81.12 + 64.57 − 69.92 − 82.71 + 82.43 glass 71.81 − 67.95 − 66.65 − 77.05 75.51 segment 96.68 92.27 − 93.14 − 97.99 + 97.84 ecoli 82.44 84.64 + 84.40 84.27 + 83.54 led7 75.20 77.60 72.80 75.00 73.60 optdigits 91.32 78.63 − 77.69 − 95.77 + 94.94 pendigits 91.65 − 78.53 − 78.24 − 96.74 + 96.51 vowel 39.61 − 34.85 − 34.85 48.05 46.54 half-letter 83.92 − 66.80 − 65.36 − 95.00 + 92.16 arrhythmia 70.00 64.67 64.67 68.67 66.67 soybean 90.43 81.38 − 79.79 − 85.90 83.51 primary-tumor 34.51 − 43.36 42.48 33.63 − 42.48 audiology 80.77 80.77 88.46 76.92 76.92 letter 76.78 − 50.53 − 44.25 − 93.25 + 86.78 average 81.16 75.67 75.44 83.38 83.09 +, − statistically significant difference to 1vs1

+ + + + +

+ + + − −

MH(100) 95.13 90.82 99.63 99.72 94.34 82.69 73.97 97.72 83.99 74.00 94.49 96.00 46.54 91.65 67.33 92.82 45.13 80.77 84.80 83.77

+ + + + − +

+ + + +

−

1-against-1, the best of the wrapper methods, both LADTree methods LT1PC and LT and AdaBoost.MH show significant promise, especially when we consider the relative sizes of the induced trees. From a different point of view one can also argue that the LADTree and AdaBoost.MH methods are the first direct induction methods for multiclass option trees, a hitherto unsolved problem. Previous attempts [4,12] were plagued by the need to specify multiple parameters, and also seemed to contradict each other in their conclusion of why and where in a tree options (i.e. alternatives) were beneficial. Contrary to these attempts, the LADTree and AdaBoost.MH methods have only a single parameter, the final tree size, and automatically add options where they seem most beneficial. A research problem that deserves attention is the determination of the stopping condition T for boosting methods. Freund and Mason [7] use crossvalidation with some success but this method is impractical for large datasets. One possible solution is to use out-of-bag samples to determine if adding new tests will continue to increase performance. This will be a topic of future work. Acknowledgements We would like to thank the anonymous referees for making us re-address the results we had earlier achieved with our first implementation of AdaBoost.MH.

172


This uncovered what appears to be a common misunderstanding of how to implement this method.

References 1. Erin Allwein, Robert Schapire, and Yoram Singer. Reducing multiclass to binary: A unifying approach for margin classifiers. Journal of Machine Learning Research, 1:113–141, 2000. 164 2. C. Blake, E. Keogh, and C. J. Merz. UCI repository of machine learning databases. Technical report, University of California, Department of Information and Computer Science, Irvine, CA, 1998. [www.ics.uci.edu/˜mlearn/MLRepository.html]. 167 3. Leo Breiman. Arcing classifiers. The Annals of Statistics, 26(3):801–849, 1998. 161 4. Wray Buntine. Learning classification trees. Statistics and Computing, 2:63–73, 1992. 171 5. Thomas G. Dietterich. Approximate statistical test for comparing supervised classification learning algorithms. Neural Computation, 10(7):1895–1923, 1998. 167 6. Thomas G. Dietterich and Ghulum Bakiri. Solving multiclass learning problems via error-correcting output codes. Journal of Artificial Intelligence Research, 2:263– 286, 1995. 164 7. Yoav Freund and Llew Mason. The alternating decision tree learning algorithm. In Proc. 16th Int. Conf. on Machine Learning, pages 124–133. Morgan Kaufmann, 1999. 162, 171 8. Yoav Freund and Robert E. Schapire. Experiments with a new boosting algorithm. In Proc. 13th Int. Conf. on Machine Learning, pages 148–156. Morgan Kaufmann, 1996. 161 9. Jerome Friedman. Another approach to polychotomous classification. Technical report, Stanford University, Department of Statistics, 1996. 164 10. Jerome Friedman, Trevor Hastie, and Robert Tibshirani. Additive logistic regression: a statistical view of boosting. The Annals of Statistic, 28(2):337–374, 2000. 161, 162, 163, 165, 170 11. Johannes F¨ urnkranz. Round robin classification. Journal of Machine Learning Research, 2:721–747, 2002. 164 12. Ron Kohavi and Clayton Kunz. Option decision trees with majority votes. In Proc. 14th Int. Conf. on Machine Learning, pages 161–169. Morgan Kaufmann, 1997. 171 13. Robert E. Schapire. Using output codes to boost multiclass learning problems. In Proc. 14th Int. Conf. on Machine Learning, pages 313–321. Morgan Kaufmann, 1997. 164 14. Robert E. Schapire and Yoram Singer. Improved boosting algorithms using confidence-rated predictions. In Proc. 11th Conf. on Computational Learing Theory, pages 80–91. ACM Press, 1998. 161

Possibilistic Induction in Decision-Tree Learning Eyke H¨ ullermeier Department of Mathematics and Computer Science University of Marburg, Germany [email protected]

Abstract. We propose a generalization of Ockham’s razor, a widely applied principle of inductive inference. This generalization intends to capture the aspect of uncertainty involved in inductive reasoning. To this end, Ockham’s razor is formalized within the framework of possibility theory: It is not simply used for identifying a single, apparently optimal model, but rather for concluding on the possibility of various candidate models. The possibilistic version of Ockham’s razor is applied to (lazy) decision tree learning.

1

Introduction

Inductive reasoning – by its very nature – is inseparably connected with uncertainty [4]. To begin with, the data presented to learning algorithms is imprecise, incomplete or noisy most of the time, a problem that can badly mislead a learning procedure. But even if observations are perfect, the generalization beyond that data is still afflicted with uncertainty. For example, observed data can generally be explained by more than one candidate theory, which means that one can never be sure of the truth of a particular model. In fact, the insight that inductive inference can never produce ultimate truth can be traced back at least as far as Francis Bacon’s epistemology. In his Novum Organum1 , Bacon advocates a gradualist conception of inductive enquiry and proposes to set up degrees of certainty. Thus, from experience one may at best conclude that a theory is likely to be true – not, however, that it is true with full certainty. In machine learning and mathematical statistics, uncertainty is often handled by means of probabilistic methods. In Bayesian approaches, for example, the data-generating process is modeled by means of a probability distribution which depends on the true model. Given the data S, a (posterior) probability (density) can thus be assigned to each model M ∈ M, where M is the class of candidate models. The specification of a probability distribution, µ, over that class of models allows one to take the uncertainty related to the learning (prediction) task into account. For example, rather than making a single prediction y0 = M ∗ (x0 ) on the basis of a particular model M ∗ (and a given query x0 ), one can derive a probability Pr(y) = µ({M ∈ M | M (x0 ) = y}) for each potential outcome y. Probabilistic approaches are not always applicable, however, and they do not capture every kind of uncertainty relevant to machine learning. Particularly, this 1

Published in 1620.

T. Elomaa et al. (Eds.): ECML, LNAI 2430, pp. 173–184, 2002. c Springer-Verlag Berlin Heidelberg 2002

174

Eyke H¨ ullermeier

appears to be true for the uncertainty or, say, unreliability connected to heuristic principles of inductive inference such as Ockham’s razor. Such principles usually suggest one particular model M ∗ ∈ M, thereby disregarding the aspect of uncertainty. Our aim in this paper is to alleviate this drawback by means of a possibilistic approach to inductive inference. More specifically, we shall propose a formalization of Ockham’s razor within the framework of possibility theory. In its generalized version, Ockham’s razor specifies the possibility of alternative models rather than selecting one particular model. Section 2 recalls some basic principles of decision tree learning. In Section 3, the possibilistic version of Ockham’s razor is introduced. The application of this generalized principle to classical decision tree learning and to a lazy variant thereof are discussed, respectively, in Sections 4 and 5. Finally, Section 6 presents some experimental results.

2

Decision Tree Learning

We proceed from the common framework for learning from examples: X denotes the instance space, where an instance corresponds to the description x of an object in attribute–value form. That is, each object x is characterized through attribute values αı (x) ∈ Aı , 1 ≤ ı ≤ k, where Aı = dom(αı ) is the (finite) domain of the ı-th attribute αı ; the set of all attributes is denoted A. L = {λ1 , . . . , λm } is a set of labels, and x, λx is called a labeled instance or an example. S denotes a sample that consists of n labeled instances xı , λxı , 1 ≤ ı ≤ n. Finally, a new instance (query) x0 ∈ X is given, whose label λx0 is to be estimated. The basic principle underlying most decision tree learners, well-known examples of which include the ID3 algorithm [12] and its successor C4.5 [13] as well as the CART system [2], is that of partitioning the set of given examples, S, in a recursive manner. Each inner node η of a decision tree τ defines a partition of a subset Sη ⊂ S of examples assigned to that node. This is done by classifying elements x ∈ Sη according to the value of a specific attribute α. The attribute is selected according to a measure of effectiveness in classifying the examples, thereby supporting the overall objective of constructing a small tree. A widely applied “goodness of split” measure is the information gain, G(S, α), which is defined as the expected reduction in entropy (impurity) which results from partitioning S according to α: |Su | . ent(Su ), (1) G(S, α) = ent(S) − |S| u∈dom(α)

. where Su = { x, λx ∈ S | α(x) = u}. The entropy of a set S is given by . ent(S) = −qλ · log2 (qλ ),

(2)

λ∈L

. where qλ = card({ x, λx ∈ S | λx = λ}) · card(S)−1 . Besides, a number of other selection measures have been devised. See [11] for an empirical comparison of such measures.

Possibilistic Induction in Decision-Tree Learning

175

Since decision tree induction is by now a well-known method, we shall restrict ourselves to a concise exposition of the basic algorithm underlying ID3 and C4.5. This algorithm derives a decision tree in a top-down fashion by means of the following heuristic (greedy) strategy: – The complete set of training samples, S, is assigned to the root of the tree. – A node η becomes a leaf (answer node) of the tree if all associated samples Sη belong to the same class λ. In this case, η is assigned the label λ.2 – Otherwise, node η becomes a decision node: It is split by partitioning the associated set Sη of examples. This is done by selecting an attribute (among those that have not been used so far) as described above and by classifying the samples x ∈ Sη according to the values α(x). Each element of the resulting partition defines one successor node. Once the decision tree has been constructed, each path can be considered as a rule. The antecedent of a rule is a conjunction of conditions of the form αı (x) = uı , where αı is an attribute and uı ∈ dom(αı ) a specific value thereof. The conclusion part determines a value for the class variable. New examples are then classified on the basis of these rules, i.e. by looking at the class label of the leaf node whose attribute values match the description of the example.

3

A Possibilistic Version of Ockham’s Razor

3.1

Possibility Theory

Here we briefly review some aspects of possibility theory without going into technical detail. Possibility theory [7] is an alternative calculus for modeling and processing uncertainty or, more generally, partial belief. Possibility theory makes a distinction between the concepts of certainty (necessity) and plausibility (possibility) of an event. As opposed to probability theory, it does not claim that the confidence in an event is determined by the confidence in the complement of that event. Consequently, possibility theory is non-additive. In fact, the basic axiom of possibility theory involves the maximum-operator rather than the arithmetic sum: Π(A ∪ B) = max Π(A), Π(B) . In plain words, the possibility of the union (disjunction) of two events A and B is the maximum of the respective possibility of the individual events. A possibility distribution Π on 2X (satisfying Π(X) = 1 and Π(∅) = 0) is related to a possibility measure . π : X → V via Π(A) = supx∈A π(x). V is a totally ordered scale which is usually taken as the unit interval [0, 1]. However, V can also be a purely qualitative scale, in which case π(x) < π(y) simply means that y is more plausible than x. . A so-called necessity measure N , defined by N (A) = 1 − supx∈X\A π(x) for all A ⊆ X, is associated measure Π. A necessity measure satisfies with a possibility N (A ∩ B) = min Π(A), Π(B) . 2

In the case of noisy data, it may happen that all attributes have already been used along the path from the root of the tree to η, though not all samples have the same label.

176

Eyke H¨ ullermeier

Where does a possibility distribution come from? Originally, the idea of Zadeh [14] was to induce a possibility distribution from vague linguistic information, as represented by a fuzzy set. For example, the uncertainty related to the vague statement that “x is a small positive integer” translates into a distribution which lets x = 1 appear fully plausible (π(1) = 1), whereas, say, 5 is regarded as more or less plausible (π(5) = 1/2) and 10 as impossible (π(10) = 0).3 More generally, a possibility distribution can be induced by a flexible constraint: Consider a set A of alternatives and suppose information about an element a0 ∈ A of interest to be given, expressed in the form of a constraint. Usually, a constraint completely excludes some alternatives a ∈ A and can hence be identified with a subset C ⊆ A of still admissible candidates. A flexible constraint may exclude alternatives to a certain extent. A possibility degree π(a) is then understood as the plausibility that remains of alternative a given the constraint. Note that two constraints are naturally combined by intersection. The possibilistic counterpart to this kind of conjunctive operation is the (pointwise) minimum, i.e. the combination of two possibility distributions π1 and π2 into a new distribution π : x → min{π1 (x), π2 (x)}. In the following section, we shall look at Ockham’s razor as a flexible constraint. More generally, our view of a heuristic inductive reasoning principle is that of a constraint which may exclude a model (from the class of candidate models) to a certain degree. 3.2

Ockham’s Possibilistic Razor

According to Ockham’s razor, a simple model is to be preferred to a more complex one. In the context of decision trees, simplicity is usually equated with size and, hence, one tries to find the smallest tree among those consistent with the data. Note that the heuristic divide and conquer algorithm outlined in Section 2 only finds an approximation to this tree. Of course, what we actually desire is the true model, and the assumption underlying Ockham’s razor is that a simple model is more likely to be true than a complex one if both explain the data equally well. Even though this assumption is not very well settled from a theoretical point of view it is intuitively appealing and has proved its worth in practice [5]. Now, consider two decision trees τ ∗ and τ , where τ is only slightly more complex than τ ∗ . In such a case, one would generally not completely reject τ . Indeed, when taking the “more likely to” in the above formulation of Ockham’s razor seriously, then τ should be assigned a certain degree of possibility as well. This, in turn, should be taken into account when making inferences about new objects. More generally, this possibilistic interpretation of Ockham’s razor suggests to define a possibility distribution πM over the class of models M, where the possibility πM (τ ) depends on the simplicity of τ in comparison to the simplicity of 3

The specific definition of π clearly depends on the context.


the simplest (and hence most plausible4 ) model τ ∗ : 0 if τ is not consistent . πM (τ ) = πM (τ | S) = , f (|τ |, |τ ∗ |) otherwise

177

(3)

where |τ | denotes the complexity of τ (a model τ is consistent if τ (x) = λx for all instances x, λx ∈ S). A possibilistic prediction, that is a possibility distribution over the class of labels L, can then be obtained by applying the well-known extension principle: . (4) πL (λ) = πL (λ | x0 ) = sup{πM (τ ) | τ (x0 ) = λ}. Needless to say, the computation of the possibility measure (3) is generally not tractable, as it requires the consideration of all (consistent) models. Apart from that, one will often not be interested in the possibility degrees of all models, but only in those models with a high degree of possibility. In the following section, we shall propose a heuristic approach which is a generalization of recursive partitioning: The problem of inducing a decision tree is decomposed into sub-problems in a hierarchical way, and the possibility of a tree τ is derived from the possibilities of its sub-trees.

4

Generalized Decision Tree Learning

Recall that the selection of an attribute in decision tree learning is made on the basis of a measure such as (1). Now, suppose that G(Sη , α∗ ) is quite large for the apparently optimal attribute α∗ , whereas G(Sη , α) is rather small for all remaining attributes. Taking the adequacy of the decision tree approach for granted, one can then be quite sure that α∗ is indeed the “correct” selection (problem decomposition) at this place. However, if G(Sη , α) is close to G(Sη , α∗ ) for some alternative attribute α, it is reasonable to say that α appears possible to a certain extent as well. More specifically, one might define a degree of possibility πA (α | Sη ) for each attribute α on the basis of the set of measures {G(Sη , α) | α ∈ A}, for example . (5) πA (α) = πA (α | Sη ) = max 0, 1 − c G(Sη , α∗ ) − G(Sη , α) , where c > 0. In order to guarantee a meaningful interpretation of the difference G(Sη , α∗ ) − G(Sη , α), the measure G(·) is assumed to be normalized such that 0 ≤ G(·) ≤ 1, with 1 being the best evaluation. This idea suggests the following generalization of the algorithm for decision tree induction: At a node η, a recursive partitioning is not only made for the best attribute a∗ but rather for all attributes in the set . A∗η = {α ∈ Aη | πA (α) > ∆} (6) 4

Letting πM (τ ∗ ) = 1 for a least one τ ∗ ∈ M means that at least one model is fully plausible. This can be seen as a kind of closed world assumption. More generally, one might allow that πM (τ ) < 1 for all τ ∈ M, suggesting that none of the candidate models is fully plausible.

178

Eyke H¨ ullermeier

of candidates whose possibility exceeds a lower threshold ∆. More precisely, a possibilistic branching is realized as follows: For each attribute α ∈ A∗η and each value u ∈ dom(α), one outgoing edge is added to η. This edge is marked with the test α = u and the possibility degree πA (α). Thus, one obtains a possibilistic tree or, say, a meta-tree T in which an instance can branch at a node in different directions. T actually consists of several ordinary trees τ . In fact, an ordinary tree is obtained by retaining at each (meta-)node η only those edges associated with a single attribute and by deleting all other edges. The possibility of a tree, πM (τ ), is determined by the smallest possibility of its edges. 4.1

Classification with Possibilistic Trees

Now, suppose that a new query x0 is to be classified. Given the possibility distribution πM (·) as defined above, a possibilistic prediction of the label λx0 can be derived from (4). However, a more efficient approach is to propagate possibility degrees in the meta-tree T directly. To this end, define possibility η for nodes η in a recursive way as follows: If η is a leaf node, distributions πL η then πL is defined by 1 if η is labeled with λ η . πL : λ → 0 otherwise Otherwise, let η1 , . . . , ηr be the successor nodes of η, and suppose the edge leading from η to ηı be marked with the possibility degree pı . The distribution associated with η is then given by ηı (λ), pı }. πL : λ → max min{πL 1≤ı≤r

(7)

The possibility distribution πL = πL (· | x0 ) is defined to be the possibility disη0 tribution πL associated with the root η0 of the meta-tree. Proposition 1. The propagation of possibility degrees in the meta-tree yields the same possibilistic prediction πL (· | x0 ) as the extension principle (4). Proof. Let πL be the possibility distribution derived from the propagation of possibility degrees in the meta-tree T . Moreover, consider a label λ ∈ L and let p = πL (λ). If p = 0 then none of the leaf nodes in T is labeled with λ, and the proposition is obviously correct. Now, let p > 0. The definition (7) of distributions associated with nodes entails the existence of a path ρ∗ = (η1 , . . . , ηk ) in T such that the following holds: (1) η1 is the root of τ and ηk is a leaf node with label λ. (2) The possibility π(ρ∗ ) of the path ρ∗ , that is the minimum of the possibility degrees assigned to the edges (ηı , ηı+1 ), 1 ≤ ı < k, is given by p. Moreover, π(ρ) ≤ p for all other paths ρ in the meta-tree whose leaf nodes are labeled with λ. Now, it is easily verified that the path ρ∗ can be completed to an ordinary decision tree τ such that πM (τ ) = d. In fact, at each node η in the meta-tree T there is an attribute α such that all edges associated with that attribute are


179

labeled with the possibility degree 1. Thus, the path ρ∗ can be extended to a tree τ such that each edges of τ which is not an edge of ρ∗ is labeled with a possibility degree of 1. Therefore, πM (τ ) = p, which means that the possibility of λ according to (4) is at least p. Clearly, (4) cannot be larger than p, since this would imply the existence of a tree τ which assigns x0 the label λ and whose edges all have possibility degrees larger than d. This tree therefore contains a path ρ whose leaf node is labeled with λ and such that π(ρ) > p, a contradiction to the definition of ρ∗ . Therefore, the possibility of λ according to (4) is also given by d. ✷ Using the classification scheme outlined above, a single estimated class label λ0 as predicted by an ordinary decision tree is replaced by a prediction in the form of a possibility distribution πL over the set of labels. This distribution is normalized in the sense that maxλ∈L πL (λ) = 1. Note that the label λ∗0 with πA (λ∗0 ) = 1 is unique unless there is an exact equivalence G(Sη , αı ) = G(Sη , α ) for a node η and two attributes αı = α . If λ∗0 is unique, it is just the label predicted by the classical approach to decision tree induction. The distribution πA reflects the uncertainty related to the classification: λ∗0 is the most plausible classification and will generally be chosen if a definite decision must be made. However, there might be further possible candidates as well, and the related possibility degrees indicate the reliability of λ∗0 . Formally, reliability is reflected by the necessity degree of λ0 , given by 1 − maxλ =λ∗0 πL (λ): If there is at least one other label with a rather high degree of possibility, the situation is ambiguous. A classification (on the basis of a decision tree) might then be rejected. More generally, one might take action on the basis of a set-valued prediction including the maximally plausible labels, or take this set as a point of departure for the acquisition of further information. The approach proposed here is related to other extensions of decision tree learning. Especially, the idea of option decision trees [3,9], which also provide a compact representation of a class of candidate decision trees, is worth mentioning in this connection. There are, however, some important differences between the two methods. For example, the outcomes at an option node are combined to a unique choice, e.g. by means of a majority vote. As opposed to this, our approach considers different choices with different degrees of possibility. 4.2

Alternative Aggregation Procedures

Consider a meta-tree T and let P = Px0 denote the class of paths ρ in T that are matched by the new query x0 (where x0 matches a path if it satisfies all tests αı (x0 ) = uı along that path). In agreement with the common max-min calculus of possibility theory we have defined the possibility of a path ρ = (η1 , . . . , ηk ) as . πP (ρ) = min poss((ηı , ηı+1 )), 1≤ı 0 and 0 < δ < 1, their algorithm returns with confidence 1 − δ, in time polynomial in n, L, |Σ|, 1δ , 1 , a PST whose per symbol Kullback-Leibler distance from the target is at most . The PST inference algorithm is presented on Figure 2. Given a sequence s = x0 , · · · , xi , we note suf (s) = x1 , . . . , xi the largest suffix of s different from s. The algorithm starts from a 0th order Markov chain (line 2) and with S, the set of suffixes to be examined, containing all the symbols with probability larger than a threshold set to Pmin (line 3). Then, for each element s in S, if there exists a symbol for which the probability conditioned by s is significantly different from the probability conditioned by the suffix of s (line 4), then the node labeled by s is added to the tree, and also all the nodes needed to go to this node from the node in the tree labeled by the largest suffix of s. If the length of s is strictly smaller than the maximum order of the tree, then all the sequences built from s added with a symbol σ, such that the probability of the sequence sσ is larger the threshold Pmin are added to S (line 10). The last step of the algorithm is a very simple smoothing procedure. The maximum likelihood probability estimator is modified so that no symbol is predicted with probability 0 whatever its suffix is (line 13).

188

Christopher Kermorvant and Pierre Dupont

However, the smoothing procedure using the modified maximum likelihood estimator has two main drawbacks. First, the same constant value γmin is added to every probability, whatever the observed frequency and the probability of the event are. Second, the same floor probability γmin is assigned to all unseen events, whatever their suffix is. Many other smoothing procedures, which do not suffer from these problems, have been proposed. We present one of them in the next section. 2.2

Back-Off Smoothing of Markov Chains

When using Markovian models in real applications, even if a large amount of data is available to estimate the model, the problem of predicting events which were not observed during the estimation procedure occurs. This is particularly true for high order Markovian models since the number of possible contexts for a kth order Markovian model is exponential in k. The probability of a large number of events are estimated on only a few occurrences, leading to poor probability estimation. Many rare but possible events are not observed, and are wrongly estimated with a null probability. The problem of predicting unseen events, also known as the zero-frequency problem, is due to the fact that the maximum likelihood estimator attributes the whole probability mass to the events seen during the estimation. Several solutions have been proposed to this problem : succession laws [12], linear interpolation of the maximum likelihood estimator with another estimator, such as an a priori distribution [8] or a more general distribution [7], discounting of a certain amount of the probability mass of seen events using the Turing-good formula [9] or absolute discounting [13]. When using discounting, the discounted probability mass is redistributed to all unseen events according to another probability distribution. This is the back-off smoothing methods, proposed by Katz [9]. For Markov chains, the back-off smoothing is based on the following idea : discount a certain amount dC from the probability mass of events which have been observed in a context of length k and redistribute this amount to all unseen events according to their probability in a context of length k−1. This probability can in turn be recursively smoothed. Formally, recalling the notation introduced in the previous section, if σ is a symbol and s a suffix (context), we have : c(s,σ)−d C if c(s, σ) > 0 σ∈Σ c(s,σ) P (σ|s) = α(s)β(s, σ) otherwise

P

where c(s, σ) is the number of times σ was seen after the suffix s and dC is the discount parameter, which may depend on c(s, σ), α(s) is a normalization factor and β(s, σ) is the back-off distribution, generally estimated on shorter suffixes. This distribution can in turn be smoothed, inducing a recursive process which ends, in the worst case, with the unconditional probability of the symbol P (σ). In this case, the back-off distribution is used only if the main distribution is null (shadowing). Kneser and Ney [10] showed that using the back-off distribution even if the main distribution is not null (non-shadowing) leads to a better

Improved Smoothing for Probabilistic Suffix Trees

model. We have then: P (σ|s) =

Pc(s,σ)−d c(s,σ) + α (s)β(s, σ)

if c(s, σ) > 0

α (s)β(s, σ)

otherwise

C

σ∈Σ

189

where α (s) is a normalization factor. This method is also named non-linear interpolation, since it can be defined as c(s, σ) − d C , 0 + α (s)β(s, σ) P (σ|s) = max σ∈Σ c(s, σ) if we suppose that dC ≤ c(s, σ) for all c(s, σ) > 0. The normalization factor is dC then α (s) = σ∈Σ | c(s,σ)>0 . Kneser and Ney propose to estimate σ∈Σ c(s,σ) the back-off probability in the following way :

P

c(•, suf (s), σ) σ∈Σ c(•, suf (s), σ)) with c(•, suf (s), σ) = {σ | c(σ , suf (s), σ) > 0}. This estimation is not based on the observed frequency of the sequence (s, σ) but on the number of different contexts in which σ has been observed after suf (s). The back-off probability can also be null, leading to a recursive smoothing using the same formula. Therefore, we see that using a recursive smoothing on a kth order Markovian model leads to build a variable order Markov chain. There are three differences between a variable order Markov chain build by recursive Kneser-Ney back-off smoothing (denoted KN-chain) and a variable order Markov chain represented by a PST infered by Ron’s algorithm: β(s, σ) =

– there is no pruning in KN-chain : if a sequence of length lower than the maximum order of the KN-chain is observed, its probability is estimated and stored, whereas in PST the estimated probability is stored only if it is above a threshold (Pmin ). For a given maximum order, PST may have less parameters than KN-chains. – in KN-chain, a different and enhanced estimation scheme is used for the backoff probability estimation, whereas in PST, for all the orders, probability estimation is based on the modified maximum likelihood estimator. – in KN-chain, both the enhanced estimation scheme and the modified maximum likelihood estimator are used (non-shadowing), whereas in PST, only one modified maximum likelihood estimator is used. In the next section, we show that KN-chains significantly outperform PST on a protein domain detection task.

3

Application to Protein Domains Detection

Many databases have been created to gather information concerning proteins. Researchers can find in these databases not only the amino-acid sequence of proteins but also information about their functions, structure, related diseases and

190


bibliographical pointers. These databases are used to help the analysis of newly sequenced proteins, for which no function or structure is known yet. They serve as a basis for learning models which are used to detect sub-sequences (called domains or motifs) which are known to be related to a particular biochemical function. Such models range from complex probabilistic models based on hidden Markov models [11,5] to purely syntactic models, like regular expressions, describing characteristic sub-sequences [1]. However, since the databases are constantly increasing and updated, the learning procedure of these models must be easy and of low complexity. 3.1

Protein Domains Detection with Variable Markov Chains

Automatic analysis of newly sequenced proteins, for which neither structure nor biochemical functions are known yet, is now very important since the number of newly sequenced proteins is increasing daily. To a certain extend, hypotheses concerning the function of a protein can be made by searching, in its amino-acid sequence, sub-sequences which are known to be related to a function in other proteins. Many of such sub-sequences, called domains, have been identified and are stored in databases like PFAM [15]. However, the sequence of a given domain is not constant through species. Substitutions, deletions and insertions occur, which make domain detection more complex than a simple exact sub-sequence detection. Domain models, like HMM[5], are trained on theses sub-sequences and used to detect domains in complete protein sequences. Variable order Markov chains may also be used to detect domains in protein sequences [3]. A variable order Markov chain is associated to each domain to be detected and is estimated on a set of examples of such domain. Then the likelihood of a new protein sequence given a domain model is related to the presence or not of the corresponding domain in the protein. A high likelihood is a sign of a probable presence of the corresponding domain in the protein. 3.2

Experimental Setup

We used two databases to test our models : the SWISSPROT database [2] which contains protein sequences from several living organisms and the PFAM database [15] which contains alignments of functional domains, grouped in families, extracted from SWISSPROT with a semi-automatic procedure. We labeled the SWISSPROT sequences with the name of the domains they contain, according to PFAM families. In order to compare with recently publish results [6,3], we used PFAM release 1.0. This release contains 22307 domains grouped in 175 families. 3.3

Training the Models

For each domain family, we estimated the models on 80% of the domain sequences extracted from the alignments available in PFAM. We trained probabilistic suffix


191

Kneser-Ney Smoothing PST Maximal order 0 1 2 3 4 20 Correct detection rate 13.9 53.0 81.3 89.5 90.0 85.8 Number of parameters 2, 2 × 101 4, 0 × 102 3, 3 × 103 9, 9 × 103 1, 8 × 104 5, 1 × 104

Fig. 3. Correct detection rate on the complete SWISSPROT database and number of parameters for variable order Markov chains with Kneser-Ney smoothing and PST tree with software and parameters given as optimal by Bejerano [3]. The maximal order of the PST is 20. We also trained variable order Markov chains with KneserNey smoothing, with maximum order ranging from 0 to 4. 3.4

Testing the Models

All the models were tested for domain detection on the protein sequences of the SWISSPROT database corresponding to the complete PFAM database. In order to measure a correct detection rate, we used the iso-point detection criterion [3,6]. For each family model, an iso-point is computed on the complete SWISSPROT sequences set. The iso-point is defined as the value v for which the number of protein sequences not containing the domain with a likelihood above v is equal to the number of protein sequences containing the domain with a likelihood under v. For a given model, a sequence containing the domain with a likelihood above the iso-point is considered correctly detected. The correct detection rate is defined as the ratio of the number of proteins correctly detected on the number of proteins containing the domain. Note that in order to compute the iso-point, the likelihood of each sequence is normalized by its length. 3.5

Results

Figure 3 shows the correct detection rate on all the SWISSPROT database sequences and the number of parameters for PST and variable order Markov chains with Kneser-Ney smoothing. Smoothed variable order Markov chains outperform PST as soon as the maximum order of the chains is greater or equal to 3. As from the 4th order Markov chain and up to the the 9th order Markov chain, the detection rate is stationary. Considering the domain detection problem as a binary classification problem (“does a sequence contain a given domain or not”), the performance difference between variable 4th order Markov chains and PST was tested with a McNemar test [4]. The H0 hypothesis “the variable 4th order Markov chains and PST have the same classification performance” was rejected (p-value< 10−15 ). The performance difference is thus significant. Figure 4 shows the detection rate on the part of the SWISSPROT database corresponding to the PFAM domains which were not used for training (named SWISSPROT test set). Results are given when the size of the training set is varying from 20% to 100%. Even on small training set, the 4th-order smoothed variable order Markov chains outperform PST.

192


100 0th 1st 2nd 4st

Correct detection rate

80

KN-chain KN-chain KN-chain KN-chain PST-20

60

40

20

0 0

20

40

60 % of traning set used

80

100

120

Fig. 4. Learning curve: correct detection rate on the SWISSPROT test set versus the size of the PFAM training set for Markov chains with Kneser-Ney smoothing with order ranging from 0 to 4 (Oth KN-chain to 4th KN-chain) and PST with maximum order 20 (PST20 )

100

3rd KN 80

4th KN

2nd KN PST-10

Correct detection rate

PST-3

PST-20

PST-2

60

1st KN

40

PST-1

20 0th KN

0 10

100

1000 Number of parameters (log scale)

10000

100000

Fig. 5. Detection rate on the SWISSPROT test set versus the number of parameters needed for Markov chains with Kneser-Ney smoothing with order ranging from 0 to 4 (Oth KN-chain to 4th KN-chain) and PST with maximum order ranging from 1 to 20 (PST1 to PST20 )


193

Finally, figure 5 shows the detection rate with respect to the number of parameter needed by the model. The 3rd-order and 4th-order Markov chains outperform PST while needing significantly less parameters.

4

Conclusion

We have shown that PST and smoothed Markov chains can be seen as equivalent variable order Markov models but for the smoothing technique. As the quality of the back-off technique as been shown to be important for Markov chain in other application domains, we proposed to enhance the smoothing technique used in PST by using a non-shadowing back-off smoothing to lower order Markov chain estimated as proposed by Kneser and Ney [10]. With this improved smoothing, we showed that the maximum order of the Markov chain can be drastically reduced, with a performance increase on a protein domain detection task. By reducing the maximum order of the Markov chain, we also reduce the number of parameters needed.

References 1. A. Bairoch. PROSITE: A dictionary of sites and patterns in proteins. Nucleic Acids Research, 19:2241–2245, 1991. 190 2. A. Bairoch and R. Apweiler. The SWISS-PROT protein sequence data bank and its new supplement TrEMBL. Nucleic Acids Res., 24:21–25, 1996. 190 3. Gill Bejerano and Golan Yona. Variations on probabilistic suffix trees: statistical modeling and prediction of protein families. Bioinformatics, 17(1):23–43, Jan 2001. 185, 190, 191 4. Thomas G. Dietterich. Approximate statistical test for comparing supervised classification learning algorithms. Neural Computation, 10(7):1895–1923, 1998. 191 5. S. Eddy. HMMER user’s guide: biological analysis using profile hidden Markov models. Department of Genetics, Washington University School of Medecine, 1998. http://hmmer.wustl.edu/. 190 6. E. Eskin, W. Grundy, and Y. Singer. Protein family classification using sparse markov transducers. In Proc. Int. Conf. on Intelligent Systems for Molecular Biology, August 2000. 190, 191 7. F. Jelinek and R. Mercer. Interpolated estimation of markov source parameters from sparse data. In E. Gelsema and L. Kanal, editors, Pattern recognition in practice, pages 381–397, Amsterdam, 1980. North-Holland. 188 8. W. E. Johnson. Probability : deductive and inductive problems. Mind, 41:421–423, 1932. 188 9. Slava M. Katz. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Trans. on Acoustics, Speech and Signal Processing, ASSP-35(3):400–401, March 1987. 188 10. R. Kneser and H. Ney. Improved backing-off for M-gram language modeling. In Proc. Int. Conf. on Acoustics, Speech and Signal Processing, pages 181–184, Detroit, MI, May 1995. 188, 193 11. A. Krogh, M. Brown, I. Mian, K. Sjolander, and D. Haussler. Hidden Markov models in computational biology: Applications to protein modeling. Journal of Molecular Biology, 235:1501–1531, 1994. 190

194


12. G. Lidstone. Note on the general case of the Bayes-Laplace formula for inductive or a posteriori probabilities. Trans. Fac. Actuar., 8:182–192, 1920. 188 13. Hermann Ney, Ute Essen, and Reinhard Kneser. On structuring probabilistic dependencies in stochastic language modelling. Computer Speech and Language, 8:1–38, 1994. 188 14. D. Ron, Y. Singer, and N. Tishby. The power of amnesia. In Jack D. Cowan, Gerald Tesauro, and Joshua Alspector, editors, Advances in Neural Information Processing Systems, volume 6, pages 176–183. Morgan Kaufmann Publishers, Inc., 1994. 187 15. E. L. L. Sonnhammer, S. R. Eddy, and R. Durbin. Pfam : a comprehensive database of protein domain families based on seed alignments. Proteins, 28(3):405–420, 1997. 190

Collaborative Learning of Term-Based Concepts for Automatic Query Expansion Stefan Klink, Armin Hust, Markus Junker, and Andreas Dengel German Research Center for Artificial Intelligence (DFKI, GmbH) P.O. Box 2080, 67608 Kaiserslautern, Germany {stefan.klink,armin.hust,markus.junker, andreas.dengel}@dfki.de http://www.dfki.de/klink

Abstract. Information Retrieval Systems have been studied in Computer Science for decades. The traditional ad-hoc task is to find all documents relevant for an ad-hoc given query but the accuracy of adhoc document retrieval systems has plateaued in recent years. At DFKI, we are working on so-called collaborative information retrieval (CIR) systems which unintrusively learn from their users search processes. In this paper, a new approach is presented called term-based concept learning (TCL) which learns conceptual description terms occurring in known queries. A new query is expanded term by term using the previously learned concepts. Experiments have shown that TCL and the combination with pseudo relevance feedback result in notable improvements in the retrieval effectiveness if measured the recall/precision in comparison to the standard vector space model and to the pseudo relevance feedback. This approach can be used to improve the retrieval of documents in Digital Libraries, in Document Management Systems, in the WWW etc.

1

Introduction

With the explosive growth of information on the Internet and Digital Libraries, an acute problem has raised called information overload. Typical search engines index billions of pages across a variety of categories, and return results ranked by expected topical relevance. But only a small percentage of these pages may be of a specific interest. Nowadays, there is an acute need for search engine technology to help users exploit such an extremely valuable resource. In weighted Information Retrieval (IR) the number of retrieved documents is related to the number of appropriate search terms. Retrieval with short queries is typical in Web search [6], but it is much harder as compared to retrieval with long queries. This is because shorter queries often provide less information for retrieval. Modern IR systems therefore integrate thesaurus browsers. They help to find additional search terms [13]. But the keywords used in short queries are not always good descriptors of T. Elomaa et al. (Eds.): ECML, LNAI 2430, pp. 195-207, 2002.  Springer-Verlag Berlin Heidelberg 2002

196

Stefan Klink et al.

contents. Nevertheless, most existing search engines still rely solely on the keywords contained in queries to search and rank relevant documents. This is one of the key reasons that affect the precision of the search engines. In many cases, the answer documents are not relevant to the user information need, although they do contain the same keyword as the query. Another problem which is typical for the Web and for Digital Libraries is that the terminology used in defining queries is often different to the terminology used in the representing documents. Even if some users have the same information need they rarely use the same terminology in their queries. Many intelligent retrieval approaches [2, 8, 12] have tried to bridge this terminological gap. Research on automatic query expansion (or modification) was already under way before the 60’s when initial requests were enlarged in the grounds of statistical evidence [14]. The idea was to obtain additional relevant documents through expanded queries based on the co-occurrence of the terms. However, this kind of automatic query expansion has not been very successful. The retrieval effectiveness of the expanded queries was often not greater than, or even less that the effectiveness of the original queries [10, 11, 16]. One idea involves the use of a relevance feedback environment where the system retrieves documents that may be relevant to a user’s query. The user judges the relevance of one or more of the retrieved documents and these judgments are fed back to the system to improve the initial search result. This cycle of relevance feedback can be iterated until the user is satisfied with the retrieved documents. In this case, we can say that the more feedback is given to the system the better is the search effectiveness of the system. This behavior is verified by [1]. He has shown that the recall-precision effectiveness is proportional to the log of the number of relevant feedback documents. But in a traditional relevance feedback environment the user voted documents are appropriate to the complete query. That means that the complete query is adapted to the users needs. If another user has the same intention but uses a different terminology or just one word more or less in his query then the traditional feedback environment doesn’t recognize any similarities in these situations.

2

Query Expansion

The crucial point in query expansion is the question: Which terms (or phrases) should be included in the query formulation? If the query formulation is to be expanded by additional terms there are two problems that are to be solved namely how are these terms selected and how are the parameters estimated for these terms. Many terms used in human communication are ambiguous or have several meanings [12]. But in most cases these ambiguities are resolved automatically without noticing the ambiguity. The way this is done by humans is still an open problem of psychological research, but it is almost certain, that the context in which a term occurs plays a central role. Most attempts at automatically expanding queries failed to improve the retrieval effectiveness and it was often concluded that automatic query expansion based on statistical data was unable to improve the retrieval effectiveness substantial [11].

Collaborative Learning of Term-Based Concepts for Automatic Query Expansion

197

But this could have several reasons. Term-based query expansion approaches are mostly using hand-made thesauri or just plain co-occurrence data. They do not use learning technologies for the query terms. On the other hand, those who use learning technologies (Neural Networks, Support Vector Machines, etc.) are query-based. That means these systems learn concepts (or additional terms) for the complete query. The vital advantage of using term-based concepts and not learning the complete query is that other users can profit from the learned concepts. A statistical evaluation of internet logging files has shown that the probability that a searcher uses exactly the same query than a previous searcher is much lower then the probability that parts of the query (phrases or terms) occurs in other queries. So, even if a web searcher never used the given search term, the probability that another searcher had used it is very high and then he can profit from the learned concept.

3

Traditional Document Retrieval

The task of traditional document retrieval is to retrieve documents which are relevant to a given query from a fixed set of documents. Documents as well as queries are represented in a common way using a set of index terms (called terms from now on). Terms are determined from words of the documents in the database, usually during pre-processing phases where some noise reduction procedures are incorporated, e.g. stemming and stop-word elimination. In the following, a term is represented by ti (1 ≤ i ≤ M) and a document by dj (1 ≤ j ≤ N), respectively, where M is the number of different terms and N is the number of documents in the data base. 3.1 Vector Space Model One of the most popular and indeed the simplest retrieval model is the vector space model (VSM) [2]. In the VSM, each document dj is represented as a M dimensional vector dj = (w1j, …, wMj)T, 1 ≤ j ≤ N

(1)

where T indicates the transpose, wij is a weight of a term ti in a document dj. A query is likewise represented as qk = (w1q, …, wMq)T, 1 ≤ k ≤ L

(2)

where wiq is a weight of a term ti in a query qk and L is the number of queries contained in the document collection (e.g. in the web-log file). The weights above can be processed in various ways. In our approach, we use the standard normalized tf · idf weighting scheme [14] defined as follows: wij = tfij * idfi

(3)

198

Stefan Klink et al.

where tfij is the weight calculated using the term frequency fij and idfi is the weight calculated using the inverse of the document frequency. The result of the retrieval is represented as a list of documents ranked according to their similarity to the given query. The similarity sim(dj, qk) between a document dj and a query qk is measured by the standard cosine of the angle between these M dimensional vectors dj and qk: sim(dj, qk) =

d Tj qk dj qk

(4)

where · is the Euclidean norm of a vector. In the case that the vectors are already normalized (e.g. have a unit length) the similarity is just the dot product between the two vectors. The VSM is one of the methods we applied to compare with our own methods. 3.2 Pseudo-Relevance-Feedback Everybody would agree that documents my be relevant to a query even if they do not share any word with the query. Unfortunately, the standard VSM will always return zero similarity in this case. So-called query expansion techniques try to overcome this problem by expanding the user given query q to a new enriched query q’ which hen used in the standard VSM. A very well-known technique is the pseudo relevance feedback (PRF) [8]. PRF enriches the original query q by the terms of the top-ranked documents with respect to q. We are using a variation of PRF described in [7]: Let ÷ be a set of document vectors for expansion given by sim(d +j ; q ) maxi sim(di ; q )

÷ =  d +j   



 

≥θ 

(5)

where q is the users query vector and θ is a similarity threshold. The sum ds of the document vectors in ÷: ds =

∑ d +j

(6)

d +j ∈÷

can be considered as enriched information about the original query. Within the VSM, this sum of documents is again a vector which contains the weights of all terms of the summed documents. Hence, all terms of the documents in ÷ are used to expand the users query. The expanded query vector q’ is obtained by q’ =

q q

+ α

ds ds

(7)

where α is a parameter for controlling the weight of the newly incorporated terms. Finally, the documents are ranked again according to the similarity sim(dj, q’) to the expanded query.


4

199

Term-Based Concept Learning (TCL)

A problem of the standard VSM is that a query is often too short to rank documents appropriately. To cope with this problem, the approach is to enrich the original query with terms occurring in the documents of the collection. Our method uses feedback information and information globally available from previous queries. Feedback information in our environment is available within the ground truth data provided by the test document collections. The ground truth provides relevance information, i.e. for each query a list of relevant documents exists. Relevance information for each query is represented by a N dimensional vector: T

rk = (r1k, …, rNk)

,

 1 if docment dj is relevant to query k rjk =  0  if docment d is not relevant to query q j k

(8)

In contrast to traditional pseudo relevance feedback methods, where the top j ranked documents are assumed to be relevant and then their terms are incorporated into the expanded query, we use a different technique to compute the relevant documents. Our method is divided into two phases: The learning phase for each term works as follows: • Select the old queries in which the specific query term occurs • From these selected old queries get the sets of relevant documents from the ground truth data • From each set of relevant documents compute a new document vector and use these document vectors to build the term concept. The expansion phase for each term is easy: • Select the appropriate concept of the current term • Use a weighting scheme to enrich the new query with the concept For the formal description of the learning phase we need the following definitions: • D = {d1 ,…, dN}: the set of all documents • Q = {q1 ,…, qL}: the set of all known queries with • qk = (w1k ,…,wik ,…, wMk)T represented within the vector space model. For each term of the query the appropriate weight wik is between 0 an 1. • R+(qk) = {dj ∈ D | rij = 1 }: the set of all documents relevant to the query qk Now, the first step of the learning phase collects all queries having the i-th term in common: Qi = { qk ∈ Q | wik ≠ 0 }

(9)

If the i-th term doesn’t occur in any query qk then Qi is empty. The second step collects all documents which are relevant to these collected queries: Dik = {dj | dj ∈ R+(qk) ∧ qk ∈ Qi }

(10)

200

Stefan Klink et al.

In the last step of the learning phase the concept of each i-th term is build as the sum of all documents (i.e. vectors of term weights) which are relevant to the known queries which have the term in common: Ci =

∑ dj dj ∈ Dik

(11)

As queries and documents, a concept is represented by a vector of term weights. If no query qk contains term i, the corresponding concept Ci is represented as (0,…,0)T. Now, where the term-based concepts are learned, the user query q can be expanded term by term. The expanded query vector q’ is obtained by

q’ = q +

M

∑ ωi Ci

(12)

i=1

where ωi are parameters for weighting the concepts. In the experiments described below ωi is set to 1. Before applying the expanded query, it is normalized by q’’ =

q’ q’

(13)

For this approach, the complete documents (all term weights wij of the relevant documents) are summed up and added to the query. Although, in some papers it is reported that using just the top ranked terms is sufficient or sometimes better. But experiments with this approach on the collections have shown that the more words are used to learn the concepts the better the results are. So, the decision was made to use always the complete documents and not only some (top ranked) terms. If no ground truth of relevant documents is available, relevant feedback techniques can be used and the concepts are learned by adding terms from the retrieved relevant documents.

5

Combination of PRF with TCL

Additionally to the approach described above, we made some experiments with a linear combination of our approach with the pseudo relevance feedback. For each query we applied PRF method in parallel to our method and the new query is build by (cmp. (7) and (12)): q’ = q + β ds +

M

∑ ωi Ci

i=1

Before applying the expanded query, it is normalized by equation (13).

(14)


6

201

Experiments and Results

6.1 Test Collections For our comparison we used four standard test collections: CACM (collection of titles and abstracts from the journal ‘Communications of the ACM’), CR (congressional report), FR88 (federal register), NPL (also known as the VASWANI) and ZF3 ('Computer Select' of Ziff-Davis Publishing Co.). These collections are contained in the TREC disks [4]. All collections are provided with queries and their ground truth (for each query a list of relevant documents). For these collections, terms used for document representation were obtained by stemming and eliminating stop words. Table 1. Statistics about collections after stemming and stop words elimination

# documents # queries # different terms avg doc length [terms] avg query length [terms]

CACM 3204 52 3029 25.8 10.9

CR 27922 34 45717 672.8 3.1

FR88 19860 199 43765 869.1 3.5

NPL 11429 93 4415 21.8 6.6

ZF3 161021 50 67108 155.6 7.7

In addition to the number of documents, a significant difference is the length of the documents: CACM and NPL consists of abstracts, while CR, FR88, and ZF3 contain (much) longer documents. Queries in the TREC collections are mostly provided in a structured format with several fields. In this paper, the “title” (the shortest representation) is used for the CR and NPL collection whereas the “desc” (description; medium length) is used for the CACM, FR88, and ZF3 collection. The short queries in FR88 are obtained by stop word elimination 6.2 Evaluation The following paragraphs describe some basic evaluation methods used in this paper. For further information and a more detailed description see Kise et al [7]. 6.2.1

Average Precision

A common way to evaluate the performance of retrieval methods is to compute the (interpolated) precision at some recall levels. This results in a number of recall/precision points which are displayed in recall-precision graphs [2]. However, it is sometimes convenient for us to have a single value that summarizes the performance. The average precision (non-interpolated) over all relevant documents [2, 3] is a measure resulting in a single value. The definition is as follows: As described in section 3, the result of retrieval is represented as the ranked list of documents. Let r(i) be the rank of the i-th relevant document counted from the top of the list. The precision for this document is calculated by i/r(i). The precision values for all documents relevant to a query are averaged to obtain a single value for the

202

Stefan Klink et al.

query. The average precision over all relevant documents is then obtained by averaging the respective values over all queries. 6.2.2

Statistical Test

The next step for the evaluation is to compare the values of the average precision obtained by different methods [7]. An important question here is whether the difference in the average precision is really meaningful or just by chance. In order to make such a distinction, it is necessary to apply a statistical test. Several statistical tests have been applied to the task of information retrieval [5,17]. In this paper, we utilize the test called “macro t-test” [17] (called paired t-test in [5]). The following is a summary of the test described in [7]: Let ai and bi be the scores (e.g., the average precision) of retrieval methods A and B for a query i and define di = ai - bi. The test can be applied under the assumptions that the model is additive, i.e., di = µ + εi where µ is the population mean and εi is an error, and that the errors are normally distributed. The null hypothesis here is µ = 0 (A performs equivalently to B in terms of the average precision), and the alternative hypothesis is µ > 0 (A performs better than B). It is known that the Student’s t-statistic ¯

t=

d s2 / n

¯

with d =

1 n

n

1

n

¯ ∑ di and s2 = n -1 ∑ (di - d)2 i=1 i=1

(15)

follows the t-distribution with the degree of freedom of n – 1, where n is the number ¯ of samples (queries), d and s2 are the sample mean and the variance. By looking up the value of t in the t-distribution, we can obtain the P-value, i.e., the probability of observing the sample results di (1 ≤ i ≤ n) under the assumption that the null hypothesis is true. The P-value is compared to a predetermined significance level σ in order to decide whether the null hypothesis should be rejected or not. As significance levels, we utilize 0.05 and 0.01. 6.3 Results and Comparison to the Standard 6.3.1

Recall and Precision

The results of the pseudo relevance feedback are depending on two parameters α (weight) and θ (similarity threshold). To get the best results, we were varying α from 0 to 5.0 with step 0.1 and θ from 0.0 to 1.0 with step 0.05. For the combined approach we calculated the best β by varying from 0 to 1.0 with step 0.1. For each collection, the best individual α, θ, and β are calculated and used for the comparison. Table 2 shows the best values for each collection: The results of our concept-based expansion are also depending on weights. But due to time restrictions, we had not enough time to vary the weights in a range for each collection. We just used the default value: ωi = 1, which means that the original term and the learned concept are weighted equal.


203

Table 2. Best values for pseudo relevance feedback and the combined method parameters

CACM

CR

FR88

NPL

α / θ (PRF)

1.70 / 0.35

0.6 / 0.75

0.60 / 0.00 2.00 / 0.45 0.80 / 0.50

β (comb.)

0.4

0.6

0.4

0.9

ZF3 0.2

Figure 1 shows the recall/precision results of the original query with the standard vector space model (VSM), the pseudo relevance feedback (PRF), the expanded query using TCL (Concepts) and the combination (Concepts+PRF):

Fig. 1. Recall / precision of CACM, CR, FR88, NPL, and ZF3

The recall/precision graphs in figure 1 indicate that the automatic query expansion method based on learned concepts yields a considerable improvement in the retrieval effectiveness in mostly all collections over all recall points compared to the standard vector space model and to the pseudo-relevance-feedback method (except with the NPL collection). There is no indication that the improvement is depending on the size of the collection, the number of documents nor on the number or size of the queries. On the small ones the method performs good on CACM but only somewhat better

204

Stefan Klink et al.

than the VSM on the NPL and on the FR it performs better than on the CR collection. On a closer look at the figures the impression could arise that our approach performs better on longer queries. But experiments with the CR collection have shown that ‘title’ queries result a better precision than ‘description’ or ‘narrative’ queries. This behavior is in contrast to the first impression of the figures. 6.3.2

Statistical Tests

To be sure that these exciting results are really meaningful and not just by chance, it is necessary to apply a statistical test. As described above, we used the “macro t-test”. The results of this test for all pairs of methods are shown in table 3. The meaning of the symbols such as “≅”, “>” and “~” is summarized at the bottom of the table. For example, the symbol “ (,< ~

: : :

CACM ≅ ≅ ≅ ≅ ≅ > 0.01 ≤ 0.05 ≤

CR ~ ≅ > ≅ > ~

FR88 ≅ ≅ ~ ≅ ≅ ≅

NPL ≅ > < ≅ ~ ≅

ZF3 ≅ ≅ ≅ ≅ ≅ ~

P-value ≤ 0.01 P-value ≤ 0.05 P-value

The macro t-tests prove our results. Our new method for expanding queries based on term-based concepts outperforms the standard VSM and outperforms or is equivalent to the pseudo relevance feedback (except at the NPL collection) and the results are not obtained by chance. Additionally, the combination of our method with the pseudo-relevance feedback outperforms the VSM and the pseudo-relevance feedback. Thus the term-based concepts are capable to improve the PRF just by adding the learned term weights.


7

205

Conclusions and Outlook

We have described a new approach for bridging the gap of different terminology within the user query and the searched documents by using term-based concepts. Each term of the query corresponds to a concept which is learned from the documents given by the ground-truth of other users. The selection relies on the similarity between the query terms and the learned concept rather than on the similarity between the terms of the collection nor on collection-based or hand-made thesauri. This approach can be used to improve the retrieval of documents in all domains like Digital Libraries, Document Management Systems, WWW etc. The approach performs the better the more user queries (users) are involved. Additionally, we combined our method with the pseudo relevance feedback by adding the term weights of our learned term-based concepts. Our experiments made on five standard test collections with different sizes and different document types have shown considerable improvements vs. the original queries in the standard vector space model and vs. the pseudo relevance feedback (except at the NPL collection). The improvements seem to be not depending on the type nor the size of the collection and they are not obtained by chance. In contrast to the relevance feedback, this approach is not relying on critical thresholds which are dangerous and mostly differ from collection to collection. Furthermore, this approach can be perfectly used in search machines where new queries with their appropriate relevant (user-voted) documents can be easily added to the ‘collection’, for example in Digital Libraries, Document Management Systems or the WWW. These new queries can be used to build an increasing approach and for a constant learning of the stored concepts. The vital advantage is that each user can profit from the concepts learned by other users. The more queries are learned (by the same or by other users) the better our approach will perform. Some experiments are planed to use user-voted relevance feedback instead of collection-given ground-truth to test the performance on ‘real-life’ data. Furthermore, it is planed to make some experiments on the influence of ωi for each term. An approach on passage-based retrieval by Kise [7] has shown good improvements vs. LSI and Density Distribution. An interesting idea for the future is not using the complete relevant documents for expanding the query and not using the N top ranked terms but using terms of relevant passages within the documents. With this idea just the relevant passages are used to learn the concepts. This should increase the quality of the expanded queries and we will be able to do a further evaluation of each concept in great detail, i.e. on the term level.

8

Acknowledgements

This work was supported by the German Ministry for Education and Research, bmb+f (Grant: 01 IN 902 B8).

206

Stefan Klink et al.

References 1.

2. 3. 4. 5. 6. 7.

8. 9. 10. 11. 12.

13. 14. 15. 16.

Buckley C., Salton G., Allen J.: The effect of adding relevance information in a relevance feedback environment. In Proceedings of the Annual Int. ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 292 300, 1994 Baeza-Yates R., Ribeiro-Neto B.: Modern Information Retrieval. AddisonWesley Pub. Co., 1999. ISBN 020139829X ftp://ftp.cs.cornell.edu/pub/smart/ http://trec.nist.gov/ Hull D.: Using Statistical Testing in the Evaluation of Retrieval Experiments. In Proceedings of the 16th Annual Int. ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 329 - 338, 1993 Jansen B.J., Spink A., Bateman J. and Saracevic T.: Real Life Information Retrieval: A Study of User Queries on the Web, In SIGIR Forum, Vol. 31, pp. 5-17, 1988 Kise K., Junker M., Dengel A., Matsumoto K.: Passage-Based Document Retrieval as a Tool for Text Mining with User’s Information Needs, In Proceedings of the 4th Internatl. Conference of Discovery Science, pp. 155-169, Washington, DC, USA, November 2001 Manning C.D. and Schütze H.: Foundations of Statistical Natural Language Processing, MIT Press, 1999 McCune B.P., Tong R.M., Dean J.S., Shapiro D.G.: RUBIC: A System for RuleBased Information Retrieval, IEEE Transaction on Software Engineering, Vol. SE-11, No.9, September 1985 Minker J., Wilson, G.A. Zimmerman, B.H.: An evaluation of query expansion by the addition of clustered terms for a document retrieval system, Information Storage and Retrieval, vol. 8(6), pp. 329-348, 1972 Peat H.J., Willet, P.: The limitations of term co-occurrence data for query expansion in document retrieval systems, Journal of the ASIS, vol. 42(5), pp. 378-383, 1991 Pirkola A.: Studies on Linguistic Problems and Methods in Text Retrieval: The Effects of Anaphor and Ellipsis Resolution in Proximity Searching, and Translation and query Structuring Methods in Cross-Language Retrieval, PhD dissertation, Department of Information Studies, University of Tampere. Acta Universitatis Tamperensis 672. ISBN 951-44-4582-1; ISSN 1455-1616. June 1999 Qiu Y.: ISIR: an integrated system for information retrieval, In Proceedings of 14th IR Colloqium, British Computer Society, Lancaster, 1992 Salton G., Buckley C.: Term weighting approaches in automatic text retrieval. Information Processing & Management 24(5), pp. 513 - 523, 1988 Sparck-Jones K.: Notes and references on early classification work. In SIGIR Forum, vol. 25(1), pp. 10-17, 1991 Smeaton A.F., van Rijsbergen C.J.: The retrieval effects of query expansion on a feedback document retrieval system. The Computer Journal, vol. 26(3), pp. 239 – 246, 1983


207

17. Yang Y. and Liu X.: A Re-Examination of Text Categorization Methods. In Proceedings of the 22nd Annual Int. ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 42 - 49, Berkeley, CA, August 1999

Learning to Play a Highly Complex Game from Human Expert Games Tony Kråkenes and Ole Martin Halck Norwegian Defence Research Establishment (FFI) P.O. Box 25, NO-2027 Kjeller, Norway {tony.krakenes,ole-martin.halck}@ffi.no

Abstract. When the number of possible moves in each state of a game becomes very high, standard methods for computer game playing are no longer feasible. We present an approach for learning to play such a game from human expert games. The high complexity of the action space is dealt with by collapsing the very large set of allowable actions into a small set of categories according to their semantic intent, while the complexity of the state space is handled by representing the states of collections of pieces by a few relevant features in a locationindependent way. The state–action mappings implicit in the expert games are then learnt using neural networks. Experiments compare this approach to methods that have previously been applied to this domain.

1

Introduction

This paper describes the application of machine learning techniques to the problem of making a software agent that plays a highly complex stochastic game. The game we consider, Operation Lucid, belongs to the class of two-person zero-sum perfectinformation stochastic games. It has been designed as a simplified military land combat model, with rules representing central concepts such as movement (and uncertainty in movement), logistics, and of course combat itself, including the asymmetry between attacking and defending a location. Our studies are concerned with the application of artificial intelligence techniques to decision making in combat models, and in this research Operation Lucid is being used as an environment that captures the important general properties of such models, while allowing us not to get bogged down in unnecessary detail. The insights and results gained in this way can then be used in the development and improvement of full-scale combat models. The problem of game playing has been extensively studied in machine learning research. A number of papers describing state-of-the-art developments in this field are collected in [1]; this reference also contains a survey of machine learning in games [2]. However, in most of the games studied in this body of research, the main challenges are different to those posed by Operation Lucid, making several of the standard techniques useless for our problem. T. Elomaa et al. (Eds.): ECML, LNAI 2430, pp. 207-218, 2002.  Springer-Verlag Berlin Heidelberg 2002

208

Tony Kråkenes and Ole Martin Halck

If we regard Operation Lucid as a decision-making problem in a combat simulation context, related research is somewhat thinner on the ground. Recent work includes [3], in which a genetic algorithm is applied to a force allocation problem not entirely dissimilar to ours, and [4], which describes how a knowledge-intensive agent is used for evaluating military courses of action. The remainder of the paper is organized as follows: Section 2 describes our problem domain. In Section 3, we describe our approach in dealing with the high complexity of the problem in order to make a game-playing agent; a more detailed description of the implementation of the agent is the subject of Section 4. Section 5 presents some experimental results, and Section 6 concludes the paper.

2

The Game of Operation Lucid

In this section we present our problem environment – the game of Operation Lucid – and describe some of the properties that make it both interesting and very challenging. A fuller description of the game is given in [5]. 2.1

Definition and Rules of Operation Lucid

In short Operation Lucid is a two-person stochastic board game where the two players start off with their pieces in opposing ends of the board, as shown in Figure 1. One player, named Blue, is the attacker. Blue starts the game with fifteen pieces; his aim is to cross the board, break through – or evade – his opponent’s defence, and move his pieces off the board into the goal node. The defending player, Red, starts with ten pieces; his task is to hinder Blue from succeeding. The result of the game is the number of pieces that Blue manages to get across the board and into the goal node; thus, there is no “winner” or “loser” of a single game. The rest of this section describes the rules of the game. The game of Operation Lucid is played in 36 turns. At the start of each turn, the right to move pieces is randomly given to either Blue or Red, with equal probabilities. The side winning this draw is allowed to move each piece to one of the neighbouring nodes (that is, to a node that is connected to the piece’s current node by an edge) or leave it where it is. The side losing the draw naturally does not get to move any pieces in that turn. The movement of the pieces is subject to two restrictions: • •

When the move is finished, no node can have more than three pieces of the same colour. Pieces cannot be moved from nodes where the player is defined as the attacker (see below).

Whenever Blue and Red pieces are in the same location at the end of a turn, combat ensues, and one of the pieces in that node is lost and taken out of the game. A weighted random draw decides which side loses a piece. In a node having combat, the player last entering is defined as the attacker of that location, while the other part is defined as the defender of the location. The weighted random draw is specified by the probability that Blue wins, that is, that Red loses a piece. This probability is given

Learning to Play a Highly Complex Game from Human Expert Games

209

by the fraction (Blue strength)/(Blue strength + Red strength), where a player’s strength in a node equals the number of own pieces in that node, modified in accordance to two rules. Firstly, the defending player in the node gains one extra point of strength. Secondly, if the Blue player does not have an unbroken path of nodes with only Blue pieces leading from the combat node to one of Blue’s starting positions (a supply line), Blue loses one point of strength. The game ends when the 36 turns are completed, or when Blue has no pieces left on the board. The result of the game is the number of Blue pieces that have reached the goal node. Goal

10 Red pieces

5 4

N

3 2 15 Blue pieces

1 A

B

C

D

E

Fig. 1. The board of the game Operation Lucid, with the pieces placed in their initial positions

2.2

A Combat Modelling Interpretation of the Game

Operation Lucid was designed to capture important aspects of military land combat modelling; in particular, it represents a scenario where the goal of one side is to break through enemy defence to reach a certain location. Movement of the pieces on the board naturally represents the movement of force units in a terrain; the stochastic ordering of Blue and Red moves is intended to capture the uncertainty inherent in manoeuvring in possibly unfamiliar territory. The rules for determining the result of combat naturally take into account the numerical strength of each side in the area of combat. In addition, they represent the advantage of being the defender of a location; this advantage is due to the defender’s opportunity to prepare himself and the environs for resisting attacks. The rule regarding Blue supply lines models the effects of logistics; an invading force will need a functioning line of supplies back to his home base to be able to perform well in enemy territory. 2.3

The Complexity of the Problem

A seemingly obvious way of playing Operation Lucid is by evaluating (in some way) each legal move in the current game state, and then choosing the one with the best

210


evaluation. This would reduce the problem of constructing a player agent to the problem of evaluating moves in given game states. This method is generally not feasible, however, as the number of possible moves in each state tends to be huge. A player may allocate one of at most five actions (stand still or move in either of four directions) to each of at most fifteen pieces, so an upper bound on the number of legal moves is 515 ≈ 3 ⋅1010. If we assume that a computer generates one thousand possible moves each second (a reasonable assumption according to our experience), it might take up to a year to enumerate all legal moves in one state. In typical game states the number is usually far lower than this – the player may have fewer than fifteen pieces left, and each of the pieces may not be free to perform all five actions. Also, a lot of the legal moves are equivalent, as all pieces of the same side are interchangeable. From the initial position, for instance, the number of possible non-equivalent moves for Blue is 60,112 (disregarding equivalence by symmetry). In intermediate states of the game the number of legal moves increases quickly, and is generally far too large for an exhaustive enumeration. Thus classical methods based on enumeration and evaluation of moves are infeasible in this domain.

3

Handling the Complexity of the Problem

How can the problem of the game’s high complexity be dealt with efficiently? In this section, we briefly mention some previous work on designing player agents for Operation Lucid, and describe the considerations that led to the agent design that is the main subject of this paper. Our efforts so far have mostly been focused on developing Blue agents, and this is the case in the present work as well. Building good Blue agents tends to be a more challenging task than building Red ones, since the nature of the game requires Blue to be the more creative and active side. However, it is usually a minor task to adjust the algorithms to fit a Red player as well. 3.1

Previous Approaches

As explained above, the usual way of making computers play games, that is, generating all possible moves and evaluating which is the best, is not feasible in Operation Lucid. We therefore looked to the way humans play games in order to create good player agents for the game. Humans generally do not test all available moves; rather, we decide on a goal, and form a plan we believe will help us reach this goal. Limiting the Set of Evaluated Moves. In one main approach we have followed, we kept part of the evaluative approach to game-play, but limited the number of moves to be evaluated to a tractable level. This was achieved by imposing constraints on the desired number of pieces in various areas of the board – these constraints could be seen as a representation of a plan for how to play. The challenging part of this procedure was deciding on which constraints to impose, so that the problem became tractable in size without eliminating good candidate moves. This approach in general, and in particular a Blue agent that used self-trained neural networks for evaluating moves, is described further in [6]. The constraint-based approach was successful in


211

that it yielded agents with fairly good performance. The main disadvantage of this method was that the constraints that had to be imposed in order to keep runtime at a reasonable level limited the range of play of the agents. In effect, this approach meant that the main strategy of play was entered by hand through the move constraints, while the move evaluator performed limited tactical optimizations. Simple Imitation Learning. In our second main approach, we took the idea of playing like a human literally, and designed an agent that played by lazy imitation learning. We made a lookup-table of game states and corresponding expert moves; the agent used this database by trying to imitate the action taken in the stored game state that was most similar to the one at hand. The challenging part of this method was to define a suitable similarity metric between states, and especially to deal with the unwanted side effects arising from states not being exactly equal. Our work with this kind of pure imitation learning produced rather disappointing results. The main reason for this was that the player very quickly found itself in game states with no very close match in the expert database, so that the move taken in the most similar state was not applicable in the present game state. It was realized that these problems were due to the fact that similarity, both of states and of moves, were seen at what can be called a syntactical level. At this level, the actual position and movement of the pieces in each single node were the basis of the agent’s behaviour, without any semantical notion of the role of the pieces in the game. The result was that this syntactical imitation would often require some pieces to perform unfeasible moves, while other pieces were left idle. One obvious fix to this problem would be to increase the size of the expert database, hoping to cover a greater range of board positions. This approach however requires man-hours, and in our case doubling the database showed only marginal improvement during play at the cost of increasing the runtime considerably. 3.2

Current Approach

Our experiments with pure imitation learning showed that the semantics of the problem domain would have to be addressed to some degree in order to achieve good and efficient performance. One way of doing this could be by following the methodology used in case-based reasoning (see e.g. [7]), where retrieval of previous cases from a database is combined with symbolic reasoning based on a semantic model of the domain1. Case-based reasoning has previously been applied to games such as chess [9] and Othello [10], as well as many real-world problems. The main disadvantages of this approach is that constructing such a domain model is a difficult and time-consuming task, and that frequent case retrievals – as is the case in game playing – may be costly in terms of runtime. These considerations led us to the conclusion that we required an agent design that imitates the expert games on a more abstract level than as single positions and movements of pieces, while not depending on a full semantic model of the game. In

1

We follow the terminology of Mitchell [8], where the term “case-based” is reserved for a subset of the broader set of “instance-based” methods, namely those using richer instance representations than simple feature vectors.

212


the following, we describe how this was done in the case of moves and game states respectively, and explain how the agent chooses its moves based on the expert games. Moves. As in the simple imitation learning above, our current approach to handling the complexity of the problem is to have an agent capable of mapping directly from game states into the proper moves to take. In order to make this work well, the action space must be reduced considerably in size. To this end, we collapsed the action space from the syntactical level consisting of all possible combinations of single-piece moves to a semantic level featuring only a few move categories. The move categories are symbols (e.g. attack, outflank, breakSupply) describing the overall character or intent of the move. The move categories are not disjunctive – they are defined so that a move may be labelled into more than one category. Game States. Reducing the action space in the manner described above raises another question: which pieces are to perform which types of action? Different pieces in a given expert game may be used for different intentions; this made it desirable to work with sub-collections of pieces sharing common intentions, rather than with all the pieces collectively. We call these sub-collections sharing common intentions force groups (FGs). Common intentions usually coincide with co-location of pieces on the board; this led us to define a FG as a collection of pieces of the same colour interconnected with each other but not with other friendly pieces. The pieces in a FG should act together pursuing the intentions of the group, and should differing intentions occur within a FG, it may break up, forming smaller FGs individually regaining conformity of intentions. The main advantage of introducing the concept of FGs is that it allows collapsing the game state space considerably. Previously, the game state was represented by the number of pieces for each side in each individual node, the number of remaining rounds and which side defended each combat node. We now employ a FG-centric state representation where individual pieces and nodes, even the board geography itself, are no longer of direct interest. What is of interest is a set of aggregated features like path lengths to the goal node, resistance along these paths, the number of own pieces in the FG and in total, the number of enemy pieces, distance to neighbouring FGs (if any), combat strength of the FG (including supply and defender status) and remaining rounds. Each FG in a game state has its individual, location-independent state perception, on which it bases its actions. This means that FGs located on different parts of the board are considered similar if their environs are similar in terms of the FG-centric state representation. The representation we use is illustrated in more detail in Figure 2. State–Action Mapping. Using the state and move representation just presented, we could repeat the lookup-table method described in Section 3.1 for mapping observed game states into actions to perform. The problem of finding a good distance metric for comparing FG states in order to choose the most similar one would then still remain. Instead, we chose to leave the lazy-learning design, and used the expert games to train a set of neural network classifiers for state–action mapping. In this way, the mappings implicit in the expert games may also generalize better to new FG states; an added advantage is that a full scan through the database at each decision point is no longer necessary, so that runtime is decreased. A more detailed look at the making of these classifiers, along with the rest of the player agent, is the subject of the next section.


213

Goal

(Distance1, resistance1)

(2,1)


(2,2)


(3,1)


(4,3)


(7,2)

Units in this group

5

Total Blue units

12

Total Red units

9

Distance to closest FG

3

Blue force in combat

0

Red force in combat

0

Proportion supplied

0

Proportion of attackers

0

Rounds left

27

Fig. 2. Representation of the game state as seen from the perspective of a force group. The ten first attributes gives the distance to each of the five exit nodes, together with the number of Red pieces on the path to each node. These are sorted in ascending order. The remaining nine attributes describe own and opposing forces, the combat situation, and the number of rounds left

4

Implementing the Agent Design

Our goal is to construct game-playing software agents. Such an agent should be able to get a game state as input, and, from this state and the rules of the game, generate a move as output. The move describes where each of the own pieces should be placed when the turn is finished. In accordance with the design choices described in the previous section, our agent selects its move as illustrated in Figure 3. Upon receiving the current game state, the agent identifies FGs and gives each FG a self-centric and simplified perception of the game state. Each FG then uses the move type classifiers in conjunction with this state representation in order to select which class (or classes) of moves is appropriate in its current situation. Finally, the agent should of course be able to translate these pieces of semantic move advice into actual board movement of the pieces in a suitable way. The remainder of this section details the steps involved in building the agent. In the following, game state or simply state refers to the overall game state (i.e. positioning of pieces, rounds left and attacking sides in the case of combat), while FG-state refers to the simplified, egocentric state perception of each FG. Similarly, move refers to the collective action of all Blue’s pieces in a turn, while FG-move refers to the action taken by a single FG only.

214


Input: Game state

Force group extraction

FG 1

FG 2

approach?

outflank? Y/N

...

Y/N

... ...

...

...

Move type classifiers

Chosen group move types

...

Move implementation Group move

Output: Full move

Fig. 3. Agent design

4.1

Database of Expert Games

A database of 20 human expert games for Blue against a fixed Red opponent was compiled. On each of Blue’s turns in these games, one or more FGs would be present; the total number of Blue FG-states throughout the game series (not necessarily distinct) was 646. The expert labelled each FG-move into one or more of twelve move categories according to the intent of the move. The move categories are given, along with brief descriptions of their meaning, in Table 1. The Red opponent employed in the expert games, named AxesRed, is an automatic playing agent adopting two main strategies of play. Firstly, it will never advance from the home row (i.e. the northernmost row) into the field, but stay home and wait for Blue to attack. Secondly, it attempts to position its pieces within the home row in a manner that proportionally mirrors the perceived threat on each vertical axis of nodes. For instance, if Blue has 13 pieces left, and 4 of these are located on the B axis (see Figure 1 for references to parts of the board), Red will to the best of his ability attempt to position 4/13 of his remaining pieces on this axis, i.e. in B5. A few simple rules apply to ensure that Red keeps a sound level of play in special cases. Red can of course not have more than 3 pieces in any node, and should its calculations require more than this, it will ensure that backup pieces are kept in the neighbourhood. Red will not reinforce in a combat node – this entails losing the defender’s advantage – if this is not more than compensated for by the strength of the extra pieces. Although AxesRed is a rather simple player, it has proved to serve well as a benchmark opponent for evaluating Blue agents.


215

Table 1. Move categories for Blue force groups

Category Approach EnsureSupply

Description Move pieces closer to the goal node Keep some pieces back, with the intent of ensuring a supply line for present or future combat BreakSupply Break the supply line, i.e. advance pieces previously withheld for supply purposes from the southernmost rows Attack Move pieces into a node containing only Red pieces ReinforceCombat Move additional pieces into a combat node ContinueCombat Neither exit from nor reinforce existing combat Outflank Perform an evading manoeuvre, sideways or backwards, aiming at a different exit node GoToGoal Move pieces from the northernmost row into the goal node ConcentrateForces Move pieces within a FG closer, i.e. occupying fewer nodes SplitIntoGroups Divide a FG into two or more FGs LinkUpGroups Join a FG to another, creating a larger FG StayInPosition Leave all pieces in the FG unmoved 4.2

Neural Network Classifiers for Force Groups

Having assembled the database of FG-states and corresponding expert semantic FGmoves, we trained an ensemble of neural networks (NNs) to serve as FG-state classifiers for the agent. One NN was trained for each label, using the database of FGstates as input data and the presence (0 or 1) of the label in question as target values. Each network was a standard feedforward NN featuring 19 input nodes, 36 hidden nodes and 1 output node (ranging from 0 to 1), sigmoid activation functions, and weights initially randomised from –0.2 to 0.2. Training was done by backpropagation. The input vector for each FG-state was scaled so that the magnitude ranges of the components were similar. About 1/3 of the data set was initially reserved and used for validating the training procedure, and the NNs were trained by repeatedly picking random examples from the training data. The learning of the NNs was evaluated by the proportion of correctly classified examples over the validation data set. The ensemble quickly attained a classification performance of 0.9, and after further training reached about 0.95. We noted that performance on the validation set did not start to decrease, even if training was continued. Taking this as an indication that the data set presented little danger of overfitting – even with the large number of network weights used – we restarted training using all available data. The total classification accuracy on the full training set reached about 0.99; the individual label-specific NNs showed minor deviations from this average. As explained above, the trained NNs are used in the game-playing agent. In a given game state, each FG inputs its FG-state into the twelve nets, each of which answers a number between 0 and 1. Output close to 1 indicates that the FG should perform a move corresponding to the category in question. The output value of 0.5 was used as the limit for choosing move categories.

216


In this procedure, the decision-making task of the player agent can be regarded as delegated to its constituent FGs, which choose and perform actions based on their own perceptions of the game state. An interpretation of the NNs when this view is adopted is as a shared overall doctrine for how to act in given situations. 4.3

Implementing the Acting Module

With the classification module of our agent properly in place, we turned to the task of actually designing and implementing the acting module. This module receives as input one or more chosen move categories for each FG, and returns the resulting movement of the individual pieces in the FG. Due to space restrictions, we are unable to go into details of this module here. Instead, we mention a number of difficulties that arose in the implementation of this part of the agent. In particular, the set of chosen move categories may be inconsistent, in which case not all of the move types may be performed, e.g. if both breakSupply and ensureSupply are chosen. Another inconsistent set of move types is the empty set – since we have specified stayInPosition as a category of its own, and this category was not selected, this is not an order to simply stand still. In cases when more than one move category is specified, we must decide which pieces should move according to which categories, or alternatively which categories should be given precedence. At the current stage, we use the simple strategy of using the numerical outputs of the respective neural nets for ranking the categories. Pieces are moved according to the first category, and if after this some pieces have not been assigned to an action, moves for the next category is implemented, and so on. The problem with empty move sets mentioned above was dealt with by defining the approach category as a default action; this category was chosen because advancing across the board is a reasonable baseline course of action for Blue – at any rate, it is almost always better than standing still. Table 2. Results and approximate runtimes for various Blue agents playing against AxesRed

Player Human SimpleBlue OneAxisBlue ConstraintNNBlue ImitationBlue Present agent

5

Average score 9.41 3.88 5.34 6.60 4.04 6.23

Approx. time (s/game) – 0.2 0.4 90 30 12

Experiments

We measure the success of a Blue agent by how many pieces it manages to move into the goal node in play against the AxesRed agent. Therefore, we need to have an idea of what constitutes a good result when playing against this particular Red opponent. The result from the expert games is a natural measure of the potential of our agent, since after all it is this expert behaviour we are trying to learn from. What then is a


217

bad result? This is difficult to say, but we can at least get a notion of a mediocre result by letting some rather naive Blue players challenge the AxesRed player. Two such benchmark Blue players have been designed. The first, SimpleBlue, employs the simple strategy of moving all its pieces forward when receiving the turn. This results in a full-breadth simultaneous attack, where three Blue pieces take on two Red pieces in each of the northernmost nodes. The second player, OneAxisBlue, initially decides upon an axis of attack, and advances as many of its pieces as possible along this axis for the rest of the game. This results in a focused attack on one of the northernmost nodes. Neither of these two players actively keeps a supply line, although the design of the game ensures that OneAxisBlue’s supply line happens to be intact in the first phase of the attack. Furthermore, it is interesting to compare the performance of our new agent with the best results from the two approaches described in Section 3.1. The agent called ConstraintNNBlue – the highest scoring agent we have managed to make during our previous work – is the constraint-based agent with NN move evaluation, while ImitationBlue is the rather less successful lazy learner. The average results obtained by the human expert and the four agents mentioned are given in Table 2, along with the best result obtained by the agent treated in this paper. For each automatic agent, 1000 games were played; the human played 20. The approximate average runtime per game is also reported. As we can see, our present agent outperforms the two benchmark players and the imitating agent, but still has some way to go to reach the human expert – this latter fact is nothing more than could be expected2. Comparing the agent to ConstraintNNBlue shows that it fails to set a new record; on the other hand, it is not discouragingly far behind, while being almost an order of magnitude quicker. Moreover, we expect the agent to have considerable potential for improvement within the limits of the current design; a larger database of expert games and better move implementations are two of the more obvious measures that can be taken.

6

Conclusion

We have presented the design and implementation of an agent playing a highly complex stochastic game. The complexity of the game makes it impossible to use standard game-playing methods; instead, the agent uses neural networks to learn to play from a database of human expert games. The high complexity is handled by collapsing the huge action space into a few categories representing the semantic intentions of moves, and representing the game states of subsets of the agent’s playing pieces by a few relevant features. An experimental evaluation of this approach shows promising results.

2

Indeed, our experience with the game leads us to suspect that the human must have been rather lucky in these 20 games to achieve this score.

218


References 1.

Fürnkranz, J., Kubat, M. (eds.): Machines That Learn to Play Games, Nova Science Publishers (2001). 2. Fürnkranz, J.: Machine learning in games: A survey. In: Fürnkranz, J., Kubat, M. (eds.): Machines That Learn to Play Games, Nova Science Publishers (2001) 11– 59. 3. Schlabach, J. L., Hayes, C. C., Goldberg, D. E.: FOX-GA: A genetic algorithm for generating and analyzing battlefield courses of action. Evolutionary Computation 7 (1999) 45–68. 4. Boicu, M., Tecuci, G., Marcu, D., Bowman, M., Shyr, P., Ciucu, F., Levcovici, C.: Disciple-COA: From agent programming to agent teaching. In: Langley, P. (ed.): Proceedings of the 17th International Conference on Machine Learning (ICML-2000), Morgan Kaufmann (2000) 73–80. 5. Dahl, F. A., Halck, O. M.: Three games designed for the study of human and automated decision making. Definitions and properties of the games Campaign, Operation Lucid and Operation Opaque. FFI/RAPPORT-98/02799, Norwegian Defence Research Establishment (FFI), Kjeller, Norway (1998). 6. Sendstad, O. J., Halck, O. M., Dahl, F. A.: A constraint-based agent design for playing a highly complex game. In: Proceedings of the 2nd International Conference on the Practical Application of Constraint Technologies and Logic Programming (PACLP 2000), The Practical Application Company Ltd (2000) 93–109. 7. Aamodt, A., Plaza, E.: Case-based reasoning: Foundational issues, methodological variations, and system approaches. AI Communications 7 (1994) 39–59. 8. Mitchell, T. M.: Machine Learning. WCB/McGraw-Hill (1997). 9. Kerner, Y.: Learning strategies for explanation patterns: Basic game patterns with application to chess. In: Veloso, M., Aamodt, A. (eds.): Proceedings of the 1st International Conference on Case-Based Reasoning (ICCBR-95). Lecture Notes in Artificial Intelligence Vol. 1010, Springer-Verlag (1995) 491–500. 10. Callan, J. P., Fawcett, T. E., Rissland, E. L.: CABOT: An adaptive approach to case-based search. In: Proceedings of the 12th International Conference on Artificial Intelligence, Morgan Kaufmann (1991) 803–809.

Reliable Classifications with Machine Learning Matjaˇz Kukar and Igor Kononenko University of Ljubljana, Faculty of Computer and Information Science Trˇzaˇska 25, SI-1001 Ljubljana, Slovenia {matjaz.kukar,igor.kononenko}@fri.uni-lj.si

Abstract. In the past decades Machine Learning algorithms have been successfully used in numerous classification problems. While they usually significantly outperform domain experts (in terms of classification accuracy or otherwise), they are mostly not being used in practice. A plausible reason for this is that it is difficult to obtain an unbiased estimation of a single classification’s reliability. In the paper we propose a general transductive method for estimation of classification’s reliability on single examples that is independent of the applied Machine Learning algorithm. We compare our method with existing approaches and discuss its advantages. We perform extensive testing on 14 domains and 6 Machine Learning algorithms and show that our approach can frequently yield more than 100% improvement in reliability estimation performance.

1 Introduction Usually Machine Learning algorithms output only bare classifications for the new unclassified examples. While there are ways for almost all Machine Learning algorithms to at least partially provide quantitative assessment of a classification in questions, so far there is no general method to assign reliability to a single classification. Note that we are interested in the classifier’s performance on a single example and not in average performance on an independent dataset. Let us define the reliability of classification as an estimated probability that the (single) classification is in fact the correct one. Some authors [16, 21] use for this purpose a statistical term confidence. We, however, have decided to use a term reliability, since its calculation and interpretation are not always strictly statistical. For a given example description xi we define the reliability of its predicted class yi as follows. Rel(yi ) = P(yi is a true class of example xi )

(1)

There have been numerous attempts to assign probabilities to Machine Learning classifiers’ (decision trees and rules, Bayesian classifiers, neural networks, nearest neighbour classifiers, . . . ) in order to interpret their decision as a probability distribution over all possible classes. In fact, we can trivially convert every Machine Learning classifier’s output to a probability distribution by assigning the predicted class the probability 1, and 0 to all other possible classes. The posterior probability of the predicted class can be viewed as a classifier’s trust in its prediction (reliability) [3, 19]. However, such estimations may not be good due to the applied algorithm’s language and representational biases. T. Elomaa et al. (Eds.): ECML, LNAI 2430, pp. 219–231, 2002. c Springer-Verlag Berlin Heidelberg 2002

220

Matjaˇz Kukar and Igor Kononenko

There is some ongoing work for constructing classifiers that divide the data space into regions that are reliable and regions that are not reliable [1]. Such meta-learning approaches have also been used for picking the most reliable prediction from the outputs of an ensemble of classifiers [14, 17]. We propose a different approach based on a general transductive method for reliability estimations. Our approach differs from the above in the following: • it does not divide the data space into reliable and unreliable regions, but works instead on single data points (examples), • it does not induce a meta-classifier at all, but instead uses a transductive framework to generate a reliability estimate for each single example. Our approach is independent of the applied Machine Learning algorithm and requires only that it is able to represent its classifications as probability distributions. The core idea is to compare differences in classification’s probability distributions between inductive and transductive steps and use them to assess reliability of single points (examples) in data space. Such assessments are very useful, especially in risk-sensitive applications (medical diagnosis, financial and critical control applications) because there it often matters, how much one can rely upon a given prediction. In such cases a general reliability measure of a classifier (e.g. classification accuracy, mean squared error, . . . ) with respect to the whole input distribution would not provide desired warranty. Another use of reliability estimations is in ensembles for selecting or combining answers from different classifiers [8]. The paper is organized as follows. In Sec. 2 we describe the basic ideas of transductive inference and outline the reasons why transductive reliability estimation should work well. In Sec. 3 we develop our idea for general and efficient implementation of transductive reliability estimation. In Sec. 4 we evaluate our approach on 14 domains with 6 Machine Learning algorithms. In Sec. 5 we present some conclusions and directions for future work.

2 Transduction Principle for Reliability Estimation Transduction is an inference principle that takes a training sample and aims at estimating the values of a discrete or continuous function only at given unlabelled points of interest from input space, as opposed to the whole input space for induction. In the learning process the unlabelled points are suitably labelled and included into the training sample. The usefulness of unlabelled data [12] has among others been advocated in the context of co-training. It has been shown that for a better-than-random [2] classifier its performance can be significantly boosted by using only additional unlabelled data. It has been suggested [20] that when solving a given problem one should avoid solving a more general problem as an intermediate step. The reasoning behind this principle is that, in order to solve a more general task, resources may be wasted or compromises made which would not have been necessary for solving only the problem at hand (i.e. function estimation only on given points). This common-sense principle reduces a more general problem of inferring a functional dependency on the whole input space (inductive inference) to the problem of estimating the values of a function only at given points (transductive inference).

Reliable Classifications with Machine Learning

221

Let X be a space of attribute descriptions of points in a training sample, and Y a space of labels (continuous or discrete) assigned to each point. Given a probability distribution P , defined on the input space X × Y , a training sample S = {(x1 , y1 ), . . . , (xl , yl )}

(2)

consisting of l points, is drawn i.i.d. (identically independently distributed) according to P . Additional m data points (working sample) W = {xl+1 , . . . , xl+m , }

(3)

with unknown labels are drawn in the same manner. The goal of transductive inference is to label all the points from the sample W using a fixed set H of functions f : X → Y in order to minimize an error functional both in the training sample S and in the working sample W (effectively, in S ∪ W ) [5, 16]. In contrast, inductive inference (excluding ensembles of classifiers) aims at choosing a single function f ∈ H that is best suited to the unknown probability distribution P . At this point arises a question how to calculate the labels for a working sample. This can be done by labelling every point from a working sample with every possible label value; however given m working points and n possible class labels this leads to a combinatorial explosion yielding nm possible labellings. For each possible labelling, an induction process on S ∪W is run, and an error functional (error rate) is calculated. By leveraging the i.i.d. sampling assumption and transductive inference, one can for each labelling estimate its reliability (a probability that it is correct). If the i.i.d. assumption holds, the training sample S as well as the joint correctly labelled sample S ∪W should both reflect the same underlying probability distribution P . If one could measure a degree of similarity between probability distributions P (S) and P (S ∪W ), this could be used as a measure of reliability of the particular labelling. Unfortunately, this problem is in non-computable [11], so approximation methods have to be used [21, 9]. 2.1 Why does Transduction Work? There is a strong connection between the transduction principle and the algorithmic (Kolmogorov) complexity. Let the sets S and S ∪ W be represented as binary strings u and v, respectively. Let l(v) be the length of the string v and C(v) its Kolmogorov complexity. We define the randomness deficiency of the string v as following [11, 21]: δ(v) = l(v) − C(v)

(4)

Randomness deficiency measures how random is the respective binary string and therefore the set it represents. The larger it is, the more regular the string (and the set). If we could calculate the randomness deficiency (but we cannot, since it is not computable), we could do it for all possible labellings of the set S ∪W and select the labelling of W that results in the largest randomness deficiency of the joint set S ∪W as the most probable one [21]. We could also construct a universal Martin-Löf’s test for randomness [11]:

∑{P(x|l(x) = n) : δ(x) ≥ m} ≤ 2−m

(5)

222


That is, for all binary strings of fixed length n, the probability of their randomness deficiency δ being greater than m is less than 2−m . The value 2−δ(x) is therefore a p-value function for our randomness test [21]. Unfortunately, the definition of randomness deficiency is based on the Kolmogorov complexity and is not computable. Therefore we need feasible approximations to use this principle in practice. Extensive work has been done by using Support Vector Machines [5, 16, 21], however no general approach exists so far. 2.2 A Machine Learning Interpretation In Machine Learning terms, the sets S and S ∪ W are represented with induced models MS and MS∪W . Randomness of the sets is reflected in the (Kolmogorov) complexity of the respective models. If for the set S ∪W the labelling with the largest randomness deficiency is selected, it follows from the definition (Eq. 4) that since the uncompressed description length l(v) is constant, the Kolmogorov complexity C(MS∪W ) is minimal. This implies that the respective labelling of W is most consistent with the training data S, since the minimal Kolmogorov complexity implies most regularities in the data. This in order implies that our Machine Learning algorithm will induce the model MS∪W that will be most similar to the MS .1 Ideally, if the training data S is sufficient for inducing a perfect model, there is no difference between MS and MS∪W . This greatly simplifies our view on the problem, namely it suffices to compare the (finite) models MS and MS∪W . Greater difference means that the set S ∪ W is more random than the set S and (under the assumption that S is sufficient for learning effective model) that W consist of (at least some) improperly labelled, untypical examples. Although the problem seems easier now, it is still a computational burden to calculate changes between model descriptions (assuming that they can be efficiently coded; black-box methods are thus out of question). However, there exists another way. Since transduction is an inference principle that aims at estimating the values of a function only at given points of interest from input space (the set W ), we are interested only in model change considering these examples. Therefore we can compare the classifications (or even better, probability distributions) of models MS and models MS∪W . Obviously, the labelling of W that would minimally change the model MS is as given by MS . We will examine this approach in more detail in the next section.

3 Efficient Transductive Reliability Estimations The prerequisite for a Machine Learning algorithm to be used in a transductive reliability framework is to represent its classifications as a probability distribution over all possible classes, although these distributions may not be very good estimates. The transductive reliability estimation process is basically a two-step process, featuring an inductive step followed by a transductive step. 1

Actually, here it would be more appropriate to use a prefix Kolmogorov complexity K( ) instead of C( ), and two-part MDL-style (model+exceptions) descriptions of the sets, since the Kolmogorov complexity C( ) itself is non-monotonic [11] wrt. the string length.


Machine Learning

Training set

Classifier

Machine Training set (with added example)

Learning

Classifer

} Independ. set

Independ. set

(a) Inductive step.

223

difference = ? or distance

(b) Transductive step.

Fig. 1. Transductive reliability estimation – An inductive step is just like an ordinary inductive learning process in Machine Learning. A Machine Learning algorithm is run on the training set, inducing a classifier. A selected example is taken from an independent dataset and classified using the induced classifier. The same example is duplicated, labelled with its assigned class, and finally included into the training set (Fig. 1a). – A transductive step is almost a repetition of an inductive step. A Machine Learning algorithm is run on the changed training set, transducing a classifier. The same example as before is taken from the independent dataset and again classified, now using the transduced classifier (Fig. 1b). Both classifications (represented by probability distributions) of the same example are compared and their difference (distance) is calculated, thus approximating the randomness deficiency. A brief algorithmic sketch is given in Fig. 2.

3.1 Calculating the Difference between Probability Distributions Since a prerequisite for a Machine Learning algorithm is to represent its classifications as a probability distribution over all possible classes, we need a method to measure the difference between two probability distributions. The difference between two probability distributions (over discrete item sets of size N < ∞) can be viewed as a distance between two vectors in RN . In principle, any metric can be used, however not all strict metric properties are required. We require only that the difference measure D between probability distributions P and Q satisfies the following: 1. D(P, Q) ≥ 0 (nonnegativity) 2. 0 ≤ D(P, Q) ≤ ∞, whereD(P, Q) = 0 ⇔ P = Q 3. D(P, Q) = D(Q, P) (symmetry law). In our case P is a probability distribution after the inductive step, and Q is a probability distribution after the transductive step. For calculating the difference between probability distributions, a Kullback-Leibler divergence is frequently used [18]. In our experiments we use a symmetric Kullback-Leibler divergence.

224


Requires:

Machine Learning classifier, a training set and an unlabelled test example Ensures: Estimation of test example’s classification reliability 1: Inductive step: • • • •

train a classifier from the provided training set select an unlabelled test example and classify this example with an induced classifier label this example with a predicted class temporarily add the newly labelled example to the training set

2: Transductive step: • train a classifier from the extended training set • select the same unlabelled test example as above and classify this example with a transduced classifier 3: Calculate a randomness deficiency approximation as a difference between inductive and transductive classification. 4: Calculate the reliability of classification as 2−difference .

Fig. 2. The algorithm for transductive reliability estimation

3.2 Kullback-Leibler Divergence Kullback-Leibler divergence, also frequently referred to as a relative entropy or Idivergence, is defined between probability distributions P and Q n

pi qi

I(P, Q) = − ∑ pi log2 i=1

(6)

Symmetric Kullback-Leibler divergence, or J-divergence, is defined between probability distributions P and Q J(P, Q) = (I(P, Q) + I(Q, P)) =

n

pi

∑ (pi − qi) log2 qi

(7)

i=1

J(P, Q) is limited to the interval [0, ∞], with J(P, P) = 0. Similarly to the p-values of the universal Martin-Löf randomness test (Eq. 5), we calculate our reliability estimation as Rel(P, Q) = 2−J(P,Q)

(8)

However, measuring the difference between probability distributions does not always perform well. There are at least a few exceptional classifiers (albeit trivial ones) where our original approach utterly fails. 3.3 The Curse of Trivial Models So far we have implicitly assumed that the model used by the classifier is good (at the very least better than random). Unsurprisingly, our approach works very well with


225

random classifiers (probability distributions are randomly calculated) by effectively labelling their classifications as unreliable [8]. On the other hand, there also exist simple constant and majority classifiers. A constant classifier is such that it classifies all examples into the same class Ck with probability 1. In such cases our approach always yields reliability 1 since there is no change in probability distribution. A majority classifier is such that it classifies all examples into the same class Ck that is the majority class in the training set. Probability distribution is always the same and corresponds to the distribution of classes in the training set. In such cases our approach yields reliability very close to 1 since there is almost no change in probability distribution (only for the example in question), that is at most for 1/N, where N is number of training examples. In large datasets this change is negligible. Note that such extreme cases do occur in practice and even in real life. For example, a physician that always diagnoses an incoming patient as ill is a constant classifier. On the other hand, a degenerated – overpruned – decision tree (one leaf only) is a typical majority classifier. In both cases all classifications are seemingly completely reliable. Obviously we also need to take in account the quality of classifier’s underlying model and appropriately change our definition of reliability. If we review our original definition of reliability (Eq. 1) it is immediately obvious that we assumed that the model was good. Our reliability estimations actually estimate the conditional reliability with respect to the model M Rel(yi |M) = P(yi is a true class of xi | model M is good)

(9)

To calculate required unconditional reliability we apply the conditional probability theorem for the whole model Rel (yi ) = P(model M is good) ∗ P(yi is true class of xi | model M is good)

(10)

or even better for the partial models for each class yi Rel (yi ) = P(model M is good for yi ) ∗ P(yi is true class of xi | model M is good for yi ) (11) Now we only need to estimate the unconditional probabilities P(model is good)

or ∀i : P(model is good for yi )

(12)

In Machine Learning we have many methods to estimate the quality of the induced model, e.g. a cross-validation computation of classification accuracy is suitable for estimation of Eq. 12. However it may be better to calculate it in a less coarse way, since at this point we already know the predicted class value (yi ). We propose a calculation of (Bayesian) probability that the classification in a certain class is correct. Our approach is closely related to the calculation of post-test probabilities in medical diagnostics [3, 13]. Required factors can be easily estimated from the confusion matrix (Def. 1) with internal testing. Definition 1. A confusion matrix (CM) is a matrix of classification errors obtained with an internal cross validation or leave-one-out testing on the training dataset. The i j-th element cij stands for the number of classifications to the class i that should belong to the class j.

226


Definition 2. Class sensitivity and specificity are a generalization of sensitivity (true positives ratio) and specificity (true negatives ratio) values for multi-class problems. Basically, for N classes we have N two-class problems. Let C p be a correct class in certain case, and C a class, predicted by the classifier in the same case. For each of possible classes Ci , i ∈ {1..N}, we define its class sensitivity Se(Ci ) = P(C = Ci |C p = Ci ) = Ci |C p = Ci ) as follows: and its class specificity Sp(Ci ) = P(C Se(Ci ) = P(C = Ci |C p = Ci ) =

cii ∑ j ci j

(13)

= Ci |C p = Ci ) = Sp(Ci ) = P(C

∑ j =i c ji ∑ j =i ∑k c jk

(14)

Class conditional probability is calculated for each class Ci , given its prior probability P(Ci ), approximated with the prevalence of Ci in the training set, its class specificity (Sp) and sensitivity (Se): Pcond (Ci ) =

P(Ci )Se(Ci ) P(Ci )Se(Ci ) + (1 − P(Ci))(1 − Sp(Ci ))

(15)

For a fixed model and a fixed class Ci its class sensitivity and specificity are typically interdependent according to the ROC (receiver operating characteristics) curve (Fig. 3). An important advantage of class conditional probability over classification accuracy is that it takes in account both classifier’s characteristics and prevalence of each class individually (Fig. 3). It is non-monotonic over all classes and therefore better describes the classifier’s performance in its problem space. To calculate the reliability estimation we therefore need the probability distributions P and Q, and index i = argmax P that determines the class with max. probability (Ci ). According to the Eq. 11 we calculate the reliability estimations by Rel(P, Q;Ci ) = Pcond (Ci ) × 2−J(P,Q)

(16)

Multiplication by class conditional probabilities accounts for basic domain characteristics (prevalence of classes) as well as classifier’s performance. This includes class sensitivity and specificity, and it is especially useful in an automatic setting for detecting possible anomalies such as default (either majority or constant classifiers) that – of course – cannot be trusted. It is easy to see that in this case we have one class with sensitivity 1 and specificity 0, whereas for all other classes we have sensitivity 0 and nonzero specificity. In the first case, the class post-test probability is equal to its prior probability, whereas in the second case it is 0. 3.4 Reliable and Unreliable Classifications Since the datasets used for training classifiers vary in their representativeness and noise levels as well as Machine Learning algorithms vary in strength and assumptions of their underlying models, it is hard to obtain absolute thresholds for reliable classifications. In our experiments they varied between 0.20 and 0.70 for different domains and Machine Learning algorithms. Therefore it is useful to calibrate our criteria in advance by


227

utilizing the training dataset. On the training set, an internal cross validation or (better) leave-one-out testing is performed. For each training example a reliability estimation is made and the predicted as well as the exact class is known. In fact, we now have a new dataset with two possible classes {incorrectly-classified, correctly-classified}, and a single numeric attribute {reliability-estimation}. On this meta-problem we perform binary discretization of the reliability estimation attribute by maximizing the information gain of the split [4] with our goal being to obtain as pure subsets as possible. The best threshold T for the dataset split is calculated by maximizing Eq. 19. H(S) = entropy of the set S S2 S1 H(S; T ) = H(S1 ) + H(S2 ) (entropy after split) S S Gain(S, T ) = H(S) − H(S; T)

(17) (18) (19)

In the set S1 there are unreliable examples {x : Rel(x) < T } whereas in the set S2 there are reliable examples {x : Rel(x) ≥ T }. An experimental result for a dataset split is presented in Fig. 4. Note that internal testing must be done only once during the preparation for transductive reliability estimation. During this calculation we may also conveniently calculate necessary frequencies needed for model quality estimations (Def. 1).

Fig. 3. Class conditional probabilities with respect to the ROC curve and the prior probability (P) of the class

Fig. 4. Reliability estimations in domain “Diabetes” using Backpropagation neural networks. To the left of the possible two boundaries are unreliable classifications, to the right are the reliable classifications

228


4 Experiments To validate our proposed methodology we performed extensive experiments with 6 different Machine Learning algorithms – naive and semi naive Bayesian classifier [7], backpropagation neural network [15], K-nearest neighbour, locally naive Bayesian classifier (a combination KNN and naive Bayesian classifier) [8], Assistant (ID3-like decision trees) [6] on 14 well-known benchmark datasets (Tab. 1a and 1b). All algorithms were modified to represent their classifications as probability distributions. As a reference method the assigned classifier’s probability was used. We performed two comparisons. Firstly, we tested how well can the original populations be split in the subpopulations of correctly and incorrectly classified examples. We applied Kolmogorov-Smirnov and χ2 statistical tests. In all cases the difference between the two populations was significant with p < 0.05, in most cases even with p 0.01. So

Table 1. Experimental results with transductive reliability estimation on 14 domains and 6 ML algorithms, obtained with leave one out testing Domain Mesh Breast cancer Nuclear Diabetes Heart Hepatitis Iris Chess endgame LED Lymphograpy Primary tumor Rheumatology Soybean Voting

Inf. gain Inf. gain Relative Kolmogorov- χ2 -test (Symm. K-L) (class prob.) improvement Smirnov test 0.32 0.18 87.97% < 0.01 < 0.01 0.14 0.06 142.76% < 0.01 < 0.01 0.11 0.06 88.48% < 0.01 < 0.01 0.23 0.09 195.44% < 0.01 < 0.01 0.13 0.12 11.45% < 0.01 < 0.01 0.15 0.10 52.43% < 0.01 < 0.01 0.18 0.15 33.98% < 0.01 < 0.01 0.07 0.04 145.28% < 0.01 < 0.01 0.08 0.06 10.93% < 0.01 < 0.01 0.13 0.10 30.66% < 0.01 < 0.01 0.22 0.13 78.54% < 0.01 < 0.01 0.29 0.15 105.28% < 0.01 < 0.01 0.17 0.11 83.05% < 0.01 < 0.01 0.11 0.09 20.31% < 0.01 < 0.01 (a) Average results on different domains

ML Inf. gain Inf. gain Relative Kolmogorov- χ2 -test algorithm (Symm. K-L) (class prob.) improvement Smirnov test Naive Bayes 0.18 0.11 82.31% < 0.01 < 0.01 Semi naive Bayes 0.16 0.10 56.31% < 0.01 < 0.01 Neural network 0.20 0.08 169.38% < 0.01 < 0.05 K-nearest neighbour 0.13 0.09 55.19% < 0.05 < 0.01 KNN + Naive Bayes 0.16 0.12 43.10% < 0.01 < 0.01 Assistant 0.15 0.11 32.26% < 0.01 < 0.01 (b) Average results of different Machine Learning algorithms


229

the splitting criterion introduced in Sec. 3.4 really produces statistically significantly different subpopulations. Secondly, we measured the improvement of our methodology over the assigned classifier’s probability. For both methods we compared information gains (Sec. 3.4) that directly correspond to the (im)purity of the split subpopulations. Results are summarized by domains (Tab. 1a) and Machine Learning algorithms (Tab. 1b). As it is clearly visible from the results, relative improvements were always in favour of transductive reliability estimation. After the split, the subpopulations were much purer than the original one, information gain (Eq. 19) was on average increased by 75%, ranging between 11% and 195%. All improvements were statistically significant using a two-tailed t-test with p < 0.05. We also performed an in-depth comparison of transductive reliability estimations and physicians’ reliability estimations in the nuclear dataset (nuclear diagnostics of Coronary Artery Disease), where expert physicians were available for cooperation[10]. Our method increased the number of correctly reliable classifications by 22.5% while the number of incorrectly marked as reliable classifications remaind the same [9]. It is estimated that such results if applicable in practice would reduce the costs of diagnostic process by 10%!

5 Discussion We propose a new methodology for transductive reliability estimations of classifications within Machine Learning framework. We provide a theoretical framework for our methodology and an efficient implementation in conjunction with any Machine Learning algorithm that can represent its predictions as probability distributions. We show that in certain extreme cases our basic approach fails and provide improvements that account for such anomalous cases. We argue that, especially in risk-sensitive applications, any serious Machine Learning tool should use a similar methodology for the assessment single of classification reliability. Another use of reliability estimations is in combining answers from different predictors, weighed according to their reliability. Our experiments in benchmark domains show that our approach is significantly better than evaluating classifier’s posterior probabilities. Experimental results of reliability estimations in the Coronary Artery Disease diagnostics also show enormous potential of our methodology. The potential improvements in diagnostic process are so big that the physicians are seriously considering introducing this approach in everyday diagnostic practice. There are several things that can be done to further develop our approach. Currently we aim to replace the discretization of reliability estimation values for obtaining a threshold value. We intend to replace it with proprietary population statistics that would hopefully eliminate impact of differently representative datasets and model weaknesses on resulting quantitative reliability estimation values.

230


Acknowledgements We thank dr. Ciril Groˇselj, from the Nuclear Medicine Department, University Medical Centre Ljubljana, for his work while collecting the nuclear data and interpreting the results, and the anonymous reviewers for their insightful comments. This work was supported by the Slovenian Ministry of Education, Science and Sports.

References [1] S. D. Bay and M. J. Pazzani. Characterizing model errors and differences. In Proc. 17th International Conf. on Machine Learning, pages 49–56. Morgan Kaufmann, San Francisco, CA, 2000. 220 [2] A. Blum and T. Mitchell. Combining labeled and unlabeled data with co-training. In Proceedings of the 11th Annual Conference on Computational Learning Theory, pages 92– 100, 1998. 220 [3] G. A. Diamond and J. S. Forester. Analysis of probability as an aid in the clinical diagnosis of coronary artery disease. New England Journal of Medicine, 300:1350, 1979. 219, 225 [4] J. Dougherty, R. Kohavi, and M. Sahami. Supervised and unsupervised discretization of continuous features. In Proc. ICML’95, pages 194–202. Morgan Kaufmann, 1995. 227 [5] A. Gammerman, V. Vovk, and V. Vapnik. Learning by transduction. In Proceedings of the 14th Conference on Uncertainty in Artificial Intelligence, pages 148–155, Madison, Wisconsin, 1998. 221, 222 ˇ ˇ [6] I. Kononenko, E. Simec, and M. Robnik-Sikonja. Overcoming the myopia of inductive learning algorithms with ReliefF. Applied Intelligence, 7:39–55, 1997. 228 [7] I. Kononenko. Semi-naive Bayesian classifier. In Y. Kodratoff, editor, Proc. European Working Session on Learning-91, pages 206–219, Porto, Potrugal, 1991. Springer-Verlag. 228 [8] M. Kukar. Estimating classifications’ reliability. PhD thesis, University of Ljubljana, Faculty of Computer and Information Science, Ljubljana, Slovenia, 2001. In Slovene. 220, 225, 228 [9] M. Kukar. Making reliable diagnoses with machine learning: A case study. In Silvana Quaglini, Pedro Barahona, and Steen Andreassen, editors, Proceedings of Artificial Intelligence in Medicine Europe, AIME 2001, pages 88–96, Cascais, Portugal, 2001. Springer. 221, 229 [10] M. Kukar, I. Kononenko, C. Groˇselj, K. Kralj, and J. Fettich. Analysing and improving the diagnosis of ischaemic heart disease with machine learning. Artificial Intelligence in Medicine, 16 (1):25–50, 1999. 229 [11] M. Li and P. Vitányi. An introduction to Kolmogorov complexity and its applications. Springer-Verlag, New York, 2nd edition, 1997. 221, 222 [12] K. Nigam, A. K. McCallum, S. Thrun, and T. Mitchell. Text classification from labeled and unlabeled documents using EM. Machine Learning, 39 (2/3):103–134, 2000. 220 [13] M. Olona-Cabases. The probability of a correct diagnosis. In J. Candell-Riera and D. Ortega-Alcalde, editors, Nuclear Cardiology in Everyday Practice, pages 348–357. Kluwer, 1994. 225 [14] J. Ortega, M. Koppel, and S. Argamon. Arbitrating among competing classifiers using learned referees. Knowledge and Information Systems Journal, 3:470–490, 2001. 220 [15] D. E. Rumelhart and J. L. McClelland. Parallel Distributed Processing, volume 1: Foundations. MIT Press, Cambridge, 1986. 228


231

[16] C. Saunders, A. Gammerman, and V. Vovk. Transduction with confidence and credibility. In Proceedings of the International Joint Conference on Artificial Intelligence, Stockholm, Sweden, 1999. 219, 221, 222 [17] A. Seewald and J. Furnkranz. An evaluation of grading classifiers. In Proc. 4th International Symposium on Advances in Intelligent Data Analysis, pages 115–124, 2001. 220 [18] I. J. Taneja. On generalized information measures and their applications. Adv. Electron. and Elect. Physics, 76:327–416, 1995. 223 [19] K. M. Ting. Decision combination based on the characterisation of predictive accuracy. Intelligent Data Analysis, 1:181–206, 1997. 219 [20] V. Vapnik. Statistical Learning Theory. John Wiley, 1998. 220 [21] V. Vovk, A. Gammerman, and C. Saunders. Machine learning application of algorithmic randomness. In Proceedings of the 16th International Conference on Machine Learning (ICML’99), Bled, Slovenija, 1999. 219, 221, 222

Robustness Analyses of Instance-Based Collaborative Recommendation Nicholas Kushmerick Computer Science Department, University College Dublin [email protected]

Abstract. Collaborative recommendation has emerged as an effective technique for a personalized information access. However, there has been relatively little theoretical analysis of the conditions under which the technique is effective. We analyze the robustness of collaborative recommendation: the ability to make recommendations despite (possibly intentional) noisy product ratings. We formalize robustness in machine learning terms, develop two theoretically justified models of robustness, and evaluate the models on real-world data. Our investigation is both practically relevant for enterprises wondering whether collaborative recommendation leaves their marketing operations open to attack, and theoretically interesting for the light it sheds on a comprehensive theory of collaborative recommendation.

1

Introduction

Collaborative recommendation has emerged as an effective personalization technique for a diverse array of electronic commerce and information access scenarios (eg, [10,5]). Such systems keep track of their customers’ preferences, and use these data to offer new suggestions. Many variations have been explored, but the basic idea is as follows: to recommended items to a target customer, the system retrieves similar customers, and then recommends items that were liked by the retrieved customers but not yet rated by the target. Collaborative recommendation has been empirically validated for many domains (eg, [3]), and has been successfully deployed in many commercial settings. However, despite some interesting efforts [7,4,2], there is no general theoretical explanation of the conditions under which a particular collaborative recommendation application will succeed or fail. Our goal is to complement existing theoretical work by investigating the robustness of collaborative recommendation. Informally, robustness measures how sensitive the technique is to changes in the customer/product rating matrix. In particular, we analyze the situation in which a malicious agent attacks a recommender system by posing as one or more customers and submitting bogus product ratings. Our analysis is designed to rigorously quantify the extent to which a malicious agent can force the recommender system to give poor recommendations to its “genuine” customers. The theoretical results reported in this paper builds on an ongoing empirical investigation of this issue [9,8]. T. Elomaa et al. (Eds.): ECML, LNAI 2430, pp. 232–244, 2002. c Springer-Verlag Berlin Heidelberg 2002

Robustness Analyses of Instance-Based Collaborative Recommendation

233

Our primary motivation is to gain deeper insights into the principles underlying effective collaborative recommendation. However, our work is also relevant for a second, more practical reason: recommender systems can represent an insecure back-door into an enterprise’s marketing operations. To lock this door, some enterprises impose substantial charges on customers to submit ratings (eg, a bookstore might only accept ratings for books that have been purchased). However, many collaborative recommenders are open Web services that malicious agents could easily attack. How much damage can they inflict? We make three contributions. First, we formalize robustness in machine learnings terms, and introduce a novel form of class noise that models an interesting suite of attacks (Sec. 2). Second, we develop two models that predict the change in accuracy as a function the number of fake ratings that have been inserted into the customer/product matrix (Secs. 3–4). Third, we empirically evaluate our predications against real-world collaborative recommendation data (Sec. 5).

2

Definitions

Our analysis on collaborative recommendation assumes the standard k-NN learning algorithms. Other (eg, model-based) approaches have been tried but k-NN is accurate, widely used and easily analyzed. In this approach, each customer is represented as a vector of product ratings (many of which will be empty for any particular customer). Unlike traditional machine learning settings, the “class” is not a distinguished attribute, but corresponds to the product that the system is contemplating for recommendation. We model an attack as the addition of noise to the training data. In general, this noise could be associated with either the attributes, the class or both. We focus exclusively on class noise, and defer attribute noise to future work. We are not concerned with malicious noise as defined by [6], because we can safely assume that the attacking agent is not omniscient (eg, the agents can not directly inspect any ratings except their own). We model attacks with a relatively benign noise model that we call biased class noise. This model is characterized by the following parameters: the noise rate β, and the class bias µ. Noise is added according to the following process. First, an instance is generated according to the underlying distribution. With probability 1 − β, the instance is noise-free and labeled by the target concept. Otherwise, with probability βµ the instance is labeled 1 and with probability β(1 − µ) the instance is labeled 0. The biased class noise model is useful because it can represent a variety of stereotypical attacks. For example, a book’s author could try to force recommendations of his book by pretending to be numerous customers who all happen to like the book. We call this a “push” attack and it corresponds to µ = 1. Alternatively, the author’s arch-enemy could insert fake customer profiles that all dislike the book; this “nuke” attack is modeled with µ = 0. We are interested in robustness: the ability of the recommender to make good recommendations in spite of an attack. There are two aspects to robust-

234

Nicholas Kushmerick

ness. First, we may be concerned with accuracy: are the products recommended after the attack actually liked? The second issue is stability: does the system recommend different products after the attack (regardless of whether customers like them)? While stability and accuracy are distinct, they are not orthogonal. For example, if a recommender has perfect accuracy for a given task both with and without noise, then it must be perfectly stable. On other the hand, consider a product that no-one likes. The recommendation policies “recommend to no-one” and “recommend to everyone” are both perfectly stable, yet the first is always correct and the second is always wrong. Our analysis of robustness focuses exclusively on accuracy. Before proceeding, we introduce some additional notation. We assume a ddimensional instance space X d . Without loss of generality, we assume X = [0, 1]. In the context of recommendation, each dimension corresponds to one of d products, and the value on the dimension is a numeric rating. The nearest neighbor approach requires a function dist(·, ·) defined over X d ×X d, but our analysis does not depend on any particular distance metric. Finally, let H be a hypothesis, C be a concept, and D be a probability distribution over X d . The error rate of H with respect to C and D is defined as E(H, C, D) = Prx∈D (H(x) = C(x)).

3

Absolute Accuracy

The first model extends Albert and Aha’s noise-free PAC results for k-NN [1] to handle biased class noise. We first review these noise-free results, and then state our model as Theorem 1. The key idea behind Albert and Aha’s (hereafter: AA) analysis is that of a “sufficiently dense” sample from the instance space. Informally, a subset S ⊂ X d is dense if most of the points in the entire space X d are near many points in the sample S. The terms “most”, “near” and “many” are formalized as follows: Let D be a distribution over X d . A subset S ⊆ X d is (k, α, γ)-dense if, except for a subset with probability less than γ under D, for every x ∈ X d , there exists at least k distinct points x1 , . . . , xk ∈ S such that dist(x, xi ) ≤ α for each i. Given this definition, AA derive [1, Lemma 2.2] a lower bound Υd (k, α, γ, |S|) Υd (k, α, γ, |S|) = on the probability that a sample S of X d is (k, α, γ)-dense: t d B (max {s/t, ρ} , s, t), 1 − m Φ (ρ, k , |S|), where Φ (ρ, t, s) = 2 0≤k 1 − δ for |S|, in order to show that k-NN can PAC-learn efficiently. Since robustness is orthogonal to efficiency, we ignore this part of their analysis.

236

Nicholas Kushmerick

correctly, and Sbad = S \Sgood ⊆ Sattack be the instances that are actually labelled incorrectly. Again, we can not know Sgood or Sbad but we do know that the number of noise-free instances is |Sgood | = (1 − β)|S| + β|S|(f µ + (1 − f )(1 − µ)) ≥ |Sreal |. and the number of noisy instances is |Sbad | = β|S|(1 − f µ − (1 − f )(1 − µ)) ≤ |Sattack |. If λ = β(µ+ f − 2µf ) is the effective attack size, then |Sgood | = (1 − λ)|S| and |Sbad | = λ|S|. We require that both Sgood be ( k/2, α, γ)-dense and Sbad be (k/2, α, γ)sparse. Since these events are independent, the probability of their conjunction is the product of their probabilities. Therefore, if we can determine appropriate values for the distance thresholds α1 and α2 and probability thresholds γ1 and γ2 , then we have that the accuracy of k-NN when training on S with biased class noise is at least Υd ( k/2, α1 , γ1 , |Sgood |) · Υs (k/2, α2 , γ2 , |Sbad |). In Appendix A we prove the following theorem. Theorem 1 (Absolute accuracy). The following holds for any , β, µ, d, k, L, B, C ∈ CL and D ∈ DB . Let S be a sample of X d according to D with biased class noise rate β, and let H = k-NN(S). Then we have that Pr [E(H, C, D) < ] ≥ Υd ( k/2, /4LB, /4, (1 − λ)|S|) · Υs (k/2, /4LB, /4, λ|S|) , where λ = β(µ + f − 2µf ), and f is the fraction of X d labeled 1 by C. To summarize, Theorem 1 yields a worst-case lower bound on the accuracy of k-NN under biased class noise. On the positive side, this bound is “absolute” in the sense that it predicts (a probabilistic bound on) the actual error E(k-NN(S), C, D) as a function of the sample size |S|, noise rate β, and other parameters. In other words, the term “absolute” draws attention to the fact that this model takes account of the actual position along the learning curve. Unfortunately, like most PAC analyses, its bound is very weak (though still useful in practice; see Sec. 5).

4

Approximate Relative Accuracy

In contrast, the second model does not rely on a worst-case analysis and so makes tighter predictions than the first model. On the other hand, the model is only “approximate” because it makes two assumptions. First, it assumes that the training sample is large enough that the learning curve has “flattened out”. Second, it assumes that, at this flat part of the learning curve, k-NN achieves perfect accuracy except possibly on the boundary of the target concept. We call this second model “approximate” to draw attention to these assumptions, and “relative” to note specifically that it does not predict error on an absolute scale. To formalize these assumptions, let S be a training sample drawn from the distribution D over X d , and let C be the target concept. Let S be the fraction 1 − β of the instances in S that were (correctly) labeled by C during the biased class noise process. Let D be the distribution that is proportional to D except


237

that D [x] = 0 for all points x on the boundary between C and X d \ C. The assumptions of the second model can be expressed as: E(k-NN(S ), C, D ) = 0

(1)

Given this assumption, we can predict the error of k-NN as follows. To classify an instance x using a training set S, k-NN predicts the majority class of the k instances x1 , . . . , xk ∈ S that are closest to x. To classify x correctly, at least

k/2 of these k instances must have the correct class. If we randomly draw from D a point x ∈ X d , there are two cases: either C(x) = 1 (which happens with probability f ), or C(x) = 0 (which happens with probability 1 − f ), where as above f is the probability under D that C(x) = 1. In the first case, we need to have at least k/2 successes out of k trials in a Bernoulli process where the probability of success is equal to the probability that a neighbor xi of x will be labeled 1. We can calculate this probability as (1 − β) + βµ. The first term is the probability that xi is labeled 1 and xi ∈ S ; by (1), we know that this probability is 1 − β. The second term is the probability that xi is labeled 1 and xi ∈ S , by definition of the biased class noise process labels we know that this probability is βµ. In the second case, again we need at least k/2 successes, but with success probability (1 − β) + β(1 − µ), the probability that a neighbor xi of x will be labeled 0. The first term is the probability that xi is labeled 0 and xi ∈ S , and by (1) this happens with probability 1 − β. The second term is the probability that xi is labeled 0 and xi ∈ S , which occurs with probability β(1 − µ). The following theorem follows from this discussion. Theorem 2 (Approximate relative accuracy). The following holds for any β, µ, d, k, C and D. Let S be a sample of X d according to D with biased class noise rate β. Let S and D be as defined above. If assumption (1) holds, then E(k-NN(S), C, D ) = k k k k 1 − f· B(1 − β(1 − µ), k , k) − (1 − f ) · B(1 − βµ, k , k), k k k = k k = k 2 2 where f is the fraction of X d labeled 1 by C. Without more information, we can not conclude anything about E(k-NN(S), C, D) (which is what one can measure empirically) from E(k-NN(S), C, D ) (the model’s prediction) or from E(k-NN(S ), C, D ) = 0 (the assumption underlying the model). For example, if D just so happens to assign zero probability to points on C’s boundary, then E(k-NN(S ), C, D) = E(k-NN(S ), C, D ) and so in the best case E(k-NN(S), C, D) = 0. On the other hand, if all of D’s mass is on C’s boundary then in the worst case E(k-NN(S), C, D) = 1. Furthermore, it is generally impossible to know whether E(k-NN(S ), C, D ) = 0. Despite these difficulties, we will evaluate the model on real-world data by simply assuming E(k-NN(S ), C, D ) = 0 and D = D, and comparing the predicted and observed error.

238

Nicholas Kushmerick

+ ✸ ✷

1

+ ✸ ✷

+ ✸ ✷

✷

✷

✷

✸ ✷ 1 +

✸

0.8

+ ✸ ✷

+ ✸ ✷

+ ✸ ✷

+ ✸ ✷ + ✸ ✷

k = 10

✸ + ✷

L−Aabs (β) L−Aabs (1)

L−Aabs (1)

L−Aabs (β)

µ=1 µ=0 µ = 0.5

+ ✸ ✷

0.8

k = 10 1

0.6

+ ✸ ✷

0.4

0.2

µ=1 µ=0 µ = 0.5

0.6

✸ + ✷

+ ✸ ✷

0.4

0.2 + ✸ ✷ +

0 0

0.2

+ ✸

0.4 0.6 noise rate β

+ ✸

+ ✸ ✷

+ ✸ ✷

0

0.8

1

0

0.2


0.8

1

Fig. 1. MUSHROOM: Empirical (left) and predicted (right) absolute accuracy ✸ 1 +

✸ +

+ ✸

✸ +

✸ 1 +

✸ +

+ ✸

+ ✸

0.8

0.8 k = 10 ✸ +

L−Aabs (β) L−Aabs (1)

µ=1 µ=0

0.6

+ ✸

+ ✸

+ ✸ + ✸

k = 10

L−Aabs (1)

L−Aabs (β)

+ ✸

+ ✸

0.4

µ=1 µ=0

0.6

✸ +

+ ✸

0.4

0.2

0.2

0

0

+ ✸

0

0.2


0.8

1

+ ✸

0

0.2


0.8

1

Fig. 2. PTV: Empirical (left) and predicted (right) absolute accuracy

5

Evaluation

We evaluated the two models against two real-world learning tasks: – The MUSHROOM data-set from the UCI repository contains 8124 instances with 23 attributes, with no missing values. – The PTV collaborative recommendation data for television listing [www.ptv.com] contains 2344 instances (people) and 8199 attributes (television programs), and only 0.3% of the matrix entries are non-null. We discarded people who rated fewer than 0.05% of the programs, and programs rated by fewer than 0.05% of the people. The resulting 241 people and 570 programs had a sparseness of 15.5%. The original ratings (values from 1–4) were converted into binary attributes (‘like’/‘dislike’). We used the standard k-NN algorithm with no attribute or vote weighting. Distance was measured using the Euclidean metric (ignoring non-null attributes). All experiments use k = 10 neighbors. Our experiments use a variation on the standard cross validation approach. We repeat the following process many times. First, we randomly partition the entire set of instances into a real set R, a fake set F , and a testing set T . To implement the biased class noise model, a noisy set N containing β|R|/(1 − β)

Robustness Analyses of Instant-Based Collaborative Recommendation

259

Fig. 3. MUSHROOM: Empirical (left) and predicted (right) relative accuracy instances is then randomly drawn from F. The class attributes of the instances in N are then modified t o 1 with probability p and 0 with probability 1 - p. The I;-NN learning algorithm is then training on R U N (so the noise rate is N / R U N = p). We measure accuracy as the fraction of correct predictions for the test instances in T. For MUSHROOM, noise is added only to the class attribute defined by the data-set's authors. For PTV the class attribute (i.e., program to attack) is selected randomly.

Absolute accuracy model. The absolute accuracy model predicts Pr[E(I;-NN(Sp), C, D)

< t],

the probability that the accuracy exceeds 1- t , where So is a sample with biased class noise rate p. Let the model's predicted absolute accuracy from Theorem 1 be

A

Our empirical estimate Aabs(p) of this probability is simply the fraction of trials for which t exceeds the error. Recall that Theorem 1 requires a bound L on the perimeter of the target concept, and a bound B on the probability of any instance under the distribution D . Thus our model is not completely general, and furthermore it is difficult to estimate these parameters for a given learning task. However, it is easily shown that for small values of I;, Aabs(p) does not depend on L and B, and thus we do not need to tune these parameters of our model for each learning task. Due to the worst-case analysis, typically Aabs(P) >> 1 - clearly an absurd value. However, for the purposes of analysing robustness, such values are useful, because we are interested in the increase in error at noise rate P compared to p = 0. We therefore report results using the ratios (L - A,b,(p))/(L - A,b,(O)) and (L - X a b s ( p ) ) / ( ~- Xabs(0)), where L = A,b,(l) is a constant chosen to scale the ratios to [0,1]. The results for MUSHROOM with t = 0.25 are shown in Fig. 1. The predicted and observed accuracies agree reasonably well, even accounting for the fact that

240

Nicholas Kushmerick

Fig. 4. PTV: Empirical (left) and predicted (right) absolute accuracy

the data have been scaled t o [0,1]. The fit is by no means perfect, but we are satisfied with these results, since worst-case PAC-like analyses are usually so weak as to be incomparable to real data. Fig. 2 shows the results for P T V with t = 0.3. Here the fit is worse: P T V appears to be much more robust in practise than predicted, particular as P increases. We conjecture that this is due to the fact that the P T V data is highly noisy, but further analysis is needed to explain these data. Relative accuracy model. The relative accuracy model predicts &(I;-NN(So),C,D), the error of I;-NN when trained on a sample So with biased class noise rate P. Let A,,l(P) be the model's prediction from Theorem 2. Our empirical estimate X r e l ( ~ of ) this probability is simply the fraction of incorrectly classified test instances. As before, we scale all data to [O-11. The results for MUSHROOM are shown in Fig. 3 and the P T V results are shown in Fig. 4. The model fits the observed data quite well in both domains, though as before P T V appears to be inherently noisier than MUSHROOM.

6

Related Work

Collaborative recommendation has been empirically validated in numerous standard lLcustomer/product" scenarios [3]. However, there is relatively little theoretical understanding of the conditions under which the technique to be effective. Our work is highly motivated by ongoing empirical investigations of the robustness of collaborative filtering [9,8]. The ideas underlying Theorem 1 borrow heavily &om Albert et al's seminal PAC analysis of noise-free I;-NN [I]. There has been substantial theoretical algorithmic work on collaborative filtering [7;2,4]. For example, Azar et a1 [2] provide a unified treatment of several information retrieval problems, including collaborative filtering, latent semantic analysis and link-based methods such as hubs/authorities. They cast these problems as matrix reconstruction: given a matrix of objects and their attributes (eg, for collaborative filtering, the objects are products, the attributes are customers, and matrix entries store customers' ratings) from which some entries have been


241

deleted, the task is to reconstruct the missing entries (eg, predict whether a particular customer will like a specific product). Azar et al prove that the matrix entries can be efficiently recovered as long as the original data has a good low-rank approximation. The fundamental difference between all of these results and ours is that our biased class noise model is more malicious than simple random deletion of the matrix entries. It remains an open question whether these results can be extended to accommodate this model

7

Discussion

Collaborative recommendation has been demonstrated empirically, and widely adopted commercially. Unfortunately, we do not yet have a general predictive theory for when and why collaborative filtering is effective. We have investigated one particular facet of such a theory: an analysis of robustness, a measure of a recommender system’s resilience to potentially malicious perturbations in the customer/product rating matrix. This investigation is both practically relevant for enterprises wondering whether collaborative filtering leaves their marketing operations open to attack, and theoretically interesting for the light it sheds on a comprehensive theory of collaborative filtering. We developed and evaluated two models for predicting the degradation in predictive accuracy as a function of the size of the attack and other parameters. The first model uses PAC-theoretic techniques to predict a bound on accuracy. This model is “absolute” in that it takes account of the exact position of the system along the learning curve, but as a worst-case model it is problematic to evaluate its predictions . In contrast, the second model makes tighter predictions, but is “relative” in the sense that it assumes perfect prediction in the absence of the malicious attack. Our preliminary evaluation of the model against two realworld data-sets demonstrates that our model fits the observed data reasonably well.

Acknowledgments I thank M. O’Mahony, N. Hurley, G. Silvestre and M. Keane for helpful discussion, B. Smyth for the PTV data, and the Weka developers. This research was funded by grant N00014-00-1-0021 from the US Office of Naval Research, and grant SFI/01/F.1/C015 from Science Foundation Ireland.

References 1. M. Albert and D. Aha. Analyses of instance-based learning algorithms. In Proc. 9th Nat. Conf. Artificial Intelligence, 1991. 234, 235, 240, 243 2. Y. Azar, A. Fiat, A. Karlin, F. McSherry, and J. Saia. Spectral analysis of data. In Proc. 32nd ACM Symp. Theory of Computing, 2001. 232, 240

242

Nicholas Kushmerick

3. J. Breese, D. Heckerman, and C. Kadie. Empirical analysis of predictive algorithms for collaborative filtering. In Conf. Uncertainty in Artificial Intelligence, 1998. 232, 240 4. P. Drineas, I. Kerenidis, and P. Raghavan. Competetive recommender systems. In Proc. 32nd ACM Symp. Theory of Computing, 2002. 232, 240 5. D. Goldberg, D. Nichols, B. Oki, and D. Terry. Using collaborative filtering to weave an information tapestry. C. ACM, 35(12):61–70, 1992. 232 6. M. Kearns and M. Li. Learning in the presence of malicious errors. In Proc. ACM Symp. Theory of Computing, 1988. 233 7. R. Kumar, P. Raghavan, S. Rajagopalan, and A. Tompkins. Recommender systems: A probabilistic analysis. In Proc. 39th IEEE Symp. Foundations of Computer Science, 1998. 232, 240 8. M. O’Mahony, N. Hurley, N. Kushmerick, and G. Silvestre. Collaborative recommendation: A robustness analysis. Submitted for publication, 2002. 232, 240 9. M. O’Mahony, N. Hurley, and G. Silvestre. Promoting recommendations: An attack on collaborative filtering. In Proc. Int. Conf. on Database and Expert System Applications, 2002. 232, 240 10. U. Shardanand and P. Maes. Social information filtering: Algorithms for automating “word of mouth”. In Proc. Conf. Human Factors in Computing Systems, 1994. 232

A

Proof of Theorem 1

The following two lemmas are easily proven. Lemma 1. Consider a binomial process where outcome o1 has probability at least ρ and outcome o2 consumes the remaining probability mass. The prob is at most Φ2 (ρ, s, t) = st B ability sof exactly s o1 -outcomes in st trials t−s max t , ρ , s, t , where B(p, s, t) = p (1 − p) . Lemma 2. Consider a trinomial process where outcome o1 has probability at least ρ and outcomes o2 and o3 consume the remaining probability mass. The probability of exactly most r ands s o2-outcomes in t trials is at r o1 -outcomes r s Φ3 (ρ, r, s, t) = rt t−r s T max t , ρ , r, t , s, t , where T (p, r, q, s, t) = p q (1 − p − q)t−r−s . The following lemma bounds the probability that a subset is sparse or dense. Lemma √ 3. The following holds for any d, α, γ, k, and distribution D. Let m = d/α and ρ = mγd . The probability that a sample S of X d drawn according to D is (k, α, γ)-dense is at least Φ2 (ρ, k , |S|), (2) Υd (k, α, γ, |S|) = 1 − md 0≤k , labels yi ∈ Y = {1, 2}, λ (a learning algorithm that computes class probability estimates), L (a loss matrix), x (an unlabeled example), [1] [2] [3] [4] [5] [6] [7] [8]

for t = 1 to T do St := (Bootstrap) sample of S; θ := Train λ(St ); // the learned model 0 (θx ) Pt (0|x) := NN(θ ; Pt (1|x) := 1 − Pt (0|x); x) endfor β := DecisionBoundary(L); P(0|x, L) := P r(P (j|x) < β); P(1|x, L) := 1 − P(0|x, L); endfor

Output: hCP E (x) = argmin y∈Y

1

P(j|x, L)L(y, j)

j=0

// the optimal prediction w. respect to L and P

As base learning algorithm λ we have used decision trees. However, given that we were not only interested in having unbiased estimates of the mean but also a good estimates of the variance of the computed probabilities, we have explored adding different sources of randomness to the original tree learning algorithm: random split selection, and random attribute selection. Breiman has proposed a unified view of these techniques under the name of Random Forests [7] and has analyzed them in the context of 0/1-loss classification. In the case of random splits, during the tree learning procedure, instead of selecting the best potential split, the algorithm will choose a split at random from among the N best potential splits. This procedure was introduced by Dietterich [10] and used classification problems. For the random attribute selection procedure, at each node, a subset of size F of the attributes is selected at random and the best potential split (the one that gives the highest gain ratio) on those attributes is chosen. Amit and Geman [1] have first explored this technique.

4

Experimental Analysis

We have implemented the methods described in the previous section by using Quinlan’s C4.5 decision tree learning algorithm [27] as base learner. The first implementation of C-PE (denoted as Bag) grows each tree using the standard procedure. The second implementation (RS) selects randomly in each node a split from among the ten best splits. The third implementation selects at each

276

Dragos D. Margineantu

node a random subset of the attributes of size F . Two versions of this method were tested: RA-1 (F = 1) and RA-logN (F = log(N ), where M is the number of attributes). Pruning was never used in the algorithms that were tested. We have also implemented Provost and Domingos’ B-PET algorithm to compare the decisions made by the C-PE methods (relying on P(y|x, L)) with the decisions that rely on class probability estimates (P (y|x)). We have tested all algorithms on ten data sets (see Table 3). Except for the Donations-bin data set, all were drawn from the UC Irvine Repository [3]. Donations-bin is the binary version of the KDD Cup 1998 data [2] for which the goal is to determine whether a person has made a donation after a direct mail campaign. The format of the data is similar to the one used in other studies: seven attributes, 95412 instances for training and 96367 instances for testing. Unfortunately, these data sets do not have associated loss matrices L. Therefore, we generated loss matrices at random according to some loss matrix models. Table 4 describes four loss models, M1 through M4. The second column of the table describes how the misclassification costs were generated for the off-diagonal elements of L. In all cases, the costs are drawn from a uniform distribution over some interval. The diagonal values are always 0. Given that the new methods presented here were specifically designed to minimize the loss associated with the classification decisions, we have used the BDeltaCost paired test presented in [23]. Appendix A gives a more detailed description of the test. We have chosen to use the BDeltaCost test rather than ROC methods because the ROC methods give an overall measure of the quality of the rankings, whereas in our case we needed a statistical test for comparing models when the loss matrix is known. In other words, we focus on the analysis of the quality of the decisions of the different models. Table 4. The models employed for generating the loss matrices used in the experData Evaluation iments. Unif[a, b] indicates a Set Size Method uniform distribution over the 95412/96367 test set 699 10-fold xval [a, b] interval. The diagonal el286 10-fold xval ements of the loss matrices are 155 10-fold xval always zero

Table 3. Data sets studied in this paper Name Donations-bin Breast cancer (Wis.) Breast cancer (Yug.) Hepatitis Horse colic King-rook vs. king-pawn Labor negotiations Liver disease Sonar Voting records

200 3196 57 345 208 435

10-fold 10-fold 10-fold 10-fold 10-fold 10-fold

xval xval xval xval xval xval

Loss Model M1 M2 M3 M4

L(i, j) i = j Unif[0, 5] Unif[0, 7] Unif[0, 10] Unif[0, 20]

Class Probability Estimation

277

Performance was evaluated either by 10-fold cross validation or by using a test set (as noted in Table 3). For each cost model we generated ten loss matrices, and performed 10-fold cross validation on the Irvine ML data sets. This gives us 10 (matrices) × 4 (models) × 10 (folds) = 400 runs of the algorithms for each Irvine ML data set. In the case of the Donations-bin data, the evaluation was performed on the test set, resulting in 80 runs. For each of the runs, we performed the BDeltaCost statistical test to determine whether the learned models had statistically significant different expected losses, based on the 95% confidence interval. Initially we have set the number of bagging rounds to be T = 100. We tested separately two versions of the C-PE algorithms. The first version (C-PE-counts) estimates P by using the counts of the individual computed class probabilities P on each side of the decision boundary. The results for the Donations-bin data are shown in Table 5. The results for the Irvine sets are presented in Table 6. Each cell of the tables represents the percentage of wins, ties, and losses (respectively) for the algorithms that are tested. For example, the cell in row RA-1, column B-PET from Table 6 indicates that when RA-1 and B-PET were compared, for 20.2% of the runs RA-1 outperformed B-PET, in 22.3% of the runs B-PET outperformed RA-1 and for 57.5% of the runs BDeltaCost could not reject the null hypothesis based on a 95% confidence interval. The second version of our algorithms (C-PE-normal) estimates P by computing the normal approximation N of P . The results for the Donations-bin data are presented in Table 7. The results for the Irvine sets are shown in Table 8. Next, we tested the influence of the size of the ensemble on the performance of the algorithms. We reran all experiments for T = 50, and T = 200. While, the quality of all C-PE decisions was slightly worse (compared to the B-PETs) for T = 50, it has improved for T = 200 only for the smaller Irvine data sets.

Table 5. Results on Donations-bin for C-PE-counts (T = 100) B-PET Bag RS RA-1 RA-logN 20-75-5 20-75-5 20-80-0 20-80-0 RA-1 15-60-25 20-60-20 0-100-0 RS 15-60-25 20-65-15 Bag 0-80-20

Table 6. Results on the UCI data sets for C-PE-counts (T = 100) B-PET Bag RS RA-1 RA-logN 44.8-42.2-11 17.4-52.5-30.1 17-59.8-23.2 48.2-43.4-8.4 RA-1 20.2-57.5-22.3 9.3-48.9-41.8 6.3-45.1-48.6 RS 42.1-48.7-9.2 22.6-51.9-25.5 Bag 43.2-48.6-8.3

278


Table 7. Results on Donations-bin for C-PE-normal (T = 100) B-PET Bag RS RA-1 RA-logN 20-75-5 25-75-0 25-75-0 20-80-0 RA-1 15-60-25 20-60-20 5-95-0 RS 15-60-25 20-60-20 Bag 0-75-25

Table 8. Results on the UCI data sets for C-PE-normal (T = 100) B-PET Bag RS RA-1 RA-logN 45.9-43.1-11 20.7-48.6-30.7 25-56.8-18.2 44-47-9 RA-1 18.9-58.1-23 9.7-51-39.3 8.7-45.8-45.5 RS 44.4-44.5-11.1 19.8-50.6-29.6 Bag 46.3-46.2-7.5

5

Summary and Conclusions

We have presented a new approach to cost-sensitive classification. The methods that we proposed make a decision not only based on an estimate of the mean of the probabilities computed by the models in the ensemble, but they employ the distribution of individual probability estimates of the classifiers together with the loss matrix. Instead of outputting the average of the individual estimates of the component classifiers the way B-PETs do, the C-PE algorithms compute an estimate of the distribution of class probabilities and makes a decision based on that estimate and the loss function. C-PE provides a mechanism to make accurate cost-sensitive decisions even if accurate class probability estimates are hard or impossible to compute (because of inherent deficiencies of the algorithms, or because of the distribution of the data). C-PE is sensitive not only to the loss function, but also to the hypothesis learned by the base algorithm. In the case of the UCI data sets we can observe that, the RA-logN, RS and Bag versions of C-PE outperform the Bagged Probability Estimation Trees (B-PET). However in the case of the very large Donations data set, B-PET is marginally outperformed only by RA-logN and performs much better than Bag. This shows that for larger data sets, B-PET is able to compute more accurate probabability estimates P (y|x), whereas in the case of smaller data sets the confidence-based estimates are better for different amounts of randomness. If we were to rank the C-PE methods based on the amount of randomness that they add to the procedure, RA-1 adds the largest amount, and the results show that this might lead to larger losses associated with the decisions. The best overall performance belongs to the RA-logN implementation of C-PE. This might be the case because it adds the right amount of randomness to the bagging procedure. It would be interesting to analyze the performance of RS for different values of the number of splits (among which the random selection is made).


279

The experiments also show that a larger value for T (the number of bagging rounds) helps improving the quality of the decisions on the smaller data sets.

6

Discussion

To our knowledge, the only decision making approach that has used a confidence measure for probability estimates was presented in the work of Pednault et al. [24]. Saar-Tsechansky and Provost [28] compute an estimate of the variance of the class probabilities for unlabeled examples to decide on the set of instances to be labeled next, within an active learning procedure. Preliminary experiments show that combining C-PE with uncertainty sampling in a cost-sensitive active learning procedure improves in terms of the number of examples that are needed to achieve similar performance, over a an active learning procedure that relies on probability estimates computed by B-PETs.

Acknowledgement I would like to thank Roberto Altschul, Foster Provost, Claudia Perlich, Tom Dietterich, Rodney Tjoelker, and Pedro Domingos for their comments and discussions on confidence-based decision making and cost-sensitive learning. I would also like to thank Ed Pednault for earlier discussions we had on this topic.

References 1. Y. Amit and D. Geman. Shape quantization and recognition with randomized trees. Neural Computation, 9:1545–1588, 1997. 275 2. S. D. Bay. The UCI KDD archive. University of California, Irvine, Dept. of Information and Computer Sciences, 1999. [http://kdd.ics.uci.edu/]. 276 3. C. L. Blake and C. J. Merz. UCI repository of machine learning databases. University of California, Irvine, Dept. of Information and Computer Sciences, 1998. [http://www.ics.uci.edu/~mlearn/MLRepository.html]. 276 4. J. P. Bradford, C. Kunz, R. Kohavi, C. Brunk, and C. E. Brodley. Pruning decision trees with misclassification costs. In C. Nedellec and C. Rouveirol, editors, Lecture Notes in Artificial Intelligence. Machine Learning: ECML-98, Tenth European Conference on Machine Learning, Proceedings, volume 1398, pages 131–136, Berlin, New York, 1998. Springer Verlag. 272, 273 5. A. P. Bradley. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 30:1145–1159, 1997. 273 6. L. Breiman. Out-of-bag estimation. Technical report, Department of Statistics, University of California, Berkeley, 1998. 273 7. L. Breiman. Random forests. Technical report, Department of Statistics, University of California, Berkeley, 2001. 275 8. L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone. Classification and Regression Trees. Wadsworth International Group, 1984. 272, 273

280


9. B. Cestnik. Estimating probabilities: A crucial task in machine learning. In L. C. Aiello, editor, Proceedings of the Ninth European Conference on Artificial Intelligence, pages 147–149, London, 1990. Pitman Publishing. 273 10. T. G. Dietterich. An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, boosting and randomization. Machine Learning, 40(2):139–158, 2000. 275 11. P. Domingos. Metacost: A general method for making classifiers cost-sensitive. In Proceedings of the Fifth International Conference on Knowledge Discovery and Data Mining, pages 155–164, New York, 1999. ACM Press. 272 12. C. Drummond and R. C. Holte. Exploiting the cost (in)sensitivity of decision tree splitting criteria. In Machine Learning: Proceedings of the Seventeenth International Conference, pages 239–246, San Francisco, CA, 2000. Morgan Kaufmann. 272 13. R. O. Duda, P. E. Hart, and D. G. Stork. Pattern Classification. John Wiley and Sons, Inc. - Interscience, second edition, 2000. 272 14. B. Efron and R. J. Tibshirani. An Introduction to the Bootstrap. Chapman and Hall, New York, 1993. 273, 281 15. C. Elkan. The foundations of cost-sensitive learning. In Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence. Morgan Kaufmann Publishers, Inc., 2001. 271 16. W. Fan, S. J. Stolfo, J. Zhang, and P. K. Chan. AdaCost: Misclassification costsensitive boosting. In Machine Learning: Proceedings of the Sixteenth International Conference, pages 97–105, San Francisco, 1999. Morgan Kaufmann. 272 17. I. J. Good. The Estimation of Probabilities: An Essay on Modern Bayesian Methods. M. I. T. Press, Cambridge, Mass., 1965. 273 18. N. Japkowicz. The class imbalance problem: Significance and strategies. In Proceedings of the 2000 International Conference on Artificial Intelligence (IC-AI’2000), 2000. 272 19. M. Kukar and I. Kononenko. Cost-sensitive learning with neural networks. In Proceedings of the Thirteenth European Conference on Artificial Intelligence, Chichester, NY, 1998. Wiley. 272 20. T. Leonard and J. S. J. Hsu. Bayesian Methods, An Analysis for Statisticians and Interdisciplinary Researchers. Cambridge University Press, 1999. 272 21. D. D. Margineantu. Building ensembles of classifiers for loss minimization. In M. Pourahmadi, editor, Models, Predictions and Computing: Proceedings of the 31st Symposium on the Interface, volume 31, pages 190–194. The Interface Foundation of North America, 1999. 272 22. D. D. Margineantu. Methods for cost-sensitive learning. Technical report, Department of Computer Science, Oregon State University, Corvallis, OR, 2001. 271 23. D. D. Margineantu and T. G. Dietterich. Bootstrap methods for the cost-sensitive evaluation of classifiers. In Machine Learning: Proceedings of the Seventeenth International Conference, pages 583–590, San Francisco, CA, 2000. Morgan Kaufmann. 276, 281 24. E. P. D. Pednault, B. K. Rosen, and C. Apte. The importance of estimation errors in cost-sensitive learning. In Cost-Sensitive Learning Workshop Notes, 2000. 279 25. F. Provost and P. Domingos. Well-trained PETs: Improving probability estimation trees. Technical Report IS-00-04, Stern School of Business, New York University, 2000. 273 26. F. Provost, T. Fawcett, and R. Kohavi. The case against accuracy estimation for comparing classifiers. In Proceedings of the Fifteenth International Conference on Machine Learning, pages 445–453. Morgan Kaufmann, San Francisco, 1998. 273


281

27. J. R. Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann, San Francisco, 1993. 273, 275 28. M. Saar-Tsechansky and F. Provost. Active learning for class probability estimation and ranking. In Proceedings of the Seventeenth International Joint Conference on Artificial Intelligence, pages 911–917. AAAI Press/MIT Press, 2001. 279 29. P. Smyth, A. Gray, and U. Fayyad. Retrofitting decision tree classifiers using kernel density estimation. In Machine Learning: Proceedings of the Twelvth International Conference, pages 506–514, 1995. 273 30. B. Zadrozny and C. Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Proceedings of the Eighteenth International Conference on Machine Learning, pages 609–616, San Francisco, CA, 2001. Morgan Kaufmann. 273

A

BDeltaCost

BDeltaCost is a paired test that computes a confidence interval for the expected difference in cost of two classifiers. The test is based on the idea of bootstrap [14], a computational method that is used for estimating the standard error of a parameter of an unknown distribution, based on a random sample S drawn from that distribution. The bootstrap works by drawing with replacement T samples for S, each consisting of a number of data values equal to the number of elements in S. The value parameter of interest is computed for each of these samples. The standard error is estimated by the sample standard deviations of the T replicates (also called bootstrap replicates). In a similar way, BDeltaCost tests the null hypothesis H0 that two classifiers have the same expected loss (on new test data) against the alternative hypothesis H1 that the two classifiers have different losses. The test draws repeated samples of the data and calculates the differences in loss for the two classifiers, sorts the resulting values in ascending order and rejects the null hypothesis if 0 is not contained by the interval defined by the middle c% values, for a c% confidence interval (e.g. for a 95% confidence interval and T = 1000 the test will check the interval between the 26th and 975th value). The way the test has been designed, Laplace corrections can be used to correct for zero values (that occured because of the small size of the test set) in the confusion matrices. Margineantu and Dietterich [23] have shown that the BDeltaCost test works better and gives tighter confidence intervals, than the standard tests based on the normal distribution.

On-Line Support Vector Machine Regression Mario Martin Software Department, Universitat Politècnica de Catalunya Jordi Girona 1-3, Campus Nord, C6., 08034 Barcelona, Catalonia, Spain [email protected] Abstract. This paper describes an on-line method for building ε-insensitive support vector machines for regression as described in [12]. The method is an extension of the method developed by [1] for building incremental support vector machines for classification. Machines obtained by using this approach are equivalent to the ones obtained by applying exact methods like quadratic programming, but they are obtained more quickly and allow the incremental addition of new points, removal of existing points and update of target values for existing data. This development opens the application of SVM regression to areas such as on-line prediction of temporal series or generalization of value functions in reinforcement learning.

1

Introduction

Support Vector Machines, from now on SVM, [12] have been one of the most developed topics in Machine Learning in the last decade. Some reasons that explain this success are their good theoretical properties in generalization and convergence –see[2] for a review. Another reason is their excellent performance in some hard problems –see for instance [9,4]. Although SVMs are being used mainly for classification tasks, they can also be used to approximate functions (what is called SVM regression). One problem that prevents a wider use of SVMs for function approximation is that, though their good theoretical approaches, they are not applicable on-line, that is, in cases where data is sequentially obtained and learning has to be done from the first data. One paradigmatic example is the on-line prediction of temporal series. When new data arrive, learning has to begin from scratch. SVMs for regression have not been either suitable for problems where the target values of existing observations change quickly, for instance, in reinforcement learning [11]. In reinforcement learning, function approximation is needed to learn value functions, that is, functions that return for each state the future expected reward if the agent follows the current policy from that state. SVMs are not used to approximate value functions because these functions are continuously update as the agent learns and changes its policy. One time, the estimated future reinforcement from state s is y, but later (usually very soon) a new estimation returns another value for the same state. Using SVM regression in this case implies again learning from scratch. T. Elomaa et al. (Eds.): ECML, LNAI 2430, pp. 282–294, 2002. c Springer-Verlag Berlin Heidelberg 2002

On-Line Support Vector Machine Regression

283

One alternative is to adapt to the regression case one of the recent algorithms for incremental SVMs [3] and on-line learning [5,7,6], but such techniques return approximate solutions (though in some cases the error is bounded) and, more important, they do not allow to remove data or update target values of data. The only exception we know of is [1] but it is only described for classification tasks. In order to allow the application of SVMs for regression to these areas, this paper describes the first (at our best knowledge) exact on-line learning algorithm for SVM function approximation. The algorithm is based in three actions that allow respectively (1) incrementally add new data to the SVM, (2) remove data from the SVM, and (3) update target values for existing data in the SVM. The algorithm we propose is an extension of the work proposed in [1] for incremental SVM learning for classification tasks, but now applied to function approximation. In brief, the key idea of the algorithm consists in finding the appropriate Karush-Kuhn-Tucker (KKT) conditions for new or updated data by modifying its influence (β) in the regression function while maintaining consistence in the KKT conditions for the rest of the data used for learning. This idea is fully explained throughout the paper.

2

Reformulation

Specifically, we propose in this paper a method for the on-line building of ε-insensitive support vector machines for regression. The goal of this kind of machines is to find a function that presents at most ε deviation from the target values [12] while being as “flat” as possible. This version of SVM regression is appealing because not all vectors become support vectors, which is not the case in other approaches [10]. SVMs for regression are usually solved by resorting to a standard dualization method using Lagrange multipliers. The dual formulation for ε-insensitive support vector regression is to find values for α, α∗ that minimize the following quadratic objective function: W =

1 (αi − α∗i )Qij (αj − α∗j ) − yi (αi − α∗i ) + ε (αi + α∗i ) 2 ij i i

(1)

subject to the following constraints: 0 ≤ αi , α∗i ≤ C (αi − α∗i ) = 0

(2) (3)

i

where Q is the positive-definite kernel matrix Qij = K(xi , xj ), and ε > 0 is the maximum deviation allowed. Including in (1) a Lagrange multiplier for constraint (3), we get the following formulation:

284

Mario Martin

1 (αi − α∗i )Qij (αj − α∗j ) − 2 ij yi (αi − α∗i ) + ε (αi + α∗i ) + b (αi − α∗i ) W =

i

i

(4)

i

with first order conditions for W : gi = gi∗ =

∂W = Qij (αj − α∗j ) − yi + ε + b ∂αi j

(5)

∂W = − Qij (αj − α∗j ) + yi + ε − b = −gi + 2ε ∂α∗i j

(6)

∂W = (αj − α∗j ) = 0 ∂b j

(7)

Renaming (αi − α∗i ) to βi for simplicity, we have: gi = gi∗ =

2.1

∂W = Qij βj − yi + ε + b ∂αi j

(8)

∂W = − Qij βj + yi + ε − b = −gi + 2ε ∂α∗i j

(9)

∂W = βj = 0 ∂b j

(10)

Separation of Data

The first order conditions for W lead to the Karush-Kuhn-Tucker (KKT) conditions, that will allow the reformulation of SVM for regression by dividing the whole training data set D into the following sets: margin support vectors S (where gi = 0 or gi∗ = 0), error support vectors E (where gi < 0), error star support vectors E ∗ (where gi∗ < 0), and the remaining vectors R. Specifically, centering on gi , KKT conditions are:  2ε < gi → gi∗ < 0 βi = −C i ∈ E ∗    ∗   gi = 2ε → gi = 0 −C < βi < 0 i ∈ S i∈R 0 < gi < 2ε → 0 < gi∗ < 2ε βi = 0  ∗  g = 0 → g = 2ε 0 < β < C i∈S  i i i   gi < 0 → gi∗ > 2ε βi = C i∈E Figure 1 shows the geometrical interpretation of these sets in the feature space. Note that j Qij βj + b − yi is the error of the target value for vector i. Thus gi and gi∗ can be thought as thresholds for error in both sides of the ε-tube.


285

Fig. 1. Decomposition of D following KKT conditions into margin support vectors S, error support vectors E, error support vectors star E ∗ and remaining vectors R. Cross marks represent vectors in the feature space. S vectors are exactly on the margin lines, R vectors are inside the ε-tube (grey zone), and E and E ∗ vectors are outside the ε-tube The division of the data set into subsets and the characterization of β values for each subset, allow us to rewrite equations (8), (9) and (10), for all vectors i ∈ D, as follows: gi =

Qij βj + C

j∈S

j∈E

Qij − C

Qij

j∈E ∗

−yi + ε + b

(11)

gi∗ = −gi + 2ε

(12)

βj + C|E| − C|E ∗ | = 0

(13)

j∈S

3

On-Line Support Vector Regression

In order to build exact on-line support vector machines for regression, we need to define three incremental actions: add one new vector: One new observation xc is added to the data set D with the target value yc . This operation should include the corresponding vector in the feature space with the “exact” βc value but without beginning from scratch. remove one vector: One existing observation xc in D with target value yc is removed from the data set. The resulting SVM should be the same that would be training from scratch a SVM with D − {c}.

286

Mario Martin

update one vector: One existing observation xc in D with target value yc changes the target value to yc . As in the previous cases the resulting machine should be the same that would be training from scratch a SVM with exact methods. In this section we will describe how these actions can be efficiently implemented. Addition and update actions will consist in finding consistent KKT conditions for the vector being added or updated. Removal will be based on diminishing the influence of the vector being removed on the regression tube until it vanishes. 3.1

Adding One New Vector

A new vector c is added by inspecting gc and gc∗ . If both values are positive, c is added as an R vector because that means that the new vector lies inside the εtube (see KKT conditions). When gc or gc∗ are negative, the new vector is added by setting its initial influence on the regression (βc ) to 0. Then this value is carefully modified (incremented when gc < 0 or decremented when gc∗ < 0) until its gc , gc∗ and βc values become consistent wrt KKT conditions (that is, gc < 0 and βc = C, or gc∗ < 0 and βc = −C, or 0 < βc < C and gc = 0, or −C < βc < 0 and gc∗ = 0). Modification of βc Variations in the βc value of the new vector c, influence gi , gi∗ and βi values of the other vectors in D, and thus, can force the transfer of some vectors from one set S, R, E or E ∗ to another set. This transfer means that gi , gi∗ and βi values for vector i become no longer consistent with the KKT conditions of the set where vector i is currently assigned, but become consistent with the KKT conditions of another set. The modification of βc must take into account these transfers between sets. This section describes how the modification of βc influences gi , gi∗ and βi values of the vectors in D while sets S, E, E ∗ and R remain constant. In the next section we describe how to deal with vector migrations between sets. From equations (11), (12), and (13) it is easy to calculate the variation in gi , gi∗ and βi when a new vector c with influence βc is added without migration of vectors between sets S, E, E ∗ and R: ∆gi = Qic ∆βc +

Qij ∆βj + ∆b

(14)

∆gi∗ = −∆gi

(15)

j∈S

∆βc +

∆βj = 0

(16)

j∈S

Note that while one vector remains in E, E ∗ or R sets, its β value does not change.


287

In particular, if margin support vectors must remain in S, then ∆gi ≡ 0 for i ∈ S. Thus, if we isolate ∆βc terms in equations (14) and (16) for vectors i ∈ S, we get: Qij ∆βj + ∆b = −Qic ∆βc (17) j∈S

∆βj = −∆βc

(18)

j∈S

That, assuming S = {S1 , S2 , · · · , Sl }, can be matricialy formulated as follows:     1 ∆b  Q S1 c   ∆βS1      (19) Q ·  .  = −  .  ∆βc  ..   ..  ∆βSl Q Sl c where Q is defined as: 

0 1  1 QS1 ,S1  Q=. ..  .. .

 ··· 1 · · · QS1 ,Sl   ..  .. . . 

(20)

1 QSl ,S1 · · · QSl ,Sl From (19), 

   ∆b 1  ∆βS1   Q S1 c       ..  = −Q−1 ·  ..  ∆βc  .   .  ∆βSl

(21)

Q Sl c

and thus, ∆b = δ∆βc

(22)

∆βj = δj ∆βc

∀j ∈ S

(23)

where 

δ





1



 Q S1 c   δS 1       ..  = −R  ..   .   .  δS l Q Sl c

(24)

and R = Q−1 . Equations (22) and (23) show how the variation in the βc value of a new vector c influences βi values of vectors i ∈ S. The δ values are named coefficient

288

Mario Martin

sensitivities from [1]1 . Note that β values for vectors not in S do not change while these vectors do not migrate to another set. Thus, we can extend equation (23) to all vectors in D by setting δi ≡ 0 for i ∈ S. Now, we can obtain for vectors i ∈ S how gi and gi∗ change as βc changes. From equation (14), we replace ∆βj and ∆b by their equivalence in equations (22) and (23). ∆gi = Qic ∆βc + Qic ∆βc +

Qij ∆βj + ∆b =

j∈S

Qij δj ∆βc + δ∆βc =

j∈S

Qic + Qij δj + δ ∆βc = j∈S

γi ∆βc

(25)

∀i ∈ S

(26)

where γi = Qic +

Qij δj + δ

j∈S

The γ values are named margin sensitivities and are defined only for non margin support vectors because for i ∈ S, ∆gi = 0. As we have done with coefficient sensitivities, if we extend equation (25) to all vectors in D, we must set γi ≡ 0 for i ∈ S. Equation (25) shows how gi changes as βc changes, but indirectly also shows how gi∗ changes, because equation (15) states that ∆gi∗ = −∆gi . Summarizing, equation (25) shows, for vectors not in S, how gi and gi∗ values change as βc changes (note that their β value does not change). Equation (22) shows how βi for vectors i ∈ S change as βc changes (note that ∆gi and ∆gi∗ is 0 for these vectors). Finally, equation (23) shows how b varies as βc changes. All these equations are valid while vectors do not migrate from set R, S, E or E ∗ to another one. But in some cases, in order to reach consistent KKT conditions for the new vector c, it could be necessary to change first the membership of some vectors to these sets. Well, do not worry. Modify βc in the right direction (increment or decrement) until one migration is forced. Migrate the vector updating S, E, E ∗ and R sets adequately, and then continue the variation of βc . Migration of Vectors between Sets This section describes all possible different kinds of migrations between sets S, E, E ∗ and R, and how they can be detected. One vector can migrate only from its current set to a neighbor set. Figure 1 shows the geometrical interpretation of each set and from it we can infer the following possible migrations. 1

Note that [1] use the β symbol for representing this concept. As β is widely used in SVM regression as (α − α∗ ), we have decided to change the notation.


289

from E to S: One error support vector becomes a margin support vector. This migration can be detected when updating gi for i ∈ E following equation (25), gi (that was negative) becomes 0. The maximum variation in βc that does not imply migrations from E to S can be calculated as follows: The maximum ∆gi allowed for one vector i ∈ E is (0 − gi ), that is, from gi < 0 to gi = 0. From equation (25) we have, ∆βc = ∆gi γi−1 . Thus, the maximum variation allowed without the migration of vector i from E to S can be equated as: (0 − gi )γi−1 . Calculating this value for all vectors in E and selecting the minimum value, we obtain the maximum variation allowed in βc that does not force migration of vectors from E to S. from S to E: One margin support vector becomes an error support vector. This migration is detected when, updating βi for i ∈ S following equation (23), βi (that was 0 < βi < C) becomes C. Similarly to the previous case, from equation (23), ∆βc = ∆βi δi−1 . Thus, the maximum variation allowed without the migration of vector i from S to E can be formulated as: (C − βi )δi−1 . Calculating this value for all vectors in S and selecting the minimum value, we obtain the maximum variation allowed in βc that does not force migration of vectors from S to E. from S to R: One margin support vector becomes a remainder vector. This happens when updating βi for i ∈ S following equation (23), βi (that was 0 < βi < C or −C < βi < 0) turns into 0. The maximum variation allowed without the migration of vector i from S to R can be formulated as in the previous case as follows: (0 − βi )δi−1 . Calculating this value for all vectors in S and selecting the minimum value, we obtain the maximum variation allowed in βc that does not force migration of vectors from S to R. from R to S: One remainder vector becomes a margin support vector. This case is detected when the update of gi or gi∗ for i ∈ R (thus with gi > 0 and gi∗ > 0) causes that one value becomes 0. The maximum variation in βc that does not imply migrations from R to S is calculated by collecting (0 − gi )γi−1 and (0 − gi∗ )γi−1 for all vectors in R and selecting the minimum value. This is the maximum variation allowed in βc that does not force migration of vectors from R to S. from S to E ∗ : One margin support vector becomes an error support vector. This case is detected when, in the update of βi for i ∈ S the value changes from −C < βi < 0 to −C. The maximum variation in βc that does not imply migrations from S to E ∗ is calculated by collecting (−C − βi )δi−1 for all vectors in S and selecting the minimum value. from E ∗ to S: One error support vector becomes a margin support vector. This last case is detected when updating gi∗ for vectors i ∈ E ∗ , the value for one vector becomes gi∗ = 0. The maximum variation in βc that does not imply migrations from E ∗ to S is calculated by collecting (0 − gi∗ )γi−1 for all vectors in E ∗ and selecting the minimum value.

290

Mario Martin

The only memory resources required in order to monitorize KKT conditions fulfilled by vectors in D are: gi and gi∗ for vectors i ∈ S, and βi for vectors i ∈ S. In addition, in order to efficiently update these variables we also need to maintain Qij for i, j ∈ S –needed in equation (26)–, and R –needed in equation (24). Note that each possible migration is from S or to S and thus, after any migration, S must be updated. This implies that, in addition to the update of gi and gi∗ for vectors i ∈ S, and the update of βi for i ∈ βi , also matrixes Qij for i, j ∈ S and R, must be updated. To update matrix Q is easy because it only consists in adding/removing the row and column with the kernel values of the margin support vector added/removed. But the efficient update of matrix R is not obvious. In the following section we describe how to efficiently maintain matrix R. Updating R Matrix R is defined in (24) as the inverse of Q, which at the same time, is defined in (20). Note that we only need R for the update of β values, not Q. When one vector becomes a margin support vector (for instance due to a migration from another set) matrix Q should be updated and, thus, R should be updated too. The naive idea of maintaining Q and calculate its inverse to obtain R is expensive in memory and time resources. Instead of this, we will work on R directly. The updating procedure is an adaptation of the method proposed by [1] for classification to the regression problem. On one hand, when we are adding one margin support vector c, matrix R is updated as follows:    δ 0  δS 1    . 1   ..   R ..  R :=   .  · δ δ S 1 · · · δS l 1 +   γ   c 0  δS l  0 ··· 0 0 1 

(27)

On the other hand, when margin support vector k is removed, matrix R is updated as follows: Rij := Rij − R−1 kk Rik Rkj

∀j, i = k ∈ [0..l]

(28)

where the index 0 refers to the b-term. Finally, to end the recursive definition of the R matrix updating, it remains to define the base case. When adding the first margin support vector, the matrix is initialized as follows: R := Q

−1

0 1 = 1 Qcc

−1

−Qcc 1 = 1 0

(29)


291

Procedure for Adding One New Vector Taking into account the considerations of the previous sections, the procedure for the incremental addition of one vector results as follows: 1. Set βc to 0 2. If gc > 0 and gc∗ > 0 Then add c to R and exit 3. If gc ≤ 0 Then Increment βc , updating β for i ∈ S and gi , gi∗ for i ∈ S, until one of the following conditions holds: - gc = 0: add c to S, update R and exit - βc = C: add c to E and exit - one vector migrates from/to sets E, E ∗ or R to/from S: update set memberships and update R matrix. Else {gc∗ ≤ 0} Decrement βc , updating β for i ∈ S and gi , gi∗ for i ∈ S, until one of the following conditions holds: - gc∗ = 0: add c to S, update R and exit - βc = −C: add c to E ∗ and exit - one vector migrates from/to sets E, E ∗ or R to/from S: update set memberships and update R matrix. 4. Return to 3 In this procedure, the influence on the regression of vector c to be added (βc ) is incremented until it reaches a consistent KKT condition. Increments in βc are done monitoring gi , gi∗ and βi of the whole set of vectors D. When one vector i does no longer fulfill the KKT conditions associated with the set where it was assigned, the vector is transferred to the appropriate set and variables are updated as necessary. This procedure always converges. The time cost to add one vector is linear in time with the number of vectors in D. The memory resources needed are quadratic in the number of vectors in S, because of matrix R. 3.2

Removing One Vector

The procedure for removing one vector from D uses the same principles that the procedure for adding one new vector. One vector c can be safely removed from D only when it does not have any influence on the regression tube. This only happens when the vector lies inside the ε-tube, or in other words, when βc = 0. If βc is not 0, the value must be incremented or decremented (depending on the sign of βc ) until it reaches 0. As in the case of adding one new vector, the

292

Mario Martin

modification of βc can change the membership to E, E ∗ , R and S of some other vectors in D. Thus, the modification of βc must be done carefully, keeping an eye on possible migrations of vectors between sets. The algorithm for the on-line removal of one vector is the following: 1. If gc > 0 and gc∗ > 0 Then remove c from R and exit 2. If gc ≤ 0 Then Decrement βc , updating β for i ∈ S and gi , gi∗ for i ∈ S, until one of the following conditions holds: - βc = 0: remove c from R and exit - one vector migrates from/to sets E, E ∗ or R to/from S: update set memberships and update R matrix. Else {gc∗ ≤ 0} Increment βc , updating β for i ∈ S and gi , gi∗ for i ∈ S, until one of the following conditions holds: - βc = 0: remove c from R and exit - one vector migrates from/to sets E, E ∗ or R to/from S: update set memberships and update R matrix. 3. Return to 2 As in the case of on-line addition of one vector, the procedure always converge. The time cost is linear in |D| while the memory cost is quadratic in |S|. 3.3

Updating Target Value for Existing Data

The obvious way to update the target value for one existing vector c in D consists in making good use of the previous actions. In order to update the pair < xc , yc > to < xc , y c > we can follow this procedure: 1. on-line removal of < xc , y c > 2. on-line addition of < xc , y c > Equations (8) and (9) show that the update of the target value yc changes gc and gc∗ . Thus, usually after an update, gc , gc∗ and βc values are no longer consistent with KKT conditions. An alternative and more efficient way of updating the target value consists in varying βc until it becomes KKT-consistent with gc and gc∗ like in the removal and addition cases. This procedure is described in [8].


4

293

Conclusions

In this paper, we have shown the first on-line procedure for building ε-insensitive SVMs for regression. An implementation of this method in Matlab is available at http://www.lsi.upc.es/~mmartin/svmr.html. The aim of this paper is to open the door to SVM function approximation for applications that receive training data in an incremental way, for instance on-line prediction of temporal series, and to applications where the target for the training data changes very often, for instance reinforcement learning. In addition to the on-line property, the proposed method presents some interesting features when compared with other exact methods like QP. First, the memory resources needed are quadratic in the number of margin support vectors, not quadratic on the total number of vectors. Second, empirical tests of the algorithm on several regression sets show comparable (or better) speeds in convergence, which means that the on-line learning procedure presented here is adequate even when the on-line property is not strictly required.

Acknowledgement I would thank to the people of the SVM seminar at the LSI department in the UPC for the enlightening discussions about SVM topics, specially to Cecilio Angulo for his comments and interest in this paper. I would also thank to Gert Cauwenberghs for making the incremental SVMc program available. This work has been partially supported by the Spanish CICyT project TIC 2000-1011.

References 1. G. Cauwenberghs and T. Poggio. Incremental and decremental support vector machine learning. In T. G. Dietterich T. K. Leen and V. Tresp, editors, Advances in Neural Infomation Processing Systems 13, pages 409–415. MIT Press, 2001. 282, 283, 288, 290 2. N. Cristianini and J. Shawe-Taylor. An Introduction to Support Vector Machines. Cambridge University Press, 2000. 282 3. C. Domeniconi and D. Gunopulos. Incremental support vector machine construction. In N. Cercone, T. Lin, and X. Wu, editors, Proceedings of the 2001 IEEE Intl. Conference on Data Mining, pages 589–592. IEEE Computer Society, 2001. 283 4. S. Dumais, J. Platt, D. Heckerman, and M. Sahami. Inductive learning algorithms and representations for text categorization. In 7th International Conference on Information and Knowledge Management, ACM-CIKM98, pages 148–155, 1998. 282 5. C. Gentile. A new approximate maximal margin classification algorithm. In T. G. Dietterich T. K. Leen and V. Tresp, editors, Advances in Neural Information Processing Systems 13, pages 500–506. MIT Press, 2001. 283 6. T. Graepel, R. Herbrich, and R. Williamson. From margin to sparsity. In T. G. Dietterich T. K. Leen and V. Tresp, editors, Advances in Neural Information Processing Systems 13, pages 210–216. MIT Press, 2001. 283

294

Mario Martin

7. J. Kivinen, A. Smola, and R. Williamson. Online learning with kernels. In S. Becker T. G. Dietterich and Z. Ghahramani, editors, Advances in Neural Information Processing Systems 14. MIT Press, 2002. 283 8. M. Martin. On-line support vector machine for function approximation. Technical report, Universitat Politècnica de Catalunya, Forthcomming. 292 9. E. Osuna, R. Freund, and F. Girosi. Training support vector machines: an application to face detection. In International Conference on Computer Vision and Pattern Recognition, CVPR97, pages 30–136, 1997. 282 10. A. Smola and B. Sch¨ olkopf. A tutorial on support vector regression. Technical Report NC2-TR-1998-030, NeuroCOLT2, 1998. 283 11. R. Sutton and A. Barto. Reinforcement Learning. MIT Press, 1998. 282 12. V. Vapnik. The nature of statistical learning theory. Springer Verlag, 1995. 282, 283

Q-Cut – Dynamic Discovery of Sub-goals in Reinforcement Learning Ishai Menache, Shie Mannor, and Nahum Shimkin Department of Electrical Engineering, Technion, Israel Institute of Technology Haifa 32000, Israel {imenache,shie}@tx.technion.ac.il [email protected]

Abstract. We present the Q-Cut algorithm, a graph theoretic approach for automatic detection of sub-goals in a dynamic environment, which is used for acceleration of the Q-Learning algorithm. The learning agent creates an on-line map of the process history, and uses an efficient MaxFlow/Min-Cut algorithm for identifying bottlenecks. The policies for reaching bottlenecks are separately learned and added to the model in a form of options (macro-actions). We then extend the basic Q-Cut algorithm to the Segmented Q-Cut algorithm, which uses previously identified bottlenecks for state space partitioning, necessary for finding additional bottlenecks in complex environments. Experiments show significant performance improvements, particulary in the initial learning phase.

1

Introduction

Reinforcement Learning (RL) is a promising approach for building autonomous agents that improve their performance with experience. A fundamental problem of its standard algorithms, is that although many tasks can asymptotically be learned by adopting the Markov Decision Process (MDP) framework and using Reinforcement Learning techniques, in practice they are not solvable in reasonable time. “Difficult” tasks are usually characterized by either a very large state space, or a lack of immediate reinforcement signals. There are two principal approaches for addressing these problems: The first approach is to apply generalization techniques, which involve low order approximations of the value function (e.g., [14], [16]). The second approach is through task decomposition, using hierarchical or related structures. The main idea of hierarchical Reinforcement Learning methods (e.g., [4], [6], [18]) is to decompose the learning task into simpler subtasks, which is a natural procedure also performed by humans. By doing so, the overall task is “better understood” and learning is accelerated. A major challenge as learning progresses is to be able to automatically define the required decomposition, as in many cases the decomposition is not straightforward and cannot be obtained a-priori. One common way of defining subtasks (statically or dynamically) is in the state-space context (e.g., [7], [11], [15]): The learning agent identifies landmark states, which are worthwhile reaching, and learns sub-policies for that purpose. T. Elomaa et al. (Eds.): ECML, LNAI 2430, pp. 295–306, 2002. c Springer-Verlag Berlin Heidelberg 2002

296

Ishai Menache et al.

This approach relies on the understanding that the path towards achieving a complex goal is through intermediate stages which are represented by states. If those states are discovered, and the policy to reach them is separately learned, the overall learning procedure may become simpler and faster. The purpose of this work is to dynamically find the target states which may usefully serve as subgoals. One approach is to choose states which have a nontypical reinforcement (a high reinforcement gradient, for example, as in [7]). This approach is not applicable in domains which suffer from delayed reinforcement (for example, a maze with ten rooms and one goal). Another approach is to choose states based on their frequency of appearance (see [7] and also [11]). The rule of thumb here is that states that have been visited often should be considered as the target of subtasks, as the agent will probably repeatedly visit them in the future, and may save time having local policies for reaching those states. The latter approach is refined in [11] by adding the success condition to the frequency measure: States will serve as subgoals if they are visited frequently enough on successful but not on unsuccessful paths. These states are defined as bottlenecks in the state space, a term that we will adopt. A problem with frequency based solutions is that the agent needs excessive exploration of the environment in order to distinguish between bottlenecks and “regular” states, so that options are defined (and learned) at relatively advanced stages of the learning process. The algorithm that will be presented here is based on considering bottlenecks as the “border states” of strongly connected areas. If an agent knows how to reach the bottleneck states, and uses this ability, its search in the state space will be more efficient. The common characteristic of the methods that were presented above is that the criterion of choosing a state as a bottleneck is local, i.e., based on certain qualities of the state itself. We shall look for a global criterion that chooses bottlenecks by viewing all state transitions. The Q-Cut algorithm, which will be shortly presented, is based on saving the MDP’s history in a graph structure (where nodes represent states and arcs represent state transitions) and performing a Max-Flow/Min-Cut algorithm on that graph in order to find bottleneck states, which will eventually serve as the target of sub-goals. In order to understand the use of the Max-Flow/Min-Cut algorithm (see [1]) in the context of Reinforcement Learning, let us first briefly review the graph theoretic problem it solves. Consider a capacitated directed network G = (N, A) (N is the set of nodes and A is the set of arcs) with a non negative capacity cij associated with each arc (i, j) ∈ A. The Max-Flow problem is to determine the maximum amount of flow that can be sent from a source node s ∈ N to a sink node t ∈ N , without exceeding the capacity of any arc. An s-t cut is a set of arcs, the deletion of which disconnects the network into two parts, Ns and Nt , where s ∈ Ns and t ∈ Nt . The problem of finding the s-t cut with the minimal capacity among all s-t cuts is called the s-t Min-Cut problem. It is well known that the s-t Min-Cut problem and the Max-Flow problem are equivalent ([1]). There are quite a few algorithms for solving the Max-Flow problem. The running time is in general a low polynomial in the number of nodes and arcs, making the algorithms an attractive choice for solving a variety of optimization problems (see [1] for

Q-Cut – Dynamic Discovery of Sub-goals in Reinforcement Learning

297

G

G

Fig. 1. Simple two room mazes. Goal is marked as “G”. After reaching the goal, the agent is positioned somewhere in the left room

further details on the Max-Flow algorithms and associated applications), and recently also for enabling efficient use of unlabeled data in classification tasks (see [3]). The specific method which we will use in our experiments is PreflowPush (described in [8]), which has a time complexity of O(n3 ), where n is the number of nodes. We shall use the Min-Cut procedure for identifying bottlenecks. This process reflects a natural and intuitive characterization of a bottleneck: If we view an MDP as a flow problem, where nodes are states and arcs are state transitions, bottlenecks represent “accumulation” nodes, where many paths coincide. Those nodes separate different parts of the state space, and therefore should be defined as intermediate goal states to support the transition between loosely connected areas. In addition, using a global criterion enables finding the best bottlenecks considerably faster. In order to explain this claim, consider the upper maze of Figure 1. Assume the agent always starts in the left room, and Goal is located in the right room. If the agent visited the wide passage between the rooms, the MinCut algorithm will identify it as a bottleneck, even if the number of visits is low in comparison to frequently visited states of the left room. In addition, consider the lower maze of Figure 1. If, for example, the agent reached the goal a significant number of times and used one of the passages in most trials, the cut will still choose both passages as bottlenecks. In both cases, the efficient discovery of bottlenecks is used for forming new options, accelerating the learning procedure. After introducing the basic algorithm we suggest to use the cut procedure for recursive decomposition of the state space. By dividing the state space to segments the overall learning task is simplified. Each of these segments is smaller than the complete state space and may be considered separately. The paper is organized as follows: In Section 2 we describe the Reinforcement Learning setup, extended to use options. Section 3 presents the Q-Cut algorithm.

298


In Section 4 we extend the basic algorithm to the Segmented Q-Cut algorithm. Some concluding remarks are drawn in Section 5.

2

Reinforcement Learning with Options

We consider a discrete time MDP with a finite number of states S and a finite number of actions A. At each time step t, the learning agent is in some state st ∈ S and interacts with the (unknown) environment by choosing an action at from the set of available actions at state st , A(st ), causing a state transition to st+1 ∈ S. The environment credits the agent for that transition through a scalar reward rt . The goal of the agent is to find a mapping from states to actions, called ∞ a policy, which maximizes the expected discounted reward over time, E{ t=0 γ t rt }, where γ < 1 is the discount factor. A commonly used algorithm in RL is Q-Learning ([5]). The basic idea behind Q-Learning is to update the Q-function at every time epoch. This function maps every state action pair to the expected reward for taking this action at that state, and following an optimal strategy for all other future states. It turns out that the learned Q-function directly approximates the optimal action-value function (asymptotical convergence is guaranteed under technical conditions, see [2]), without the need to explicitly learn a model of the environment. The formula for the update is: Q(st , at ) := Q(st , at ) + α(n(t, st , at )) rt + γ max Q(st+1 , a) − Q(st , at ) a∈A(st+1 )

where α(n(t, st , at )) is the learning rate function which depends on n(t, st , at ), the number of appearances of (st , at ) until time t. We now recall the extension of Q-Learning to Macro-Q-Learning (or learning with options, see [12] and [15]). Following an option means that the agent executes a sequence of (primitive) actions (governed by a “local” policy) until a termination condition is met. Formally, an option is defined by a triplet I, π, β, where: I is the options input set, i.e., all the states from which the option can be initiated; π is the option’s policy, mapping states belonging to I to actions; β is the termination condition over states (β(s) denotes the termination probability of the option when reaching state s). When the agent is following an option, it must follow it until it terminates. Otherwise it can choose either a primitive action or initiate an option, if available (we shall use the notation A (st ) for denoting all choices, i.e., the collection of primitives and options available at state st ). Macro-Q-Learning [12] supplies a value for every combination of state and choice. The update rule for an option ot , initiated at state st , becomes: Q(st , ot ) := Q(st , ot ) + α(n(t, st , ot ))(γ k

max

a ∈A (st+k )

Q(st+k , a )

−Q(st , ot ) + rt + γrt+1 + . . . γ k−1 rt+k−1 ) where k is the actual duration of ot . The update rule for a primitive action remains the same as in standard Q-Learning.


3

299

The Q-Cut Algorithm

The basic idea of the Q-Cut algorithm is to choose two states, s, t, which will serve as source and target nodes for the Max-Flow/Min-Cut algorithm, and perform the cut. If the cut is “good” (we shall define a criterion for its quality), the agent establishes new options for reaching the discovered bottlenecks. The whole procedure is outlined in Figure 3. We add the details for the steps of the algorithm below. Choosing s and t: The procedure for choosing s and t is task dependent. Generally, it is based on some distance metric between states (e.g., states that are separated in time or in some state space metric), or on the identification of states with special significance (such as the start state or the goal state). In some cases, choice of s and t is more apparent. Consider, for example, the mazes of Figure 1, under the following experiment: The agent tries to reach the goal in the right room, and when the goal is reached, the agent is transferred back to somewhere in the left room. A natural selection of s and t in this case, is to choose s as one of the states in the “returning area” and t as the goal. The reason for this choice is that the agent is interested in the bottlenecks along its path from start to goal. Activating cut conditions: The agent may decide to perform a cut procedure at a constant rate, which is significantly lower than the actual experience frequency (in order to allow a meaningful change of the map of process history between sequential cuts), and might depend on the available computational resources. Another alternative is to perform a cut when good source and target candidates are found according to the procedure for choosing s and t.

Repeat: – Interact with environment and learn using Macro-Q Learning – Save state transition history – If activating cut conditions are met, choose s, t ∈ S perform Cut Procedure(s,t)

Fig. 2. Outline of the Q-Cut Algorithm Cut Procedure(s,t) – Translate state transition history to a graph representation – Find a Minimum Cut partition [Ns , Nt ] between nodes s and t – If the cut’s quality is “good” Learn the option for reaching new derived bottlenecks from every state in Ns ,using Experience-Replay

Fig. 3. The Cut Procedure

300


Building the graph from history: Each visited state becomes a node in the graph. Each observed transition i → j (i, j ∈ S), is translated to an arc (i, j) in the graph. We still need to determine the capacity of the arc. Few alternatives are possible. First, capacity may be frequency based, which means setting the capacity of (i, j) to n(i → j), where n(i → j) stands for the number of transitions from i to j. Second, the capacity may be fixed, i.e., assigning a constant capacity (say of 1) to every transition, no matter how many times it occurred. The problem with the frequency-based definition is that we strengthen the capacity of frequently visited areas (e.g., early transitions near the source state, where the policy is actually random) over rarely visited areas (e.g., states that are visited just before performing the cut), thus making it more difficult to find the true bottlenecks. Fixed capacity is lacking in the sense that the same significance is attached to all transitions from some state i ∈ S, a deviation from the actual dynamics the agent faces. Our choice is a compromise between the two alternatives. The capacity is based on the relative frequency, i.e., the capacity of an arc (i, j) is set to the ratio n(i→j) n(i) , where n(i) is the number of visits at state i. Experiments show that capacity based on relative frequency achieves the best performance in terms of bottleneck identification. Determining the cut’s quality: The idea behind the design of the quality factor is that we are interested only in “significant” s-t cuts, meaning those with small number of arcs (forming a small number of bottlenecks) on the one hand, and enough states both in Ns and Nt (s ∈ Ns and t ∈ Nt ) on the other hand. Let |Ns | and |Nt | be the number of states in Ns and Nt , respectively. If |Ns | is too small, we need not bother defining an option from a small set. On the other hand, if |Nt | is small the meaning is that the area of states that we wish to enable easy access to will not contribute much to the overall exploration effort. In summary, we look for a small number of bottleneck states, separating significant balanced areas in the state space. Based on the above analysis, the quality factor of a cut is the ratiocut bipartitioning |Ns ||Nt | where A(Ns , Nt ) metric (See [9] and [17]). We define Q[Ns , Nt ] A(N s ,Nt ) is the number of arcs connecting both sets, and consider cuts whose quality factor is above a predetermined threshold. The threshold may be determined beforehand based on appropriate analysis of the problem domain. It is also possible to change it in the course of learning (e.g., lower it if no “significant” cuts were found). Learning an option: If the cut’s quality is “good”, then the minimal cut (i.e., a set of arcs) is translated into a set of bottleneck states by picking state j for each min-cut arc (i, j), with j ∈ Nt . After bottlenecks have been identified, the local policy for reaching each bottleneck is learned by an Experience Replay [10] procedure. Dynamic programming iterations are performed on all states belonging to Ns , using the recorded interaction with the environment. The bottleneck itself is given an artificial positive reward for the policy learning sake.


3.1

301


We illustrate the Q-Cut algorithm on the two simple grids of Figure 1. The experiment conditions are the same as in [11]: The agent starts each trial at a random location in the left room. It succeeds in moving in the chosen direction with probability 0.9. It receives a reward of 1 at the goal, and zero otherwise. The agent uses an -greedy policy, where = 0.1. The learning rate was also set to 0.1, and the discount factor γ to 0.9. A Cut Procedure was executed every 1000 steps, choosing t as the goal state and s as a random state in the left room. If the cut’s quality was good according to the above mentioned criterion, a new option was learned and added to the agent’s set of choices. The performance of the Q-Cut algorithm is depicted in Figure 4, which presents a 50-runs average of the number of steps to goal as a function of the episode. Comparing Q-Cut to standard Q-Learning (using the same learning parameters) emphasizes the strength of our algorithm: Options, due to bottleneck discovery are defined extremely fast, leading to noticeable performance improvement within 2 to 3 episodes. In comparison, the frequency based solution of [11] that was applied to the upper maze of Figure 1 yielded significant improvement within about 25 episodes. As a consequence, the goal is found a lot faster than by other algorithms, with near-optimal performance reached within 20 to 30 episodes. In order to clarify the inner working of Q-Cut, we added state frequency maps for both mazes, under Q-Cut and also Q-Learning. Figure 5 presents “snapshots” taken after 25 episodes. Bright areas represent states which were visited often during the course of learning, while darker areas stand for less frequently visited

One−passage maze

Two−passage maze

1600

1200 Q−Learning Q−Cut

Q−Learning Q−Cut

1400 1000 1200 800 Steps to Goal

Steps to Goal

1000

800

600

600 400 400 200 200

0

0

10

20

30

40 Episode

50

60

70

80

0

0

10

20

30

40

50

60

70

80

Episode

Fig. 4. Performance curves for Q-Cut compared to standard Q-Learning. The left graph presents simulation results for the upper maze of Fig. 1, the right graph presents simulation results for the lower maze of the same figure. The graphs depict the number of steps to goal vs. episode number (averaged over 50 runs)

302

Ishai Menache et al. Q-Learning

Q-Cut G

G

G

G

Fig. 5. State frequency maps for both mazes of Fig. 1 (upper maps describe the upper maze of Fig. 1). All measurements are averaged over 50 runs, and were taken each time after 25 episodes. Bright areas describe more frequently visited states. We can see that in both mazes, the Q-Learning agent suffers from exhaustive exploration of the left room. On the other hand, the Q-Cut agent learns the right path towards the bottlenecks, and therefore the bottlenecks themselves are the most visited states of the environment states. We conclude from the Q-Learning maps that the Q-Learning agent spent major efforts in exploring the left room. On the other hand, having discovered appropriate options, the Q-Cut agent wandered less in the left room, and used shorter paths for the passages of the maze (which have the brightest color in the Q-Cut frequency graphs). Being able to efficiently reach the right room, the global policy for reaching the goal is learned in less time, significantly improving performance.

4

The Segmented Q-Cut Algorithm

The Q-Cut algorithm works well when one bottleneck sequentially leads to the other (for illustration, imagine a wide hallway of sequential rooms, where adja-


303

cent rooms are separated by one or more doors). In general, if cuts are always performed on the entire set of visited states (which grows with time), chances of finding good bottlenecks decrease. Consider the more complex maze of Fig. 6. To solve the above mentioned problem, we may divide the state space into different segments, using bottlenecks that were already found. If, for example, the agent has found Bottlenecks 2 and 3, it may use them to divide the state space into two segments, where the first contains states from the two upper left rooms and the second contains all other states. In that way, cuts may be performed separately on each segment, improving the chances of locating other bottlenecks (Bottleneck 1, for example). The above idea is the basis for the Segmented QCut algorithm. The agent uses the discovered bottlenecks (each of which may consist of a collection of states) as a segmentation tool. We use here a “divide and conquer” approach: Work on smaller segments of states in order to find additional bottlenecks and define corresponding new options. The pseudo-code for the algorithm is presented in Figure 7. Instead of working with one set of states, Segmented Q-Cut performs cuts on the segments that were created, based on previously found bottlenecks. When a good quality cut is found (using the same criterion as in Section 3), the segment is partitioned into two new segments. New cuts will be performed in each of these segments separately. Before performing cuts, each segment is extended to include newly visited states, belonging to the segment. The extension is achieved by a graph connectivity test (a simple O(nm) search in the graph, where n is the number of states and m is the number of arcs representing state transitions), where arcs that belong to a certain valid cut are removed for the connectivity testing procedure. Performing a cut procedure on a segment N means activating the Min-Cut algorithm on several (s, t) pairs, where the sources s ∈ S(N ) are the bottleneck states leading to the segment. The targets are chosen as in the Q-Cut algorithm, based on some distance metric from their matching s.

S

2

1

3

S

5

4

6

G

Fig. 6. A 6-room maze. In each episode the agent starts at a random location in the upper left room. Bottleneck states are numbered for illustration reasons

304


Initialize: – Create an empty segment N0 – Include starting state s0 in segment N0 – Include starting state s0 in S(N0 ) Repeat: – Interact with environment/Learn using Macro-Q Learning – Save state transition history – For each segment N , if activating cut conditions are met: Cut Procedure(N)

Fig. 7. The Segmented Q-Cut algorithm Cut Procedure(N ) – Extend segment N by connectivity testing – Translate state transition history of segment N to a graph representation – For each s ∈ S(N ) Perform Min-Cut on the extended segment (s as source, choice of t is task depended) – If the cut’s quality is good (bottlenecks are found) • Separate the extended N into two segments Ns and Nt • Learn the Option for reaching the bottlenecks from every state in Ns , using Experience Replay • Save new bottlenecks in S(Nt )

Fig. 8. The Cut Procedure for a segment

4.1


The Segmented Q-Cut algorithm was tested on the six-room maze of Figure 6. The agent always started at a random location in the upper left room. Learning parameters were the same as in the experiments made on the simple maze examples. Results with comparison to Q-Learning are summarized in Figure 9. The Segmented Q-Cut has a clear advantage over Q-Learning. It is interesting to note when in the course of learning the agent found the real bottlenecks of the environment. On average, a first bottleneck was discovered at the middle of the first episode, the second at beginning of the second episode, and the third at the middle of the same episode. This indicates a fast discovery and definition of subgoals (even before goal location is known), which accelerates the learning procedure from early stages.

5

Conclusion

The Q-Cut algorithm (and its extension to the Segmented Q-Cut algorithm) is a novel approach for solving complex Markov Decision Processes, which are characterized by the lack of immediate reinforcement. Through very fast discovery


305

9000 Q−Learning Segmented Q−Cut

8000

7000

Steps to Goal

6000

5000

4000

3000

2000

1000

0

0

50

100

150

Episode

Fig. 9. Performance curves for Segmented Q-Cut compared to standard QLearning for the six-room maze simulations. The graphs depict the number of steps to goal vs. episode number. Results are averaged over 50 runs of bottlenecks, the agent immediately sets its own sub-goals on-line. By doing so, exploration of different areas in the state space, which are weakly connected, becomes easier, and as a by product learning is enhanced. The main strength of the algorithm is the use of global information: Viewing the Markov Decision Process as a map of nodes and arcs is a natural perspective for determining the strategic states, which may be worth reaching. The Min-Cut algorithm is used to efficiently find bottleneck states, which divide the observed state connectivity graph into two disjoint segments. Experiments on grid-world problems indicate the potential of the Q-Cut algorithm. The algorithm significantly outperforms standard Q-Learning in different maze problems. An underlying assumption of this work is that the off-line computational power is at hand, while actual experience might be expensive. Also note that the cut procedure is computationally efficient and is required only once in a while. The distinctive empirical results motivate the application of the Q-Cut algorithm to a variety of problems where bottlenecks may arise. A car parking problem, a robot learning to stand up (see [13]), and some scheduling problems, are characterized by the existence of bottlenecks that must be reached in order to complete the overall task. Performance of the algorithm in different learning problems, specifically those with a large state-space, is under current study. Additional algorithmic enhancements, such as alternative quality factors and region merging mechanism should also be considered.

Acknowledgements This research was supported by the fund for the promotion of research at the Technion. The authors would like to thank Yaakov Engel and Omer Ziv for helpful discussions.

306


References 1. R. K. Ahuja, T. L. Magnati, and J. B. Orlin. Network Flows Theory, Algorithms and Applications. Prentice Hall Press, 1993. 296 2. D. P. Bertsekas and J. N. Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, 1995. 298 3. A. Blum and S. Chawla. Learning from labeled and unlabeled data using graph mincuts. In Proceedings of the 18th International Conference on Machine Learning, pages 19–26. Morgan Kaufmann, 2001. 297 4. P. Dayan and G. E. Hinton. Feudal reinforcement learning. In Advances in Neural Information Processing Systems 5. Morgan Kaufmann, 1993. 295 5. P. Dayan and C. Watkins. Q-learning. Machine Learning, 8:279–292, 1992. 298 6. T. G. Dietterich. Hierarchical reinforcement learning with the MAXQ value function decomposition. Journal of Artificial Intelligence Research, 13:227–303, 2000. 295 7. B. Digney. Learning hierarchical control structure for multiple tasks and changing environments. In Proceedings of the Fifth Conference on the Simulation of Adaptive Behavior: SAB 98, 1998. 295, 296 8. A. V. Goldberg and R. E. Tarjan. A new approach to the maximum-flow problem. Journal of ACM, 35(4):921–940, October 1988. 297 9. D. J. Huang and A. B. Kahng. When clusters meet partitions: A new densitybased methods for circuit decomposition. In Proceedings of the European Design and Test Conference, pages 60–64, 1995. 300 10. L. G. Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning, 8(3):293–321, 1992. 300 11. A. McGovern and A. G. Barto. Automatic discovery of subgoals in reinforcement learning using diverse density. In Proceedings of the 18th International Conference on Machine Learning, pages 361–368. Morgan Kaufmann, 2001. 295, 296, 301 12. A. McGovern, R. S. Sutton, and A. H. Fagg. Roles of macro-actions in accelerating reinforcement learning. In Proceedings of the 1997 Grace Hopper Celebration of Women in Computing, pages 13–18, 1997. 298 13. J. Morimoto and K. Doya. Acquisition of stand-up behavior by a real robot using hierarchical reinforcement learning. In Proceedings of the 17th International Conference on Machine Learning, pages 623–630. Morgan Kaufmann, 2000. 305 14. S. P. Singh, T. Jaakkola, and M. I. Jordan. Reinforcement learning with soft state aggregation. In Advances in Neural Information Processing Systems, volume 7, pages 361–368. The MIT Press, 1995. 295 15. R. S. Sutton, D. Precup, and S. Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112:181–211, 1999. 295, 298 16. J. N. Tsitsiklis and B. Van Roy. An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control, 42(5):674–690, 1997. 295 17. Y. C. Wei and C. K. Cheng. Ratio cut partitioning for hierarchical designs. IEEE/ACM Transaction on Networking, 10(7):911–921, 1991. 300 18. M. Wiering and J. Schmidhuber. HQ-learning. Adaptive Behavior, 6(2):219–246, 1997. 295

A Multistrategy Approach to the Classification of Phases in Business Cycles Katharina Morik and Stefan R¨ uping Univ. Dortmund, Computer Science Department, LS VIII {morik,rueping}@ls8.informatik.uni-dortmund.de http://www-ai.cs.uni-dortmund.de Abstract. The classification of business cycles is a hard and important problem. Government as well as business decisions rely on the assessment of the current business cycle. In this paper, we investigate how economists can be better supported by a combination of machine learning techniques. We have successfully applied Inductive Logic Programming (ILP). For establishing time and value intervals different discretization procedures are discussed. The rule sets learned from different experiments were analyzed with respect to correlations in order to find a concept drift or shift.

1

Introduction

The ups and downs of business activities have been observed for a long time It is, however, hard to capture the phenomenon by a clear definition. The National Bureau of Economic Research (NBER) defines business cycles as “recurrent sequences of altering phases of expansion and contraction in the levels of a large number of economic and financial time series.” This definition points at the multi-variate nature of business cycles. It does not specify many of the modeling decisions to be made. There is still room for a variety of concepts. – What are the indices that form a phase of the cycle? Production, employment, sales, personal income, and transfer payments are valuable indicators for cyclic economic behavior. Are there others that should be included? – What is the appropriate number of phases in a cycle? The number of phases in a cycle varies in the various economic models from two to nine. The NBER model indicates two alternating phases. The transition from one phase to the next is given by the turning points trough and peak. In the model of the Rheinisch-Westf¨ alisches Institut f¨ ur Wirtschaftsforschung (RWI), a cycle consists of a lower turning point, an upswing, an upper turning point, and a downswing. Here, the turning points are phases that cover several months. – Do all cycles follow the same underlying rules or has there been a drift of the rules? There are two tasks investigated by economic theory, the prediction and the dating problem. Where the prediction of values of economic indicators is quite successful handled by macro-economic equations [6], the dating problem remains a challenge. In this paper, we tackle the dating problem: T. Elomaa et al. (Eds.): ECML, LNAI 2430, pp. 307–318, 2002. c Springer-Verlag Berlin Heidelberg 2002

308

Katharina Morik and Stefan R¨ uping

Dating: Given current (and past) business measurements, in which phase is the economy currently? In other words, the current measurements are to be classified as phases of a business cycle. The dating problem is solved in the United States of America by a board of experts, the NBER. The data on German business cycles are classified by experts, as well. The aim is now to learn from these classified business data rules that state in which phase of the cycle a country is in. This task is less clearly defined than the task of predicting business activities, because business cycles themselves are basically a theoretical model to explain the variation in business data. Linear discriminant analysis has been proposed as the baseline of empirical models1 . Univariate rules were learned that used threshold values for separating phases. The accuracy of the 18 learned rules was 54% in cross validation. It has been investigated how the classification can be enhanced by the use of monthly data [8]. More sophisticated statistical models have been developed and achieved 63% accuracy [15]. The use of Hidden Markov Models led to developing two signals for an increase in the probability of a turning point [3]. The results cannot be tranformed into classification accuracy2. Also extensive experiments with other learning techniques (linear and quadratic discriminant analysis, neural networks, support vector machines) in [13] did not deliver a better accuracy. In summary, the results of statistical economy show how hard it is to classify business phases correctly. In this paper, we investigate the applicability of inductive logic programming to the problem of dating phases of a business cycle. ILP was chosen because the results can easily be interpreted by the experts, experts are allowed to enter additional economic knowledge into the rule set, and ILP automatically selects the relevant features. We were given quarterly data for 13 indicators concerning the German business cycle from 1955 to 1994 (see Figure 1), where each quarter had been classified as being a member of one of four phases [7]. The indicators are: IE real investment in equipment (growth rate) C real private consumption (growth rate) Y real gross national product (growth rate) PC consumer price index (growth rate) PYD real gross national product deflator (growth rate) IC real investment in construction (growth rate) LC unit labour cost (growth rate) L wage and salary earners (growth rate) Mon1 money supply M1 RLD real long term interest rate 1 2

Claus Weihs at a workshop on business cycles at the “Rheinisch-Westf¨ alisches Institut f¨ ur Wirtschaftsforschung” in January 2002 The signals precede or follow a turning point by 5 to 7 quarters of a year [3].

A Multistrategy Approach

309

Y LC L

12

10

8

6

4

2

0

-2 up

utp

down

ltp

up

utp

down

ltp

-4 90

100

110

120

130

140

150

Fig. 1. Plot of the indicators Y, LC and L in two successive business cycles, starting with upswing in the quarter 82 (1976/1) and ending with lower turning point in the quarter 156 (1994/3) RS nominal short term interest rate GD government deficit X net exports We experimented with different discretizations of the indicator values (see Section 2.1). The discretization into ranges (levels) of values was also used in order to form time intervals. A sequence of measurements within the same range is summarized into a time interval. Relations between the different time intervals express precedence or domination of one indicator’s level to another ones level. We also compared the two phase with the four phase business cycle. In summary, the following three models were inspected: – business cycle with four phases, without time intervals, (Section 2.2) – business cycle with four phases, with time intervals, (Section 2.3). – business cycle with two phases, without time intervals (Section 2.4). Particular attention was directed towards the appropriate sample size for the dating problem. The homogeneity of the data set of business cycles with two phases was investigated (Section 2.5).

2

Experiments on German Business Cycle Data

Our leading question was whether ILP can support economists in developing models for dating phases of the business cycle. Given the quarterly data for

310


13 indicators concerning the German business cycles from 1955 to 1994 where each quarter is classified as member of one of four phases, we used all but one cycle for learning rules and tested the rules on the left-out cycle. The leave-onecycle-out test assesses the accuracy (how many of the predicted classifications of quarters corresponded to the given classification) and the coverage (how many of the quarters received a classification by the learned rules). For ILP learning, we applied RDT [11] with the following rule schemata: m1 (Index1, Value, Phase): Index1(T, V ), V alue(V ) → P hase(T ) m2 (Index1,Value,Index2, Phase): Index1(T, V ), V alue(V ), Index2(T, V ) → P hase(T ) m3 (Index1, Value1,Index2,Value2,Phase): Index1(T, V 1), V alue1(V 1), Index2(T, V 2), V alue2(V 2), opposite(V 1, V 2) → P hase(T ) The predicates that can instantiate the predicate variable Index are the 13 indicators of the economy (see above). The predicates that can instantiate the predicate variable V alue express the discretization of the real values of the indicators. The phase variable can be instantiated by down, ltp, up, utp for the phases of downswing, lower turning point, upswing and upper turning point of a four-phase business cycle model or by down, up for a business cycle model with two phases. 2.1

Discretization

The goal of discretization is to provide the learning algorithm with data in a representation from which it can generalize maximally. Actually, two discretization tasks have to be solved: Discretization of Values: split the continuous range of possible values into finitely many discrete For example, a gross national product of 9.21 in the third quarter could be expressed as the fact y(3, high)3 . Interval Segmentation: for a given time series, find a segmentation of the time points into maximal sub-intervals, such that the values of the series in this interval share a common pattern. For example, the time series of gross national products Y = (10.53, 10.10, 9.21, 5.17, 4.93) could be described as the temporal facts y(1, 3, high), y(4, 5, medium), but can also be described as y(1, 5, decreasing). Interval segmentation can be viewed as discretization of the temporal values, therefore in this section we will use the name discretization as a generic term for both discretization of values and interval segmentation. 3

Note, that the economic indicator Y is expressed as a predicate y and not as a variable in ILP.


311

The key in interval segmentation is to find a representation, that is adequate for the learner. There are many representations for time series [12], e.g. as piecewise constant or piecewise linear functions [10], using template patterns [4] or as temporal rules [5, 9]. In our case, the data is already extensively pre-processed using economical knowledge (e.g. the gross national product was originally developed as a single indicator for the state of national economy). Also, the data is given free of trends (as growth rates). It can be assumed that the relevant information lies in the value of the indicator alone. Hence, a representation of the given time series as piecewise constant functions seems to be appropriate. This has the additional advantage, that the interval segmentation can easily be found by discretizing the attribute values and joining consecutive time points with identical discretization. To find a high-quality value discretization, we can use the information that is given by the class of the examples in addition to the distribution of the numerical values [17]. Our goal is to find a discretization of the indicators, that already contains as much information about the cycle phase as possible. This directly leads to the use of information gain as the discretization criterion. In contrast to the usual approaches, we did not use an artificial criterion to determine the optimal number of discrete values, but used the number of interval segments that were induced by the discretization as our quality criterion. Using four discrete values usually led to a representation with a suitable number of facts. Note that this also deals with the information gains tendency to over-discretize the data, that was reported in [17]. A closer look at the resulting discretization showed that in certain cases, the indicators had a very high variation, which leads to many intervals that contained only one time point. In this case, the relevant observation may not be the value of the indicator, but the fact that this indicator was highly varying, i.e. that no definite value can be assigned to it. This can be expressed by a new fact indicator(T 1, T 2, unsteady), which replaces the facts indicator(T 1, T 1 + 1, value1), indicator(T 1 + 1, T 1 + 2, value2 ), . . . , indicator(T 2 − 1, T 2, valuen). 2.2

Modeling Four Phases without Time Intervals

The data correspond to six complete business cycles, each with four phases. We tested our model by a kind of leave-one-out test, where in each turn a full cycle was left out (LOO1 to LOO6). For the upper and lower turning point phases, no rule could be learned. Only for the upswing, each learning run delivered rules. For the downswing, only two learning runs, namely leaving out cycle 3 and leaving out cycle 5, delivered rules. Misclassifications at the turning points are strikingly more frequent than in other phases. Figure 2 shows the results. The results miss even the baseline of 54% in the average. Leaving out the fifth cycle (from 1974 until 1982) delivers the best result where both, accuracy and coverage, approach 70%. This might be due to its length (32 quarters), since also in the other experiment dealing with four phases the prediction of upper turning point and upswing is best, when leaving out the fifth cycle. Since the sixth cycle is even longer (45 quarters), we would expect best results in LOO6

312

Katharina Morik and Stefan R¨ uping Cycle LOO1 LOO2 LOO3 LOO4 LOO5 LOO6 Average

Accuracy 0.125 0.5 0.462 0.375 0.696 1.0 0.526

Coverage 0.25 1.0 0.462 1.0 0.696 0.36 0.628

No.of learned rules 13 upswing 12 upswing 10 upswing, 2 downswing 11 upswing 10 uspwsing, 1 downswing 1 upswing total: 60

Fig. 2. Results in the four phase model using time points

I1 I2

11111111 00000000 11111111 00000000 00000 11111 00000 11111 contains(I1,I2)

1111111 0000000 1111111 0000000 000000 111111 000000 111111 overlaps(I1,I2)

Fig. 3. The temporal relations contains and overlaps

which is true for the accuracy in this experiment. In the other experiment with four phases, the accuracy is best for upswing in LOO6 and second best for it in LOO5. 2.3

Modeling Four Phases with Time Intervals

Let us now see, whether time intervals can improve the results. We have used the discretization of the indicator values for the construction of time intervals (see Section 2.1). We end up with facts of the form Index(I,Range), and for each time point within the time interval I a fact stating that this time point T (i.e. quarter) lies in the time interval I: covers(I, T). We then described the relations between different time intervals by means of Allen’s temporal logic [2]. From the 13 possible relationships between time intervals, we chose contains and overlaps. The relation contains(I1, I2) denotes a larger interval I1 in which somewhere the interval I2 starts and ends. contains(I1, I2) is true for each time point within the larger interval I1. overlaps(I1, I2) is true for each time point of the interval I1 which starts before I2 is starting (see Figure 3). We left out the other possible relations, because they were either too general or too specific to be used in a classification rule or would violate the constraint, that only information about past events can be used in the classification 4 . The time intervals were calculated before the 4

A relation that would require that the end point of one interval was identical to the starting point of another interval would be too specific. A relation that would only require that an interval would happen before another interval, regardless of the amount of time in between, would be too general.


313

training started. The rule schemata were defined such that they link two indicators with their corresponding time intervals. One rule schema is more specialised in that it requires the time intervals of the two indicators to either overlap or include each other. This more specific rule schema was intended to find rules for the turning phases, where no rules were learned in the previous experiment. In fact, rules for the upper turning point, upswing, and downswing were learned, but no rules could be learned for the upper turning point. Another intention behind the time interval modeling was to increase the accuracy of the learned rules. Indeed, rules for the upper turning point could be learned with the average accuracy of 75% in the leave-one-cycle-out runs. However, the accuracy for upswing decreased to 34% in the average. Hence, overall the time interval model did not enhance the results of the time point model in as much as we expected (see Table 4).

Cycle LOO1

Phase upswing downswing utp ltp LOO2 upswing downswing utp ltp LOO3 upswing downswing utp ltp LOO4 upswing downswing utp ltp LOO5 upswing downswing utp ltp LOO6 upswing downswing utp ltp Average upswing downswing utp ltp

Accuracy Coverage 0.167 1 0 0 0 0 0 0 0 0.461 1 1 0.200 0 0 0 0.167 1 0.333 1 0 0 0.481 1 0 0 0 0.75 0.857 0.667 0.296 0.243 1 0 0 0.388 0.716 0.104 0.500 0 0 0.75 0.143

No. learned rules 73 1 0 2 103 3 2 0 87 2 2 2 59 7 0 4 88 3 0 4 6 2 0 0 69.3 3 0.667 2

Fig. 4. Results in the four phase model using time intervals

314

2.4


Modeling Two Phases

Theis and Weihs [14] have shown, that in clustering analyses of German macroeconomic data at most three clusters can be identified. The first two clusters correspond to the cycle phases of upswing and downswing and the eventual third cluster corresponds to a time period around 1971. This suggests, that two phases instead of four may be more suited for the description of business data. It also points at a concept drift (see Section 2.5). In our third experiment we mapped all time points classified as upper turning point to upswing and all quarters of a year classified as lower turning point to downswing. We then applied the rule schemata of the first experiment. An example of the learned rules is: ie(T, V 1), low(V 1), c(T, V 2), high(V 2) → down(T ) stating that a low investment into equipment together with high private consumption indicates a downswing. Again, leaving out the fifth or the sixth cycle gives the best results in the leave-one-cycle-out test. Accuracy and coverage are quite well balanced (see Figure 5). These learning results are promising. They support the hypothesis that a two phase model is more appropriate for the dating task. Concerning the selection of indicators, the learning results show that all indicators contribute to the dating of the phase. However, the short term interest rate does not occur in three of the rule sets. Consumption (both the real value and the index), net exports, money supply, government deficit, and long term interest rate are missing in at least one of the learned rule sets. For the last four cycles, i.e. leaving out cycle 1 or cycle 2, some indicators predict the upswing without further conditions: high or medium number of salary earners (l), high or medium investment in equipment (ie), high or medium investment in construction (ic), medium consumption (c), and the real gross national product (y). It is interesting to note, that a medium or high real gross national product alone classifies data into the upswing phase only when leaving out cycle 1,2, or 4. Since RDT performs a complete search, we can conclude, that in the data of cycle 1 to cycle 4, the gross national product alone does not determine the upswing phase. Further indicators are necessary there, for instance money supply (mon1) or consumer price index (pc).

Cycle LOO1 LOO2 LOO3 LOO4 LOO5 LOO6 Average

Accuracy 0,8125 0,588 0,823 0,8 0,869 1,0 0,815

Coverage 0,795 1,0 0,571 0,35 0,8 0,701 0,703

No. learned rules 9 upswing, 69 downswing 17 upswing, 35 downswing 2 upswing, 15 downswing 6 upswing, 8 downswing 10 upswing, 39 downswing 6 upswing, 41 downswing total 50 up, 207 down

Fig. 5. Results in the two phase model using time points


2.5

315

Concept Shift

Starting from the two-phase model, we analyzed the homogeneity of the business cycle data. The learning results from different leave-one-cycle-out experiments were inspected with respect to their correlation. If the same rule is learned in all experiments, this means that the underlying principle did not change over time. If, however, rules co-occur only in the first cycles or in the last cycle, we hypothesize a concept drift in business cycles. We used the correlation analysis of the APRIORI algorithm [1, 16]. We want to know whether some rules are learned in all training sets, or, at least, whether there are rules that are more frequently learned than others. Enumerating all learned rules we get a vector for each training set (corresponding to a transaction in APRIORI) where the learned rule is marked by 1 and the others are set to 0. The frequency of learned rules and their co-occurrence is identified. There is no rule which was learned in all training sets. Eight rules were learned from three training sets. No co-occurrence of learned rules could be found. There is one rule, which was learned in four training sets, namely leaving out cycle 1, cycle 4, cycle 5, or cycle 6: rld(T, V ), l(T, V ), low(V ) → down(T ) stating that the real long term interest rate and the number of wage and salary earners being low indicates a downswing. We now turn around the question and ask: which training sets share rules? For answering this question, a vector for each learned rule is formed where those training sets are marked by 1 which delivered the rule. – Eighteen rules were shared in the training sets leaving out cycle 5 and leaving out cycle 6. Four of the rules predict an upswing, fourteen rules predict a downswing. This means, that cycles 1 to 4 have the most rules in common. The data from the last quarter of 1958 until the third quarter of 1974 are more homogeneous than all the data from 1958 until 1994. – When leaving out cycle 1 or cycle 2, eleven rules occur in both learning results. This means, that cycles 3 to 6 have second most rules in common. The data from the second quarter of 1967 until the end of 1994 are more homogeneous than all data together. The rule set analysis shows that cycles 1 to 4 (1958 – 1974) and cycles 3 to 6 (1967 - 1994) are more homogeneous than the overall data set. We wonder what happened in cycles 3 and 4. The first oil crisis happened at the end of cycle 4 (November 1973 – March 1974). This explains the first finding well. It shows that our rule set analysis can indeed detect concept drift, where we know that a drift occured. However, the oil crisis cannot explain why cycles 3 to 6 share so many rules. The second oil crises occured within cycle 5 (1979 – 1980). We assume that the actual underlying rules of business cycles may have changed over time. The concept drift seems to start in cycle 3. The periods of cycles 1 and 2 (1958 – 1967) are characterized by the reconstrucion after the world war. Investment in construction (ic) and in equipment (ie) is not indicative in this period, since it is rather high, anyway. A low number of earners (l) together with

316


a medium range of the gross national product deflator (pyd) best characterizes the downswing in cycles 1 to 3 – this rule has been found when leaving out cycles 4 or 5 or 6. Since the unemployment rate was low after the war, it is particularly expressive for dating a phase in that period. This explains the second finding of our rule set analysis.

3

Conclusion and Further Work

ILP can be applied to the problem of classifying the phases of a business cycle with a performance that is comparable to state-of-the-art statistical methodes like linear discriminant analysis, quadratic discriminant analysis, support vector machines or neural nets [13]. There is evidence, that the high error rate, compared to other classification problems, is a result of the four phase model of business cycles. The two phase model seems to be fitting the data much better. Machine learning techniques in concert have answered the questions that have been our starting point (see Section 1). – ILP offers opportunities for the analysis of business cycle data. It is easy to interpret the results so that the learned rules can be easily inspected by economists. The multi-variate nature of ILP and the automatic selection of most relevant indicators fits the needs of dating problem. – The two-phase model of the business cycle clearly outperformed the fourphase model. Where the best average accuracy in the four-phase model was 53%, the average accuracy of the two-phase model was 82%. – Rule set analysis in terms of correlations between training set results shows that cycles 1–4 (1958 - 1974), leaving out cycle five or cycle six, had more rules in common than other cycles. The second most common rules were found when leaving out the first or the second cycle, i.e. with training on cycles 3–6 (1967 - 1994). Both findings can be explained in economical terms. The results could well be further enhanced. We used discretization in a straightforward manner by creating the interval segmentation based on the discretization of values. This can be extended by using some of the work of [10, 4, 5, 9]. However, in many of these approaches it is unclear, how the resulting discretization can be interpreted. For our application understandability is a main goal. The partitioning into two phases was very simple. A more sophisticated split within the upper and the lower turning phase, respectively, should lead to enhanced accuracy. Concept drift could be the reason for not reaching the level of accuracy that is often achieved in other domains. Training seperately cycles 4 to 6 and restricting the leave-one-cycle-out testing to these cycles could enhance the learning results. Finally, ILP allows a close cooperation with economists, who can easily inspect the learned rules, inspect contradictions of the model to the data and add further background knowledge to the model. This makes ILP a very suitable tool for working on the validation / falsification of economical theories.


317

Acknowledgments This work has been partially sponsored by the Deutsche For-schungsgemeinschaft (DFG) collaborative research center 475 “Reduction of Complexity for Multivariate Data Structures”. The authors thank Ullrich Heilemann, vice president of the Rheinisch-Westf¨ alische Institut f¨ ur Wirtschaftsforschung, for data of high quality and many valuable suggestions. We also thank Claus Weihs and Ursula Sondhauss for raising our interest in the task and providing insight in its statistical nature.

References [1] R. Agrawal and R. Srikant. Fast algorithms for mining association rules in large data bases. In Proceedings of the 20th International Conference on Very Large Data Bases (VLDB ‘94), pages 478–499, Santiago, Chile, sep 1994. 315 [2] J. F. Allen. Towards a general theory of action and time. Artificial Intelligence, 23:123–154, 1984. 312 [3] Marlene Amstad. Konjunkturelle Wendepunkte: Datierung und Prognose. St.Gallen, 2000. 308 [4] Donald J. Berndt and James Clifford. Finding patterns in time series: A dynamic programming approach. In Usama M. Fayyad, Gregory Piatetsky-Shapiro, Padhraic Smyth, and Ramasamy Uthurusamy, editors, Advances in Knowledge Discovery and Data Mining, chapter 3, pages 229–248. AAAI Press/The MIT Press, Menlo Park, California, 1996. 311, 316 [5] Gautam Das, King-Ip Lin, Heikki Mannila, Gopal Renganathan, and Padhraic Smyth. Rule Discovery from Time Series. In Rakesh Agrawal, Paul E. Stolorz, and Gregory Piatetsky-Shapiro, editors, Proceedings of the Fourth International Conference on Knowledge Discovery and Data Mining (KDD-98), pages 16 – 22, Ney York City, 1998. AAAI Press. 311, 316 [6] Rheinisch-Westf¨ alisches Institut f¨ ur Wirtschaftsforschung. Arbeitsbericht 2000. Rheinisch-Westf¨ alisches Institut f¨ ur Wirtschaftsforschung, Essen, Germany, 2000. 307 [7] U. Heilemann and H. J. M¨ unch. West German Business Cycles 1963-1994: A Multivariate Discriminant Analysis. CIRET-Conference in Singapore, CIRETStudien 50, 1996. 308 [8] U. Heilemann and H. J. M¨ unch. Classification of German Business Cycles Using Monthly Data. SFB-475 Technical Reports 8/2001. Universitaet Dortmund, 2001. 308 [9] Frank H¨ oppner. Learning temporal rules from state sequences. In Miroslav Kubat and Katharina Morik, editors, Workshop notes of the IJCAI-01 Workshop on Learning from Temporal and Spatial Data, pages 25–31, Menlo Park, CA, USA, 2001. IJCAI, AAAI Press. Held in conjunction with the International Joint Conference on Artificial Intelligence (IJCAI). 311, 316 [10] Eamonn Keogh, Selina Chu, David Hart, and Michael Pazzani. An online algorithm for segmenting time series. In Nick Cercone, T. Y. Lin, and Xindong Wu, editors, Proceedings of the 2001 IEEE International Conference on Data Mining, pages 289–296, San Jose, California, 2001. IEEE Computer Society. 311, 316

318


[11] J¨ org-Uwe Kietz and Stefan Wrobel. Controlling the complexity of learning in logic through syntactic and task-oriented models. Arbeitspapiere der GMD 503, GMD, mar 1991. 310 [12] Katharina Morik. The representation race - preprocessing for handling time phenomena. In Ramon L´ opez de M´ antaras and Enric Plaza, editors, Proceedings of the European Conference on Machine Learning 2000 (ECML 2000), volume 1810 of Lecture Notes in Artificial Intelligence, Berlin, Heidelberg, New York, 2000. Springer Verlag Berlin. 311 [13] Ursula Sondhauss and Claus Weihs. Incorporating background knowledge for better prediction of cycle phases. Technical Report 24, Universit¨ at Dortmund, 2001. 308, 316 [14] Winfried Theis and Claus Weihs. Clustering techniques for the detection of business cycles. SFB475 Technical Report 40, Universit¨ at Dortmund, 1999. 314 [15] Claus Weihs Ursula Sondhauß. Using labeled and unlabeled data to learn drifting concepts. In Miroslav Kubat and Katharina Morik, editors, Workshop notes of the IJCAI-01 Workshop on Learning from Temporal and Spatial Data, pages 38–44, Menlo Park, CA, USA, 2001. IJCAI, AAAI Press. Held in conjunction with the International Joint Conference on Artificial Intelligence (IJCAI). 308 [16] Ian Witten and Eibe Frank. Data Mining // Practical Machine Learning Tools and Techniques with JAVA Implementations. Morgan Kaufmann, 2000. 315 [17] D. A. Zighed, S. Rabaseda, R. Rakotomalala, and Feschet F. Discretization methods in supervised learning. In Encyclopedia of Computer Science and Technology, volume 40, pages 35–50. Marcel Dekker Inc., 1999. 311

A Robust Boosting Algorithm Richard Nock and Patrice Lefaucheur Université des Antilles-Guyane Grimaag-Dépt Scientifique Interfacultaire, Campus de Schoelcher 97233 Schoelcher, Martinique, France {Richard.Nock,Patrice.Lefaucheur}@martinique.univ-ag.fr

Abstract. We describe a new Boosting algorithm which combines the base hypotheses with symmetric functions. Among its properties of practical relevance, the algorithm has significant resistance against noise, and is efficient even in an agnostic learning setting. This last property is ruled out for voting-based Boosting algorithms like AdaBoost. Experiments carried out on thirty domains, most of which readily available, tend to display the reliability of the classifiers built.

1

Introduction and Motivations

Recent advances in Machine Learning (ML) have shown experimentally and theoretically the power of ensemble methods, that is, algorithms combining the predictions of multiple classifiers to make a single classifier [BK99]. Some of the most popular and widely used techniques are Arcing [Bre96b], Bagging [Bre96a], and Boosting [Sch90], in increasing order of the quantity of dedicated works. Arcing and Bagging are voting methods; they differ essentially by a scheme which iteratively modifies the training sample to build the voters. In Bagging [Bre96a], a new sample is generated at each step by bootstrap sampling the initial learning sample. Arcing [Bre96b], on the other side, keeps the initial examples but modifies their weights according to a rule which reweights higher the examples that have been difficult to classify by the voters built so far. Finally, Boosting is related to a general methodology in which an algorithm, called “strong” learner, requests and combines the output of so-called “weak” learners. The weak and strong adjectives are used advisedly, since the weak hypotheses are only required to perform a little bit better than the unbiased coin (but for any distribution over the learning sample). The strong learner combines them and outputs an hypothesis of arbitrary accuracy provided a sufficiently large number of weak hypotheses have been combined. Boosting draws its roots in the weak and strong learning frameworks [KV89, KV94], and further on the PAC model of Valiant [Val84]. One of the very first argumentation in favor of Boosting is due to [Kea88]. Historically, this paper is most interesting because it proposes, without proofs though, three potential Boosting algorithms, that are all voting procedures. The first evidence that Boosting is theoretically viable does not exactly use this combination scheme, but a recursive, decisiontree type majority vote combination [Sch90]. Beyond theory, the first evidences T. Elomaa et al. (Eds.): ECML, LNAI 2430, pp. 319–331, 2002. c Springer-Verlag Berlin Heidelberg 2002

320

Richard Nock and Patrice Lefaucheur

that the practical importance of Boosting is much more than “possible” (quote from [Kea88]) and can actually be of great help to solve challenging problems, culminates in the paper of [FS97] and its algorithm, AdaBoost, and more recently in refined analyzes of AdaBoost [FHT00, SS98]. Interestingly, this approach follows the voting approach proned by [Kea88], but with a powerful reweighting scheme, that Arcing further studied [Bre96b]. This scheme is a stepwise multiplicative update of the training example’s weights, so as to bring higher importance to those that have been hard to classify for the last hypothesis. Most approaches derived from Boosting are voting procedures (see e.g. the papers [Bre96b, FS97, SS98]), and more generally many ensemble methods are also voting procedures [Bre96a]. A set of voters is grown, which is a way to cast the initial examples onto a new representation space of different dimension, space into which each hypothesis built defines a new variable. Afterwards, a linear separator on this new set of variables is used to classify observations. Linear separators have certain desirable properties. They have good VC-dimension (n + 1 for a ndimensional space), that is, they bring discriminative power at affordable sample complexity for learning [Vap98]. Furthermore, provided the learning sample is separable, i.e. provided there exists a theoretical hyperplane separating the examples, they are efficiently learnable [Val84, NG95], which means the existence of polynomial-time, accurate induction algorithms. However, a major drawback is that whenever the dimension is fixed, if no assumption can be made about the separability of the data, then achieving the error of the optimal hyperplane is hard; Even constant factor approximations of this error are intractable [HS92]. This is a drawback that Support Vector Machines avoid by projecting the data into a very high dimensional space in which they are separable [Vap98]. Boosting, however, cannot guarantee such a separability property. Thus, it may face (in)approximability results depending on the nature of the target concept. Because seldom are the real domains for which assumptions can be made on this target concept, one should consider that learning occurs in a relaxed, sort of “agnostic” learning setting. Fortunately, a model of agnostic (or robust) learning, making no assumption on the target concept as well as on the unknown (but fixed) distribution used to generate the examples, has been receiving much attention and an extensive theoretical cover [KSS94]. Obviously, such an ideally relaxed learning setting is also inherently hard, and many results obtained are actually negative, precluding in one sense the use of most interesting classes of concept representations (even simple rules are not agnostically learnable [HS92]). Fortunately, most, but not all. This paper exhibits the Boosting abilities a class of concept representations among the computationally easiest to manipulate, which allows agnostic learning as well as a handles record noise levels (another crucial issue in Boosting [BK99]): symmetric functions. Symmetric function are Boolean functions whose outputs are invariant under permutation of the input bits [KL88]. Their discriminative power is the same as linear separators: they have the same VC-dimension. Computationally speaking,

A Robust Boosting Algorithm

321

most the learning algorithms require record times or space when compared to many other classes of concept representations [Bou92]. Efficient learning algorithms are known to learn in the PAC or agnostic model, even under malicious noise [Bou92, KL88]. These algorithms have two common-points: first, they are purely theoretical, and studied with absolutely no experiment in the aforementioned papers or books. Second, they follow a similar, simple induction scheme which, informally, proceeds by giving to each of the n + 1 possible summations the most frequent class in the corresponding subset of the learning sample. Boosting consists in our case in an algorithm creating a symmetric function of potentially high dimension, by stepwise additions of so-called weak hypotheses whose input is the set of initial variables, and whose output is the set {0, 1}. The set of weak hypotheses defines the new binary observation of each example into this new representation space. To be more precise, our contribution is twofold. First, we provide the Boosting algorithm and a theoretical study on its capabilities. Among all, we show an interesting relationship with previous works on symmetric functions: Boosting also suggests to build each symmetric function with the robust schemata of [Bou92, KL88]. Armed with an original margin definition adapted to symmetric functions, we prove various results on the algorithm. One is used to establish the theoretical Boosting ability of our algorithm, and makes it, among the quantity of works and algorithms around “Boosting”, one of the few proven to have the Boosting property in its original acceptation [Sch90, FS97]. Our algorithm is also the first to be, for any possible set of weak hypotheses, a theoretically efficient robust learner. Such a property is definitely ruled out by [HS92] for the approaches of [SFBL98, SS98] and related. Second, we provide numerous experiments on our algorithm, and compare it with other approaches on thirty domains, most of which are readily available. Dedicated experiments are also devoted to studying noise handling, and a criterion to stop Boosting. The following section presents our Boosting algorithm, SFboost. The two next sections study and discuss SFboost respectively from a theoretical and an experimental point of view.

2

From Boosting to SFboost

Due to the space constraints, we shall assume basic knowledge of Boosting algorithms, and in particular of their main representative: AdaBoost. All this basic knowledge is included in the clever paper of [SS98], for example. Let us give some general definitions. We let LS = {(x1 , y1 ), (x2 , y2 ), ..., (xm , ym )} denote a set of |LS| = m training examples, where |.| denotes the cardinal. Here, each instance xi belongs to a domain X , and is described using n variables. Each yi is a class, belonging to a set {−1, +1}, where −1 (resp. +1) is called the negative (resp. positive) class. Sometimes, the classes shall also be noted “−” and “+” respectively. This paper is mainly focused on binary classification problems, but multiclass classification problems can be handled with the AdaBoost technique, by making from a c-class classification problem c binary

322


problems, discriminating between one class and all others. They can also be handled with symmetric functions, as these classifiers are naturally multi-class, which is not the case for linear separators. Note that we do not require that examples be initially described using binary variables. Only the weak hypotheses shall be required to have their output in {0, 1}. Such binary output hypotheses can be decision trees, decision lists, monomials (simple rules), etc.

Algorithm 1: SFboost (LS) Input: a learning sample LS = {(x1 , y1 ), (x2 , y2 ), ..., (xm , ym )} for i = 1 to m do D0 (i) = 1/m; v 0 [0] = Z0 =

1 2

ln

m

+ D 0,0 − D 0,0

i=1

;

D0 (i) exp(−yi H0 (xi )); D (i) exp(−yi H0 (xi )) ; Z0

for i = 1 to m do D1 (i) = 0 for t = 1 to T do ht = Weak learn(LS, Dt ); for j = 0 to t do

   v t−1 [0] +     v t [j] =

Zt =

     

+ D (t−1,0)0 − D (t−1,0)0 + D (t−1,t−1)1 v t−1 [t − 1] + 12 ln − D (t−1,t−1)1 + + D exp(v t−1 [j−1])+D exp(v t−1 [j]) (t−1,j−1)1 (t−1,j)0 1 − 2 ln D− exp(−v t−1 [j−1])+D exp(−v t−1 [j]) (t−1,j−1)1 (t−1,j)0 1 2

ln

iff j = 0, iff j = t, (1) otherwise

m

Dt (i) exp (−yi (Ht (xi ) − Ht−1 (xi ))); D (i) exp(−yi (Ht (xi )−Ht−1 (xi ))) for i = 1 to m do Dt+1 (i) = t ; Z i=1

Output: HT (x) = v T

T

t=1

ht (xi )

t

We consider that a symmetric function is a function H : {0, 1}n → IR which is invariant under permutation of its input. Note that in a two-class framework, symmetric functions have their output generally restricted to {0, 1} [KL88]. We prefer to adopt our slightly more general definition which casts its output in IR, so as to make the output give both a label (its sign) and a confidence (its absolute value), thereby rejoining the convention of [SS98]. Suppose we have T weak hypotheses, h1 , h2 , ..., hT . Building a symmetric function HT using this intermediate set of hypotheses is actually building a symmetric function over the transformed set of examples {(x1 , y1 ), (x2 , y2 ), ..., (xm , ym )}, with xi = ∧Tt=1 ht (xi ). HT makes a partition of X into what we call buckets, the j th T bucket (j = 0, 1, ..., T ) receiving the examples (xi , yi ) for which t=1 ht (xi ) = j. represented by a T + 1-dimension “bucket vector” v T , The output of HT can be T such that HT (xi ) = v T t=1 ht (xi ) .


323

Suppose HT is built stepwise for t = 0, 1, ..., T , by adding weak hypotheses one at a time, so that at the very beginning, when no such hypothesis exists (t = 0), all examples share the same (empty) description and the symmetric function consists of a single bucket. At t = 1, we dispose of h1 , and two buckets, receiving examples (xi , yi ) for which h1 (x) = 0, and h1 (x) = 1 respectively, and so on for the next steps (t = 2, ..., T ). Each weak hypothesis is built by a so-called weak learner, Weak learn, which takes as input LS, and a distribution Dt . Note that the distribution takes t as index, which means that the weak learner shall be trained on Dt to output hypothesis ht . We consider that D0 is the uniform distribution, where D0 (i), the weight of example (xi , yi ) ∈ LS, is 1/m for all b i = 1, 2, ..., m. We also adopt the notation Dt,j (b ∈ {+, −}) to denote the sum of the weights of the examples at time t (i.e. computed with Dt ) falling in bucket j, b and belonging to class b (for 0 ≤ j ≤ t ≤ T ). Finally, D(t,j)b (b ∈ {0, 1}) is the sum of weights of examples at time t falling in bucket j, belonging to class b, and b b b = D(t,j)0 + D(t,j)1 ). Algorithm 1 for which ht+1 (.) = b (Therefore, we have Dt,j presents our approach to Boosting with symmetric functions, called SFboost. Note that it does not unveil how Weak learn works: though any learning algorithm can be used in place of Weak learn as long as the output of its hypotheses is {0, 1}, the next section presents in particular a criterion that Weak learn should optimize in its synergy with SFboost.

3

Analysis of SFboost

SFboost Repeatedly Levels Out the Distributions. SFboost proceeds by repeatedly leveling out the weights of the classes. Indeed, it is easy to show that after the computation of distribution Dt+1 , i.e. the distribution onto which ht+1 + − shall be built, we have ∀j = 0, 1, ..., t, Dt+1,j = Dt+1,j . SFboost Is a Boosting Algorithm. For t = 1, 2, ..., T , we define a new distribution Dt such that ∀(xi , yi ) ∈ LS, we have D0 = D0 , D1 = D1 , and (i) = Dt (i) exp(−yi Ht (xi ))/Zt (with Zt its normalization coefficient). We Dt>1 call D the “AdaBoost distribution”, as we would have in AdaBoost Dt+1 = Dt . With Dt , the algorithm SFboost can be much simplified. For example, it is a simple matter of fact to show that v t admits the much simpler expression + − b b v t [j] = 12 ln(Dt,j /Dt,j ). ∀b ∈ {+, −}, j = 0, 1, ..., t, fix D|t,j = Dt,j /Dt,j , Dt,j = t + − b b Dt,j + Dt,j , and Dt = j=0 Dt,j . Then Zt (for t = 0, 1, ..., T ) can be computed and upperbounded as follows: Zt =

t t

+ − + + 2 Dt,j Dt,j = 2 Dt,j D|t,j (1 − D|t,j ) j=0

j=0

 

t t

D + 1 − D +  = 2 D + (1 − D + ) . Dt,j D ≤ 2 t t t,j |t,j |t,j j=0

j=0

(2)

324


We now show a theorem which displays the ability of HT to separate the classes in the training sample. In the case of SFboost, [SFBL98] show that not only does the error decreases exponentially as t increases, but also the fraction of “risky” examples, close to the frontier. More precisely, [SFBL98] define in the case of a linear separator the notion of margin for an example (xi , yi ): following µ(xi ) = yi Tt=1 αt ht (x) / Tt=1 αt . If this margin is positive, the classifier assigns the right label to the example, and the larger its magnitude, the more confident is the classification given. [SFBL98] have shown that the accuracy of a linear separator depends on the margins over the training sample LS, and one should already strive to maximize them to optimize the quality of the classifier over the whole domain. If the training error rates t of each weak hypotheses does not exceed 1/2 − γ (for any possible Dt ), then we have [SFBL98]: PrLS [µ(xi ) ≤ θ] ≤

T 1−θ 1+θ (1 − 2γ) (1 + 2γ) .

(3)

The subscript LS in Pr denotes the probability w. r. t. random uniform choice T in LS. Fix S b = t=0 Dtb (b ∈ {0, 1}). Fix VT = (1/2)| ln(S + /S − )|. VT quantifies a deviation between the average distributions generated throughout the growth of HT . This is a separation parameter to which LS contributes: indeed, when there is no weak hypothesis in HT , a symmetric function can already be constructed, and its accuracy only depends on the balance between the classes in LS. For any example (xi , yi ), its margin µSF (xi ) equals T yi v T t=1 ht (xi ) yi HT (xi ) µSF (xi ) = = . (4) |VT | |VT | Like µ(.), if µSF (xi ) is positive, then the classifier gives the right class to the example. Furthermore, its magnitude quantifies a relative confidence of bucket h (x ) in the last AdaBoost distribution. The larger it is, the more useful t i t is the bucket partitioning generated by HT (w. r. t. xi ). Armed with our margin definition, we are able to prove the following theorem: Theorem 1. Fix bt = arg maxb∈{+,−} Dtb . We have: PrLS [µSF (xi ) ≤ θ] ≤ 2T +1

T θ θ (Dt bt )1+ T +1 (1 − Dt bt )1− T +1 .

(5)

t=0

Proof sketch: If µSF (xi ) ≤ θ, then 1 T bt θ θ max{S + , S − } t=0 Dt T +1 ≤ ln yi HT (xi ) ≤ ln T bt 2 min{S + , S − } 2 1− 1 t=0 Dt T +1 T b D t θ ln T t=0 t bt . ≤ 2(T + 1) t=0 (1 − Dt )

(6)


325

(the last ineq. follows from Jensen’s inequality). Fix K as the right-hand side parameter in ineq. 6. Then we have exp(−yi HT (xi )+K) ≥ 1, and PrLS [µSF (xi ) ≤ θ] ≤ ELS [exp(−yi HT (xi ) + K)] (E denotes the expectation). Remarking that T HT (xi ) = H0 (xi ) + t=1 (Ht (xi ) − Ht−1 (xi )), we easily obtain by unraveling DT +1 : PrLS [µSF (xi ) ≤ θ] ≤ exp(K)

T t=0

Zt

m

DT +1 (i) .

(7)

i=1

Plugging in the upperbound of each Zt in ineq. 2 and the expression of K, we obtain the statement of the theorem. Theorem 1 has two incidences. First, ineq. 7 shows that at each round, one should strive to select the weak hypothesis ht which minimizes the corresponding Zt . Second, let us consider that the training error rate t of each ht on its associated AdaBoost distribution is no more than 1/2 − γ for some constant γ > 0, i.e. ht performs only slightly better than random. Then, theorem 1 says that the fraction of examples having margin upperbounded by θ decreases as: T +1 θ θ 1+ T +1 1− T +1 PrLS [µSF (xi ) ≤ θ] ≤ (1 + 2γ) (1 − 2γ) .

(8)

If θ < γ(T + 1), then the right-hand side of ineq. 8 decreases exponentially with T . This result is stronger than the one we actually need to bring the Boosting ability of SFboost. Indeed, the Occam’s razor argument of [Fre95] (section 3.2) can be used to show that if the weak learner is such that it returns with probability 1 − δ0 a hypothesis whose error is no more than 0 = 1/2 − γ < 1/2, then SFboost returns with high probability 1−δ (∀δ > 0) a symmetric function whose error is no more than (∀ > 0), after a reasonable number of rounds (T ), and provided |LS| is large enough. The sample size is lowerbounded by a quantity almost linear in (1/)(ln(1/δ) + γ −2 ln Q), where Q is the quantity of weak hypotheses available to the weak learner at each call. For classes used in practice with a reasonable exponential cardinal (depth-bounded decision trees, monomials, etc.), our lowerbound can be very small, and make SFboost an efficient Boosting algorithm in the sense of [Sch90]. For infinite-cardinality classes, a more complicated argument is needed which integrates the VC dimension. The emphasis on the Boosting ability of SFboost is important, as throughout the literature, a rapidly increasing number of so-called “boosting” algorithms have been developed. However, with respect to the original theory of [KV89, Sch90], only a few of them (such as [Fre95, FS97, Sch90, SS98]) are really Boosting algorithms. SFboost Is an Agnostic/Robust Learning Algorithm. The agnostic learning model of [KSS94] (cast in approximation complexity on the form of a robust learning model [HS92]) is a relaxed variant of the PAC learning model [Val84], which virtually makes no assumption about the target concept used to label the

326


examples. In this model, the learner draws examples according to a fixed but unknown distribution D, and is given two parameters , δ > 0. In time polynomial in 1/, 1/δ, n, the learner has to return a hypothesis from its class of concept representation (e.g. bounded-depth decision trees), such that with probability > 1 − δ, the error of this hypothesis is no more than the best achievable in the class plus . If we look further into the formula computing v T , it admits another expression which is actually T 1 |{(xi , +) ∈ LS : t=1 ht (xi ) = j}| v T [j] = ln . 2 |{(xi , −) ∈ LS : Tt=1 ht (xi ) = j}| Our way to compute the class associated to the buckets is the same as a well known agnostic learning algorithm for symmetric functions [Bou92] with record complexity. Therefore, for each possible t, SFboost agnostically learns the target concept. To our knowledge, SFboost is the first Boosting algorithm which is also an agnostic/robust learning algorithm. As linear separators are not robustly learnable [HS92] (modulo adequate complexity hypotheses), such a property is definitely out of reach for AdaBoost and all its related algorithms. SFboost Has Optimal Malicious-Noise Tolerance. [KL88] have studied the learnability of concepts when data can be corrupted by errors from whom absolutely no assumption can be made. Their “malicious noise” model takes place in the same setting as the PAC learning model, but with an adversary which manipulates any requested example with probability β, to return something from which nothing can be assumed. This adversary has unbounded computational resources, knows everything about the task (the target concept to be learned, the distribution D), and knows the internal state of the learner. [KL88] show that the maximal amount of such malicious noise is Ω(), where is the error parameter of the PAC-learning model (see before, or [Val84]). They also show that the class of symmetric functions admit an algorithm which does not only tolerate this optimal bound, but also with a minimal sample complexity. It turns out that at each time t, the symmetric function SFboost builds is the same as the one which would be chosen in theorem 11 of [KL88]. To the best of our knowledge, no other Boosting algorithm is known to bring such a noise resistance. Note that noise handling is one of the main problems of Boosting algorithms [BK99].

4

Experiments

Numerical Problems. As SFboost proceeds by repeatedly splitting the training sample into subsamples, there may be some problems to compute the components of v T (eq. 1) whenever the weight of one class approaches zero, or equals zero in a bucket, which in turn would severely bias the update of distributions Dt and Dt . To avoid such situations, we have chosen to follow experimentally the setup proned by [SS98], which boils down to replacing eq. 1 by what follows: v t [0] =

1 2

D+

exp(v t−1 [0])+

ln D−(t−1,0)0exp(−v (t−1,0)0

t−1

, v t [t] = [0])+

1 2

D+

exp(v t−1 [t−1])+

ln D−(t−1,t−1)1exp(−v (t−1,t−1)1

t−1 [t−1])+

, and

A

Robust Boosting Algorithm

Fig. 1. Scatterplots of the errors of SFBOOST(x) vs ADABOOST (y). Points above the y = x line indicate datasets for which SFBOOST performs better. See text for details

Fig. 2. Cumulative distributions of the margin psp(.) as in eq. 4, for three problems. The ILt= x" values show the respective margin distribution curves when the symmetric function contaim T = x rules

~ h ,,,- - ~exp(vi-~u - ~ I ) + D ~ enp(vi-l [j])+i exp(-vi-l U-ll)+D; -,,,,,exp(-vt-t Ul)+i

v t [ j ]= $ l n D;-i,,-i,i t). We also iix

t =

i)i

otherwise (0 ij

/

llm, as proposed by [SSSR].

SFboost vs. AdaBoost. In this experiments, we have chosen to test the behavior of SFBOOST against its principal opponent: ADABOOST. Each algorithm was tested on a set of 30 problems, most of which come from the UCI repository of ML database [BKF&8]. The error is evaluated by averaging over a ten-fold stratified cross validation procedure [Qui96]. Finally, on each couple (training set, test set) generated, both algorithms SFBOOST and ADABOOST are ran. For the sake of comparison, we have chosen for the weak learners a simple class of concept representations: monomials (rules) with a maximal number of literals 5 1 for some 1 > 0. Note that whenever 1 = 1 we induce decision stumps [SFBLQ8]. Each algorithm is ran with a iixed value for 1, and requests a number of rules equal to T , for some T > 0. As suggested by theory, the weak learners are de-

328


signed to optimize respectively Zt (ineq. 2 for SFboost ) and the Z of AdaBoost (section 3 in [SS98] for AdaBoost ). The weak learners are also stepwise greedy optimization procedures for their corresponding Z criterion, building each monomial from scratch. Figure 1 summarizes the results obtained over each of the 30 datasets, for couples of values (l, r) ∈ {(1, 10), (2, 10), (2, 20), (2, 50)}. They clearly depicts the ability of SFboost to beat AdaBoost on many of the datasets. We have also observed that, as r increases for fixed l, the gap between SFboost and AdaBoost tends to increase, but with the same best algorithm: domains for which SFboost performs better than AdaBoost at fixed r, l tend to be domains for which SFboost shall perform even better when increasing r, and reciprocally for AdaBoost. We emphasize the fact that our choice to use monomials was simply for implementation purposes. Only theory, and the weak learning assumptions of AdaBoost or SFboost, could guide reliably through the choice of a more or less complicated class of concept representation to address a domain. Unfortunately, nothing can a priori states on an arbitrary domain that some algorithm satisfies the weak learning hypothesis better than another one. So far, only an induction scheme has seemingly brought an experimental accurate answer to the building of weak hypotheses, and has been supported by theoretical comparison studies [KM96]. This scheme has previously been successful to build formulas such as decision trees [Qui94], decision lists [NJ98], and, of course, our simple rules in our experiments with SFboost and AdaBoost. Figure 2 presents the margin distribution for SFboost over one run, for three problems of the UCI (Monks 1, 2, 3) [BKM98] over which we ran SFboost for a maximum of r = 800 iterations (with l = 3). They display nicely the decreasing of the training error. They also display the decreasing of the maximal margin with r, but the fractions of examples whose margin is no more than reasonable positive thresholds also decreases, which accounts for a concentration of the examples near positive, reasonably large margins. Noise Handling. Usual boosting algorithms are well known to be sensitive to noise [BK99]. In the case of SFboost, theory suggests that the algorithm should handle reasonable noise, and be at least as good as AdaBoost, if not better. On 28 out of the 30 problems (for lisibility purposes), we have ran SFboost and AdaBoost again with (l = 4, r = 20), either on the original datasets, or when plugging 10% class noise on the examples. Figure 3 (left table) shows the results obtained. The plot for each dataset gives three indications: the error comparison before noise, that after, and which algorithm is the most resistant to noise addition (if the slope is > 1, it is SFboost). There are two parts on the plot: datasets plotted before noise with an (x, y) such that approximately x, y ≤ .3, and the others. The second set contains problems that were so “hard” to handle without noise that noise addition sometimes even reduces the errors. A more reliable study can be carried out with the first set of problems. In that set, out of 17 problems, only 3 are problems for which the segment slope is < 1. In other words, there are 14 problems on which SFboost is more resistant to noise addition. A simple sign test reveals a p = 0.00636 threshold probability to reject the


0.5

0.5

SFBoost err.

0.25

0

0.3

0.25

0 0

0.25

0.5

0

SFBoost* err.

0.5

0.5 l=2, r=50 SFBoost err.

l=2, r=20

0.1

0.25 SFBoost* err.

0.5

0.2 SFBoost err.

AdaBoost err.

0.4

l=2, r=10 SFBoost err.

l=1, r=10

0.5

329

0.25

0.25

0 0

0.1

0.2

0.3 SFBoost err.

0.4

0.5

0

0 0

0.25 SFBoost* err.

0.5

0

0.25

0.5

SFBoost* err.

Fig. 3. Left table: scatterplots for the errors of SFboost (solid line) vs AdaBoost (dashed line) on 28 out of the 30 datasets, with and without 10% class noise (l = 4, r = 20). The squares depicts the errors without noise; dashed lines link them with the errors on their corresponding noisy dataset. Right table: error scatterplots of SFboost∗ (x) vs SFboost (y) for the 30 datasets with (l, r) = (1, 10), (2, 10), (2, 20) and (2, 50) for SFboost (see text for details) hypothesis for an identical behavior against noise. Therefore, SFboost seems to handle noise in our experiments in a better way than AdaBoost does. Stopping Boosting. There is a lack of criteria to choose the T parameter for Boosting. In the case of SFboost, we have tried a very simple alternative, suggested by ineq. 7. When putting T θ = K = 0, ineq. 7 shows that the training error is upperbounded by P = t=0 Zt . But each Zt can sometimes be > 1, on hard enough domains. This suggests that, out of a classifier containing T weak hypotheses h1 , h2 , ..., hT , one could choose the one containing h1 , h2 , ..., hT ∗ ≤T which minimizes P , out of the T + 1 possible subclassifiers (with the empty one). This is a simple, yet reasonable test to carry out. Figure 3 (right table) reports the results of SFboost against this variant called SFboost∗ on the 30 datasets, where HT ∗ is built after T = 50 iterations. SFboost∗ beats SFboost on most of the datasets, even when the points gather around the y = x line as r increases in SFboost : for r = 50, SFboost∗ still beats SFboost on 21 datasets, and is beaten only on 3 of them.

References [BK99]

E. Bauer and R. Kohavi. An empirical comparison of voting classification algorithms: Bagging, boosting, and variants. Machine Learning Journal, 36:105–139, 1999. 319, 320, 326, 328

330


[BKM98]

C. L. Blake, E. Keogh, and C. J. Merz. UCI repository of machine learning databases, 1998. http://www.ics.uci.edu/∼mlearn/MLRepository.html. 327, 328 [Bou92] S. Boucheron. Théorie de l’apprentissage, de l’approche formelle aux enjeux cognitifs. Hermes, 1992. 321, 326 [Bre96a] L. Breiman. Bagging predictors. Machine Learning Journal, 24:123–140, 1996. 319, 320 [Bre96b] L. Breiman. Bias, Variance and Arcing classifiers. Technical Report 460, UC Berkeley, 1996. 319, 320 [FHT00] J. Friedman, T. Hastie, and R. Tibshirani. Additive Logistic Regression: A Statistical View of Boosting. Annals of Statistics, 28:337–374, 2000. 320 [Fre95] Y. Freund. Boosting a weak learning algorithm by majority. Information and Computation, 121:256–285, 1995. 325 [FS97] Y. Freund and R. E. Schapire. A Decision-Theoretic generalization of online learning and an application to Boosting. Journal of Computer and System Sciences, 55:119–139, 1997. 320, 321, 325 [HS92] K-U. H¨ offgen and H. U. Simon. Robust trainability of single neurons. In Proceedings of the 5 th International Conference on Computational Learning Theory, 1992. 320, 321, 325, 326 [Kea88] M. J. Kearns. Thoughts on Hypothesis Boosting, 1988. ML class project. 319, 320 [KL88] M. J. Kearns and M. Li. Learning in the presence of malicious errors. In Proceedings of the 20 th ACM Symposium on the Theory of Computing, pages 267–280, 1988. 320, 321, 322, 326 [KM96] M. J. Kearns and Y. Mansour. On the boosting ability of top-down decision tree learning algorithms. Proceedings of the 28th Annual ACM Symposium on the Theory of Computing, pages 459–468, 1996. 328 [KSS94] M. J. Kearns, R. E. Schapire, and L. M. Sellie. Toward efficient agnostic learning. Machine Learning Journal, 17:115–141, 1994. 320, 325 [KV89] M. J. Kearns and L. Valiant. Cryptographic limitations on learning boolean formulae and finite automata. Proceedings of the 21 th ACM Symposium on the Theory of Computing, pages 433–444, 1989. 319, 325 [KV94] M. J. Kearns and U. V. Vazirani. An Introduction to Computational Learning Theory. M. I. T. Press, 1994. 319 [NG95] R. Nock and O. Gascuel. On learning decision committees. In Proceedings of the 12 th International Conference on Machine Learning, pages 413–420, 1995. 320 [NJ98] R. Nock and P. Jappy. On the power of decision lists. In Proceedings of the 15 th International Conference on Machine Learning, pages 413–420, 1998. 328 [Qui94] J. R. Quinlan. C4.5 : programs for machine learning. Morgan Kaufmann, 1994. 328 [Qui96] J. R. Quinlan. Bagging, Boosting and C4.5. In Proceedings of AAAI’96, pages 725–730, 1996. 327 [Sch90] R. E. Schapire. The strength of weak learnability. Machine Learning Journal, pages 197–227, 1990. 319, 321, 325 [SFBL98] R. E. Schapire, Y. Freund, P. Bartlett, and W. S. Lee. Boosting the Margin: A new explanation for the effectiveness of Voting methods. Annals of statistics, 26:1651–1686, 1998. 321, 324, 327

A Robust Boosting Algorithm [SS98]

[Val84] [Vap98]

331

R. E. Schapire and Y. Singer. Improved boosting algorithms using confidence-rated predictions. In Proceedings of the 11 th International Conference on Computational Learning Theory, pages 80–91, 1998. 320, 321, 322, 325, 326, 327, 328 L. G. Valiant. A theory of the learnable. Communications of the ACM, 27:1134–1142, 1984. 319, 320, 325, 326 V. Vapnik. Statistical Learning Theory. John Wiley, 1998. 320

Case Exchange Strategies in Multiagent Learning Santiago Onta˜ noń and Enric Plaza IIIA, Artificial Intelligence Research Institute CSIC, Spanish Council for Scientific Research Campus UAB, 08193 Bellaterra, Catalonia (Spain) {santi,enric}@iiia.csic.es http://www.iiia.csic.es

Abstract. Multiagent systems offer a new paradigm to organize AI applications. We focus on the application of Case-Based Reasoning to Multiagent systems. CBR offers the individual agents the capability of autonomously learn from experience. In this paper we present a framework for collaboration among agents that use CBR. We present explicit strategies for case retain where the agents take in consideration that they are not learning in isolation but in a multiagent system. We also present case bartering as an effective strategy when the agents have a biased view of the data. The outcome of both case retain and bartering is an improvement of individual agent performance and overall multiagent system performance. We also present empirical results comparing all the strategies proposed. Keywords: Cooperative CBR, Multiagent CBR, Collaboration Policies, Bartering, Multiagent Learning.

1

Introduction

Multiagent systems offer a new paradigm to organize AI applications. Our goal is to develop techniques to integrate CBR into applications that are developed as multiagent systems. CBR offers the multiagent system paradigm the capability of autonomously learn from experience. The individual case bases of the CBR agents are the main issue here. If they are not properly maintained, the overall system behavior will be suboptimal. These case bases must be maintained having in mind that the agents are not isolated, but inside a multiagent system. This enables the agent not to learn only from its own experience, but collaborating with the other agents in the system. The lacks in the case bases of some agents can be compensated by the experience of other agents in the system. In a real system, there will be agents that can very often obtain certain kind of cases, and that will very seldom obtain other types of cases. It will be beneficial for two agents if they reach an agreement to trade cases. This is a very well known strategy in the human history called bartering. Using case bartering, agents that have a lot of cases of some kind will T. Elomaa et al. (Eds.): ECML, LNAI 2430, pp. 331–344, 2002. c Springer-Verlag Berlin Heidelberg 2002

332

Santiago Onta˜ n´ on and Enric Plaza

give them to another agents in return to more interesting cases for them, and both will profit by improving their performance. Our research focuses on the scenario of separate case bases that we want to use in a decentralized fashion by means of a multiagent system, that is to say a collection of CBR agents that manage individual case bases and can communicate (and collaborate) with other CBR agents. Separate case bases make sense for different reasons like privacy or efficiency. If the case bases are owned by some organizations, perhaps they are not willing to donate the contents of its case bases to a centralized one where CBR can be applied. Moreover, in the case that the case bases where not private, more problems can arise from having all the cases in a single one, such as efficiency, storage or maintenance problems [5]. All these problems suggest difficulties that may be avoided by having separate case bases. In this paper we focus in multiagent systems, where individual agents learn from its own experience using CBR. We show how the agents can improve its learning efficiency by collaborating with other agents, and show results comparing several strategies. The structure of the paper is as follows. Section 2 presents the collaboration scheme that the agents use. Section 3 explains the strategies used by the agents to retain cases in their individual case bases . Then, section 4 presents a brief description of the bartering process. Finally, The experiments are explained in section 5 and the paper closes with related work and conclusion sections.

2

Multiagent Learning

A multiagent CBR (MAC) system M = {(Ai , Ci )}i=1...n is composed on n agents, where each agent Ai has a case base Ci . In this framework we restrict ourselves to analytical tasks, i.e. tasks (like classification) where the solution is achieved by selecting from an enumerated set of solutions K = {S1 . . . SK }. A case base Ci = {(Pj , Sk )}j=1...N is a collection of pairs problem/solution. When an agent Ai asks another agent Aj help to solve a problem the interaction protocol is as follows. First, Ai sends a problem description P to Aj . Second, after Aj has tried to solve P using its case base Cj , it sends back a message that is either :sorry (if it cannot solve P ) or a solution endorsement record (SER). A SER has the form Sk , P, Aj , meaning that the agent Aj has found Sk as the most plausible solution for the problem P . Voting Scheme. The voting scheme defines the mechanism by which an agent reaches an aggregate solution from a collection of SERs coming from other agents. Each SER is seen as a vote. Aggregating the votes from different agents for each class, we can now obtain the winning class as the class with the maximum number of votes. We will show now the Committee collaboration policy that uses this voting scheme (see [6] for a detailed explanation and comparison of several collaboration policies, and a generalized version of the voting scheme to allow more complex CBR methods).

Case Exchange Strategies in Multiagent Learning

333

Committee Policy. In this collaboration policy the agent members of a MAC system M are viewed as a committee. An agent Ai that has to solve a problem P, sends it to all the other agents in M. Each agent Aj that has received P sends a solution endorsement record Sk , P, Aj to Ai . The initiating agent Ai uses the voting scheme above upon all SERs, i.e. its own SER and the SERs of all the other agents in the multiagent system. The problem’s solution is the class with maximum number of votes. In a single agent scenario, when an agent has the opportunity to learn a new case, the agent only has to decide whether the new case will improve its case base or not. Several retain policies exist to take this decision [2, 9]. But when we are in a multiagent scenario, new factor must be considered. Imagine the following situation: an agent Ai has the opportunity to learn a new case C, but decides that C is not interesting to him. But there is another agent Aj in the system, that could obtain a great benefit from learning the case C. It would be beneficial for both agents, that the agent Ai retained the case, and then give or sell it to Aj . Two different scenarios may be considered, when there are ownership rights over the cases and when the agents are free to make copies of the cases to send them to other agents. We will call the first scenario the non-copy scenario, and the second one the copy scenario. Several strategies for retaining cases and bargaining with the retained cases can be defined for each scenario. The learning process in our agents has been divided in two subprocesses: The case retain process, and the case bartering process. During the case retain process the agent that receives the new case decides whether to retain or not the new case, and whether to offer the new case to the other agents or not. An alternative to offer the cases to other agents for free, is to offer them in change of more interesting cases. This is exactly what the case bartering process consists on. Thus, when an agent has recopilated some cases that are not interesting for him, he can interchange them for more interesting cases during a bartering process. This bartering process has not to be engaged necessary each time an agent learns a new case, but just when the agents decide that they have enough cases to trade with. In the following sections we will first describe all the strategies that we have experimented with, and then we will give a brief description of the bartering process.

3

Case Retain

In this section, we will explain with detail all the strategies for the case retain process used in the experiments. This case retain strategies are used when an agent has the opportunity to learn a new case, and has to decide whether to retain it or not. Each case retain strategy is composed on two different policies: the individual retain policy and the offering policy. For the individual retain policy, we have experimented with three options: Never retain (NR), where the agent that has

334


the opportunity to learn a new case never retains the case; Always retain (AR), where the agent always retains the case and When-Interesting retain (WIR), where the agent only retains cases founded interesting. Notice that we can define the interestingness of a case in several ways. In our experiments the criterion for a case being interesting for an agent is when that case is incorrectly solved by the agent. And for the offering policy, we have only two options: Never offer (NO), where the agent that has the opportunity to learn a new case never offers it to any other agent in the system and Always offer (AO), where the agent always asks if any of the other agents is interested in each case the agent has the opportunity to learn. Now, combining all these options, we can define all the possible case retain strategies for both scenarios: copy scenario and non-copy scenario. In the following subsections, all the possible combinations that have sense for both scenarios are explained. Non-copy Scenario Strategies. The following combinations have sense for the non-copy scenario: – Never retain - Never offer strategy (NR-NO): The agents never retain the cases neither offer it to any other agent. Therefore, this is equivalent to a system where the agents do not learn from its experience. – Always retain - Never offer strategy (AR-NO): The agent has the opportunity to learn a new case always retains it, but never offer it to the other agents. In this case, every agent works as if learning in isolation. And all the collaborative work is delegated to the case bartering process. – When-Interesting retain - Never offer strategy (WIR-NO): Equivalent to the previous one, but the agent only retains the case if it is interesting for him. – When-Interesting retain - Always offer strategy (WIR-AO-non-copy): In this strategy, the agent Ai that has the opportunity to learn a new case, retains the case only if deemed interesting. If the case is not retained, it is offered to the other agents. Then, as we are in the non-copy scenario, the agent has to choose just one of the agents that have answered requesting the case to send only one copy of it. Several strategies can be used to make this selection, but in the experiments this is made randomly. Copy Scenario Strategies. The NR-NO, AR-NO and WIR-NO strategies are the same than in the non-copy scenario. Thus, the only new strategy that can be applied in the copy scenario is: When-Interesting retain - Always offer strategy (WIR-AO-copy) where the agent that has the opportunity to learn a new case, retains the case only if deemed interesting. Then, it is offered to the other agents. The, a copy of the case is sent to each agents that answers requesting a copy. Notice that now this is possible because we are in the copy scenario. There is another combination of policies that generates a new strategy: Always retain - Always offer strategy, where the cases are always retained by the


335

agent. And then, offered to the other agents. This is a non interesting strategy although, because all the agents in the system will have access exactly to the same cases and will retain all of them. Therefore, as all the agents will have exactly the same case bases there is no reason to use a multiagent system instead of a single agent that centralizes all the cases.

4

Case Bartering

In the previous section, we have explained the case retain process strategies used by the agents. Now, in this section, we will give a brief description of the case bartering process, see [7] for a complete description. Previous results [6] have shown that agents can obtain better results using the Committee collaboration policy than working alone. However, those experiments assumed that every agent had a representative (with respect to the whole collection of cases) sample of cases in its individual case base. When one agent’s case base is not representative we say it is biased, and the Committee accuracy starts decreasing. Case bartering addresses this problem, and each agent will exchange cases with other agents in order to improve the representativeness (i.e. diminish the bias) of its case base. 4.1

Case Base Bias

The first thing we have to define is the way that the agents measure its case base bias. Let be di = {d1i , . . . , dK i } the individual distribution of cases for an agent Ai , where dji is the number of cases with solution Sj ∈ K in the the case base of Ai . Now, we can estimate the overall distribution of cases D = {D1 , . . . , DK } where n n K Dj = ( i=1 dji )/( i=1 l=1 dli ) is the estimated probability of the class Si . To measure how far is the case base Ci of a given agent Ai of being a representative sample of the overall distribution we will define the Individual Case Base (ICB ) bias, as the square distance between the distribution of cases D and the (normalized) individual distribution of cases obtained from di : ICB(Ci ) =

K k=1

dki

Dk − K

j=1

2 dji

It has been empirically shown [7] that when the ICB bias is high (and thus the individual case base representativeness is low), the agents using the Committee policy obtain lower accuracies. 4.2

Bartering Offers

The way bartering has to reduce the ICB bias of a case base is through case interchanging. In order to interchange cases between two agents, they must reach a bartering agreement. Therefore, there must be an offering agent Ai that sends

336


an offer to another agent Aj . Then Aj has to evaluate whether the offer of interchanging cases with Ai is interesting or not, and accept or reject the offer. If the offer is accepted, we say that Ai and Aj have reached a bartering agreement, and they will interchange the cases in the offer. Formally an offer is a tuple o = Ai , Aj , Sk1 , Sk2 where Ai is the offering agent, Aj is the receiver of the offer, and Sk1 and Sk2 are two solution classes, meaning that the agent Ai will send one of its cases with solution Sk2 and Aj will send one of its cases with solution Sk1 . The Case Bartering protocols do not force to use some concrete strategy to accept or to send offers, so each agent can have its own strategy. However, in our experiments each agent follow the same strategy. Let us start with the simpler one. When an agent receives a set of offers, it has to choose which of these offers to accept and which not. In our experiments the agents use the simple rule of accepting every offer that reduces its own ICB bias. Thus, we will define the set of interesting offers Interesting(O, Ai ) of a set of offers O for an agent Ai as those offers that will reduce the ICB bias of Ai . The strategy to make offers in our experiments is slightly more complicated. In [7] the agents used a deterministic strategy to make offers, but for the experiments reported here, we have chosen a probabilistic strategy which obtains better results. Each agent Ai decide which offers to make in the following way: from the set possible solution classes K, each agent choose the set K ⊆ K of those solution classes they are interested in (i.e. those classes that incrementing the number of cases with that solution class, will diminish the ICB bias measure). For each class Sk1 ∈ K , the agent will send one bartering offer to an agent Aj ∈ A. This agent Aj is chosen probabilistically, and the probability of an agent to be chosen as Aj is a function of the number of cases that the agent have with solution class Sk1 (as more cases, higher probability). Now, the agent Ai has to decide which solution class Sk2 ∈ K will offer to Aj in change for the class Sk1 . The solution class Sk2 ∈ K (where K ⊆ K is the subset of solution classes that decreasing the number of cases with that solution class, will diminish the ICB bias measure) is also chosen probabilistically, and the probability of each solution class to be chosen is a function of the number of cases that Ai has of that solution class (as more cases, higher probability). 4.3

Case Bartering Protocol

Using the previous strategies, two different protocols for Case Bartering have been experimented. The first one is called the Simultaneous Case Bartering Protocol, and the second one the Token-Passing Case Bartering Protocol. However, since the experiments presented in this paper use only the second one, only the Token-Passing protocol is going to be explained here. When an agent member of the MAC wants to enter in the bartering process, it sends an initiating message to all the other agents in the MAC. Then all the other agents answer whether or not they enter the bargaining process. This initiating message contains the parameters for bartering: a parameter tO ,


337

corresponding to the time period that the agents have to make offers; a parameter tA , corresponding to the time period that the agents have to send the accept messages; the number n of agents taking part in the bartering, and Rmax , the maximum number of bartering rounds that the bartering will have. Once the agents have answered to this initial message, the bartering starts. The main characteristic of this protocol is the Token-Passing mechanism, so that only the agent who has the Token can make offers to the others. 1. The initiating agent sends a start message containing the the protocol parameters (tO , tA , and Rmax ). 2. Each agent broadcasts its local statistics di . 3. When all agents have send di , they are able to compute the overall distribution estimation D. 4. Each agent computes the ICB bias of all the agents taking part in the bartering (including itself), and sorts them in decreasing order. This defines the order in which the Token will be passed through. 5. The agent with higher ICB bias is the first to have the Token. So, the initiating agent gives the token to him. 6. The agent who has the Token sends its bartering offers. 7. When the time tO is reached each agent chooses the subset of accepted offers from the set of received offers from the owner of the token and sends accept messages. 8. When the maximum time tA is over, all the unaccepted offers are considered as rejected. 9. Each agent broadcasts its new individual distribution di . 10. When all agents have send di , three different situations may arise: (a) If there are agents that still haven’t owned the token in the current round, the owner of the token gives it to the next agent and the protocol moves to state 6. (b) If every agent has owned the token once in this round, there have been some interchanged cases and the maximum number of iterations Rmax still has not been reached, the protocol moves to state 4. (c) If every agent has owned the token once in this round, but there have been no interchanged cases or the maximum number of iterations Rmax has been reached, the protocol moves to state 11. 11. If there have been no interchanged cases, the Case Bartering Protocol ends, otherwise the protocol moves to state 4. Notice that the procotol does not specify when the agents have to barter the cases. It only defines a way to reach bartering agreements. It’s a matter of the agents when they will really interchange the cases.

5


In this section we want to compare the classification accuracy of the Committee collaboration policy using all the strategies presented in this paper. We also present results concerning case base sizes.

338


We use the marine sponge identification (classification) problem as our test bed. Sponge classification is interesting because the difficulties arise from the morphological plasticity of the species, and from the incomplete knowledge of many of their biological and cytological features. Moreover, benthology specialists are distributed around the world and they have experience in different benthos that spawn species with different characteristics due to the local habitat conditions. We have designed an experimental suite with a case base of 280 marine sponges pertaining to three different orders of the Demospongiae class (Astrophorida, Hadromerida and Axinellida). In each experimental run the whole collection of cases is divided in two sets, a training set (that contains a 10% of the cases), and a test set (that contains a 90% of the cases). The training set is distributed among the agents, and then incremental learning is performed with the test set. Each problem in the test set arrive randomly to one agent in the MAC. The goal of the agent receiving a problem is to identify the correct biological order given the description of a new sponge. Once an agent has received a problem, the MAC will use the Committee collaboration policy to obtain the prediction. Since our experiments are supervised learning ones, after the committee has solved the problem, there is a supervisor that tells the agent receiver of the problem which was the correct solution. After that, the retain policy is applied. In order to test the generality of the strategies, we have tested them using systems with 3, 5 and 8 agents. Each agent apply the nearest neighbor rule to solve the problems. The results presented here are the average of 50 experimental runs. For experimentation purposes, the agents do not receive the problems randomly. We force biased case bases in every agent by increasing the probability of each agent to receive cases of some classes and decreasing the probability to receive cases of some other classes. This is done both in the training phase and in the test phase. Therefore, each agent will have a biased view of the data. Figure 1 shows the learning curve for several multiagent systems and using several retain strategies and without using bartering. The three charts shown in Figure 1 correspond to multiagent systems composed of 3, 5 and 8 agents respectively. For each multiagent system, 5 strategies have been tested: NR-NO, AR-NO, WIR-NO, WIR-AO-copy and WIR-AO-non-copy. The figure shows the learning curve for each strategy. The horizontal axis of Figure 1 represents the number of problems that the agents have received of the test set. The baseline for the comparison is the NR-NO strategy, where the agents do not retain any cases, and therefore (as we can see in the figure) the agents do not learn, resulting in an horizontal learning curve around an accuracy of 50% in all the multiagent systems. This is because the training set is extremely small, 28 cases. The Committee collaboration policy has been proven to obtain results above 88% when the agents have a reasonable number of cases [6]. Considering the other four strategies we can see that, in all the multiagent systems, there are two pairs of strategies with similar learning curves. Specifically, the AR-NO and WIR-NO have nearly the same learning curve, and there-


339

5 Agent Accuracy comparison

3 Agent Accuracy comparison 100

100

90

90

80

80

70

70

60

60

50

50

40

40

30

30

20

20

10

10 0

0 0

50

100

150

200

0

50

100

150

200

When-interesting retain - Always offer (non-copy scenario)

8 Agent Accuracy comparison 100 90

When-interesting retain - Always offer (copy scenario)

80 70

When-interesting retain - Never offer

60 50 40

Always retain - Never offer

30 20

Never retain - Never offer

10 0 0

50

100

150

200

Fig. 1. Accuracy comparison for several configurations without using bartering

fore we cannot distinguish them. They both start from an accuracy of 50% and end with an accuracy around 81%. Therefore, they are significantly better than the NR-NO strategy. The WIR-NO-copy and WIR-NO-non-copy also have very similar learning curves. Both starting at around 50% in all the scenarios, and arriving to 90% in the case of the WIR-NO-non-copy and 88% in the WIR-NO-copy also in all the scenarios. Summarizing we can say that (when the agents do not use case bartering) the strategies that use When-interesting retain and Always retain policies are not distinghishable in terms of accuracy. The strategies that use the Always offer policy (WIR-AO-copy and WIR-AO-non-copy) obtain higher accuracy than the strategies that use the Never offer policy (AR-NO and WIR-NO). Thus, it is always better for the Committee collaboration policy that the agents that receive cases offer them to the other agents; the reason is that these cases perhaps are

340


Table 1. Average case base size of each agent at the end of the learning process for agents using and not using bartering without bartering with bartering 3 Agents 5 Agents 8 Agents 3 Agents 5 Agents 8 Agents NR-NO 9.33 5.60 3.50 9.33 5.60 3.50 AR-NO 93.33 56.00 35.00 93.33 56.00 35.00 WIR- NO 23.80 14.32 10.70 29.13 19.50 13.64 WIR- AO-copy 58.66 57.42 56.60 59.43 57.42 57.09 WIR- AO-non-copy 45.00 34.42 25.90 44.33 35.14 26.55

not interesting to agent receiving the problem, but may be another agent in the system that founds some of those cases interesting. We can also compare the case base sizes reached after the learning process. The left part of Table 1 shows the average size of each individual case base at the end of the learning process (i.e. when all the 252 cases of the test set have been sent to the agents) when the agents do not use bartering. In all the experiments just 28 cases (the training set) are owned by the agents at the beginning. When the agents use the NR-NO strategy, since they do not retain any new cases, they just keep the initial cases. For instance, we can see in the 3 agents scenario, where the agents have in average a case base of 9.33 cases, that 3 times 9.33 is exactly 28, that is the number of cases initially given to them. Comparing the AR-NO strategy with the WIR-NO strategy (that achieved undistinguishable accuracies), we can see that the case bases sizes obtained with WIR-NO are four times smaller than the case base sizes obtained with AR-NO for the 3 and 5 agents scenarios, and about 3.2 times smaller for the 8 agents scenario. Thus, we can conclude that the WIR-NO strategy is better than ARNO strategy because achieves the same accuracy but with a smaller case base size. A similar comparison can be made with the WIR-AO-copy and WIR-AOnon-copy. Remember that WIR-AO-non-copy and WIR-AO-copy have a similar learning curve, but WIR-AO-non-copy obtains slightly better results (90% vs 88% at the end of the test phase). The case base sizes reached are smaller for the WIR-AO-non-copy than for the WIR-AO-copy strategy. Thus, WIR-AOnon-copy achieves higher accuracy with a smaller case base size. The explanation is that when allowing multiple copies of a case in the system (in WIN-AO-copy), we are increasing the correlation between the case bases of the agents. Moreover, It is known that the combination of uncorrelated classifiers has better results that the combination of correlated ones [4]; increasing the correlation is the cause of WIR-AO-copy achieving a lower accuracy than WIR-AO-non-copy. Figure 2 shows exactly the same experiments as in Figure 1, but with agents using bartering. The agents in our experiments perform bartering each 20 cases of the test phase. Figure 2 shows that with the use of bartering, the accuracy of all the different strategies is increased. The NR-NO strategy gets boosted from an accuracy of 50% to 70% in the 3 agents scenario and to 67% and 60% in


341

5 Agent Accuracy comparison

3 Agent Accuracy comparison 100

100

90

90

80

80

70

70

60

60

50

50

40

40

30

30

20

20

10

10 0

0 0

50

100

150

200

0

50

100

150

200

When-interesting retain - Always offer (non-copy scenario)

8 Agent Accuracy comparison 100 90

When-interesting retain - Always offer (copy scenario)

80 70

When-interesting retain - Never offer

60 50 40

Always retain - Never offer

30 20

Never retain - Never offer

10 0 0

50

100

150

200

Fig. 2. Accuracy comparison for several configurations using bartering

the 5 and 8 agents scenario respectively. Notice that when the agents use the NR-NO, no cases are retained and thus their case base sizes are very small, and that just reducing the bias of the individual case bases, the agents obtained a great improvement. This shows the benefits that bartering can provide to the multiagent system. Figure 2 also shows that the WIR-AO-copy and WIR-AO-non-copy strategies still achieve the highest accuracies, and that their accuracies are not distinguishable. The accuracies of the WIR-NO and AR-NO strategies also improved and are now closer to WIR-AO-copy and WIR-AO-non-copy than without bartering. Moreover, the AR-NO strategy achieves now higher accuracy than WIR-NO. The agents retain more cases in the AR-NO strategy than in the WIR-NO, thus they have more cases to trade with in the bartering process. Therefore, when the agents use bartering, they have an incentive to retain cases because they can later negotiate with them in the bartering process.

342


The right part of Table 1 shows the average size of each individual case base at the end of the learning process when the agents use bartering. We can see that the case base sizes reached are very similar to the case base sizes reached without bartering. Therefore, with bartering we cannot say that AR-NO is better than WIR-NO (as happened without bartering), because AR-NO achieves higher accuracy but greater case base sizes, and WIR-NO has smaller case base size but with a slightly lower accuracy. Summarizing all the experiments presented (with and without bartering), we can say that using bartering the system always obtains an increased accuracy. We have also seen that the strategies where the agents use the Always offer policy also obtains higher accuracies. And that if we let each agent to decide when a case is interesting to be retained (When-Interesting retain) instead of retaining every case (Always retain), we can reduce significantly the case bases with practically no effect on the accuracy. Finally, we can conclude that WhenInteresting Retain - Always offer strategy (with no copy) outperforms all the other strategies, since it obtains the higher accuracies with rather small case bases, and that the use of bartering is always beneficial.

6

Related Work

Related work can be divided in two areas: multiple model learning (where the final solution for a problem is obtained through the aggregation of solutions of individual predictors) and case base competence assessment. A general result on multiple model learning [3] demonstrated that if uncorrelated classifiers with error rate lower than 0.5 are combined then the resulting error rate must be lower than the one made by the individual classifiers. However, these methods do not deal with the issue of “partitioned examples” among different classifiers as we do—they rely on aggregating results from multiple classifiers that have access to all data. The meta-learning approach in [1] is applied to partitioned data. They experiment with a collection of classifiers which have only a subset of the whole case base and they learn new meta-classifiers whose training data are based on predictions of the collection of (base) classifiers. Learning from biased data sets is a well known problem, and many solutions have been proposed. Vucetic and Obradovic [10] propose a method based on a bootstrap algorithm to estimate class probabilities in order to improve the classification accuracy. However, their method does not fit our needs, because it requires the availability of the entire test set. Related work is that of case base competence assessment. We use a very simple measure comparing individual with global distribution of cases; we do not try to assess the areas of competence of (individual) case bases - as proposed by Smyth and McKenna [8]. This work focuses on finding groups of cases that are competent.


7

343

Conclusions and Future Work

We have presented a framework for cooperative Case-Based Reasoning in multiagent systems, where agents can cooperate in order to improve its performance. We have presented also that a market mechanism (bartering) can help the agents and improve the overall performance as well as the individual performance of the agents. The agent autonomy is maintained because all the agents are free, because if an agent do not want to take part in the bartering, he just have to reject all the offers and do not make any one. We have also shown the problem arising when data is distributed over a collection of agents than can have a skewed view of the world (the individual bias). Case bartering shows that the problems derived from distributed data over a collection of agents can be solved using a market-oriented approach. We have presented explicit strategies for the agents to accumulate experience (retaining cases) and to share this experience with the other agents in the system. The outcome of this experience sharing is an improvement of the overall performance of the system (i.e. higher accuracy). But further research is needed in order to find better strategies that allow the agents obtain the higher accuracy with the smallest case base size.

Acknowledgements The authors thank Josep-Llu´ıs Arcos of the IIIA-CSIC for its support and for the development of the Noos agent platform. Support for this work came from CIRIT FI/FAP 2001 grant and projects TIC2000-1414 “eInstitutor” and IST1999-19005 “IBROW”.

References [1] Philip K. Chan and Salvatore J. Stolfo. A comparative evaluation of voting and meta-learning on partitioned data. In Proc. 12th International Conference on Machine Learning, pages 90–98. Morgan Kaufmann, 1995. 342 [2] G. W. Gates. The reduced nearest neighbor rule. IEEE Transactions on Information Theory, 18:431–433, 1972. 333 [3] L. K. Hansen and P. Salamon. Neural networks ensembles. IEEE Transactions on Pattern Analysis and Machine Intelligence, (12):993–1001, 1990. 342 [4] Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. In G. Tesauro, D. Touretzky, and T. Leen, editors, Advances in Neural Information Processing Systems, volume 7, pages 231–238. The MIT Press, 1995. 340 [5] David B. Leake and Raja Sooriamurthi. When two case bases are better than one: Exploiting multiple case bases. In ICCBR, pages 321–335, 2001. 332 [6] S. Onta˜ n´ on and E. Plaza. Learning when to collaborate among learning agents. In 12th European Conference on Machine Learning, 2001. 332, 335, 338 [7] S. Onta˜ n´ on and E. Plaza. A bartering aproach to improve multiagent learning. In 1st International Joint Conference in Autonomous Agents and Multiagent Systems, 2002. 335, 336

344


[8] B. Smyth and E. McKenna. Modelling the competence of case-bases. In EWCBR, pages 208–220, 1998. 342 [9] Barry Smyth and Mark T. Keane. Remembering to forget: A competencepreserving case deletion policy for case-based reasoning systems. In IJCAI, pages 377–383, 1995. 333 [10] S. Vucetic and Z. Obradovic. Classification on data with biased class distribution. In 12th European Conference on Machine Learning, 2001. 342

Inductive Confidence Machines for Regression Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman Department of Computer Science, Royal Holloway, University of London Egham, Surrey TW20 0EX, England {harris,konstant,vovk,alex}@cs.rhul.ac.uk

Abstract. The existing methods of predicting with confidence give good accuracy and confidence values, but quite often are computationally inefficient. Some partial solutions have been suggested in the past. Both the original method and these solutions were based on transductive inference. In this paper we make a radical step of replacing transductive inference with inductive inference and define what we call the Inductive Confidence Machine (ICM); our main concern in this paper is the use of ICM in regression problems. The algorithm proposed in this paper is based on the Ridge Regression procedure (which is usually used for outputting bare predictions) and is much faster than the existing transductive techniques. The inductive approach described in this paper may be the only option available when dealing with large data sets.

1

Introduction

When presented with a test example, traditional machine learning algorithms only output a bare prediction, without any associated confidence values. For example, Support Vector Machine (Vapnik, 1998, Part II) outputs just one number (a bare prediction, as we will say), and one has to rely on the previous experience or relatively loose theoretical upper bounds on the probability of error to gauge the quality of the given prediction. This is also true for the more traditional Ridge Regression (RR) procedure as it is used in machine learning (see, e.g., Saunders, Gammerman, & Vovk, 1998). Gammerman, Vapnik, and Vovk (1998) proposed what we call in this paper “Transductive Confidence Machine” (TCM), which complements the bare predictions with measures of confidence in those predictions. Both Transductive (see, e.g., Proedrou et al., 2001) and Inductive (proposed in this paper) Confidence Machines are currently built on top of the standard machine learning algorithms for outputting bare predictions; we will call the latter the underlying algorithms. TCM suggested in Gammerman et al. (1998) was greatly improved in (Saunders, Gammerman, & Vovk, 1999). Vovk, Gammerman, and Saunders (1999) introduced the universal confidence values: the best confidence values one can hope to obtain. The universal confidence values are defined using the algorithmic theory of randomness (or, in the simplest situations, Kolmogorov complexity; see Li and Vit´ anyi, 1997) and are computable only in a very weak sense (“computable in the limit”). There are reasons to believe that the version T. Elomaa et al. (Eds.): ECML, LNAI 2430, pp. 345–356, 2002. c Springer-Verlag Berlin Heidelberg 2002

346

Harris Papadopoulos et al.

of TCM defined in Saunders et al. (1999), when coupled with a good underlying algorithm, can give confidence values as good as the universal values provided by the algorithmic theory of randomness (Nouretdinov et al., 2001). The main disadvantage of the existing variants of TCM is their relative computational inefficiency. An original motivation behind the idea of transductive inference (Vapnik, 1998) was to obtain more computationally efficient versions of learning algorithms. Whereas this remains an interesting long-term goal, so far in the theory of confident predictions a side-effect of using transduction has been computational inefficiency; for every test example, all computations need to be started from scratch. It was not clear, however, how prediction with confidence could be implemented without resorting to transduction. Saunders, Gammerman, and Vovk (2001) proposed a much more efficient version of TCM; other efficient versions are described in Vovk and Gammerman (2001). This paper makes a much more radical step introducing Inductive Confidence Machine, ICM. The computational efficiency of ICM is almost as good as that of the underlying algorithm. There is some loss in the quality of the confidence values output by the algorithm, but we show that this loss is not too serious. On the other hand, the improvement in the computational efficiency is massive. ICM will be defined in Section 2. In the following section we will prove the validity of the predictive regions it outputs. Finally, in the last section we give some experimental results that measure the efficiency of our algorithm based on those criteria. In the rest of this introductory section we will briefly describe the relevant literature. Computing confidence values is, of course, an established area of statistics. In the non-parametric situations typically considered in machine learning the most relevant notion is that of tolerance regions (Fraser, 1957; Guttman, 1970). What we do in this paper is essentially finding tolerance regions without parametric assumptions, only assuming that the data is generated by some completely unknown i.i.d. distribution (we will call this the i.i.d. assumption). Traditional statistics, however, did not consider, in this context, the high-dimensional problems typical of machine learning, and no methods have been developed in statistics which could compete with TCM and ICM. The two main areas in the mainstream machine learning which come close to providing confidence values similar to those output by TCM and ICM are the Bayesian methods and PAC theory. For detailed discussion, see (Melluish et al., 2001); here our discussion will be very brief. Quite often Bayesian methods make it possible to complement bare predictions with probabilistic measures of their quality (theoretically this is always possible, but in practice there can be great computational difficulties); e.g., Ridge Regression can be obtained as a Bayesian prediction under specific assumptions and then it can be complemented by a measure of its accuracy (such as the variance of the a posteriori distribution). They require, however, strong extra assumptions, which the theory of TCM and ICM avoids. In fact, Bayesian

Inductive Confidence Machines for Regression

347

methods are only applicable if the stochastic mechanism generating the data is known in every detail; in practice, we will rarely be in such a happy situation. (Melluish et al., 2001) show how misleading Bayesian methods can become when their assumptions are violated and how robust TCM results are (ICM results are as robust). PAC theory, in contrast, only makes the general i.i.d. assumption. There are some results, first of all those by Littlestone and Warmuth (1986; see also Cristianini & Shawe-Taylor, Theorem 4.25 and 6.8), which are capable of giving non-trivial confidence values for data sets that might be interesting in practice. However, in order for the PAC methods to give non-trivial results the data set should be particularly clean; they will fail in the vast majority of cases where TCM and ICM produce informative results (see Melluish et al., 2001). The majority of relevant results in the PAC theory are even less satisfactory in this respect: they either involve large explicit constants or do not specify the relevant constants at all (see, e.g., Cristianini and Shawe-Taylor, 2000, Section 4.5).

2

Inductive Confidence Machine

In this paper we are only interested in the problem of regression, with Ridge Regression as the underlying algorithm. In contrast to the original Ridge Regression method, every prediction output by ICM is not a single real value, but a set of possible values, called a predictive region. We are given a training set {(x1 , y1 ), . . . , (xl , yl )} of l examples, where xi ∈ IRn are the attributes and yi ∈ IR are the labels, i = 1, . . . , l, and the attributes of a new example xl+1 ∈ IRn . When fed with a confidence level, such as 99%, ICM is required to find a predictive region such that one can be 99% confident that the label yl+1 of the new example will be covered by that predictive region. The idea of ICM is as follows. We split the training set into two subsets: – the proper training set {(x1 , y1 ), . . . , (xm , ym )} with m < l elements, and – the calibration set {(xm+1 , ym+1 ), . . . , (xl , yl )} with k := l − m elements; m and k are parameters of the algorithm. We apply the Ridge Regression method to the proper training set, and using the derived rule we associate a strangeness measure with every pair (xi , yi ) in the calibration set. This measure can be defined as αi := |ym+i − yˆm+i |, i = 1, . . . , k, (1) where yˆm+i are the predictions given by the derived rule; later we will also consider other definitions. For every potential label y of the new unlabelled example xl+1 we can analogously define αk+1 := |y − yˆl+1 |, where yˆl+1 is the prediction for the new example given by the derived rule. Let us defined the p-value associated with the potential label y as p(y) :=

#{i = 1, . . . , k + 1 : αi ≥ αk+1 } , k+1

348


where #A stands for the number of elements in the set A; to emphasize the dependence on the training set and xl+1 , we will also write p(x1 , y1 , . . . , xl , yl , xl+1 , y) in place of p(y). In Section 3 we will prove that p(y) are indeed valid p-values. Suppose we are given a priori some confidence level 1 − δ, where δ > 0 is a small constant (typically one takes 1% or 5%); sometimes we will say that δ is the significance level. Given the significance level δ, the predictive region output by ICM is {y : p(y) > δ} . (2) In Section 4 we will see that this can be done efficiently.

3

Validity of the Predictive Regions

Recall that valid p-values p(y) should satisfy, for any i.i.d. distribution P and for every significance level δ, P{p(y) ≤ δ} ≤ δ.

(3)

The next proposition shows that (2) defines valid p-values under the general i.i.d. assumption when the randomization is done over the training as well as over the new example (xl+1 , yl+1 ). Proposition 1. For every probability distribution P in IRn × IR and every significance level δ > 0, P l+1 (x1 , y1 , . . . , xl , yl , xl+1 , yl+1 ) : p(x1 , y1 , . . . , xl , yl , xl+1 , yl+1 ) ≤ δ ≤ δ. Proof. We will actually prove the stronger assertion that (3) is true if the randomization is done only over the calibration set and the new example. Let us fix the proper training set x1 , y1 , . . . , xm , ym ; our goal is to prove P k+1 (xm+1 , ym+1 , . . . , xl+1 , yl+1 ) : (4) p(xm+1 , ym+1 , . . . , xl+1 , yl+1 ) ≤ δ ≤ δ. We can imagine that the sequence (xm+1 , ym+1 ), . . . , (xl+1 , yl+1 ) is generated in two stages: – first the unordered set {xm+1 , ym+1 , . . . , xl+1 , yl+1 } is generated;

(5)


349

– one of the (k + 1)! possible orderings {xπ(m+1) , yπ(m+1) , . . . , xπ(l+1) , yπ(l+1) } (where π : {m + 1, . . . , l + 1} → {m + 1, . . . , l + 1} is a permutation) of (5) is chosen (some of these orderings may lead to the same sequence if some example occurs twice in (5)). Already the second stage will ensure (4): indeed, p(yl+1 ) ≤ δ if and only if αl+1 is among the δ(k + 1) largest αi ; since all permutations π are equiprobable, the probability of this event will not exceed δ.

This proof shows that the method of computing α1 , . . . , αk+1 should only satisfy the following condition in order for the computed p-values to be valid: every αi , i = 1, . . . , k + 1, should be computed only from (xm+i , ym+i ), the proper training set, and the unordered set {xm+1 , ym+1 , . . . , xl+1 , yl+1 }, where yl+1 is understood to be the postulated label y of xl+1 . Definition (1) and definition (7) (see section 4) obviously satisfy this requirement. Fix some significance level δ (small positive constant). Proposition 1 shows that ICM is valid in the following sense. Either the ICM prediction is correct (i.e., the prediction region contains the true label yl+1 ) or an event of small (at most δ) probability occurred. If δ is chosen so that we are prepared to ignore events of probability δ, we can rely on the predictive region covering the true label.

4

Explicit ICM

In this section we will give a slightly more explicit representation of ICM. Let us denote by α(1) , . . . , α(k∗ ) the sequence of all αi corresponding to the calibration set sorted in the descending order, with all repetitions deleted; let js := #{αi : αi ≥ α(s) }, s = 1, . . . , k ∗ , be the number of αs at least as large as α(s) (if all αi are different, j1 = 1, j2 = 2, . . .). Fix the confidence level 1 − δ. The “attainable” significance levels will be js js of the form k+1 ; decrease δ, if necessary, so that it is of this form: δ = k+1 for ∗ some s = 1, . . . , k . It can be easily checked that the predictive region output by ICM can be represented as (6) (ˆ yl+1 − α(s) , yˆl+1 + α(s) ), provided the αs are computed according to (1). Notice that the computational overhead of ICM is light; it is almost as efficient as the underlying algorithm. The decision rule is computed from the proper training set only once, and it is applied to the calibration set also only once. The value of s corresponding to the given significance level δ and the value α(s) can be also computed in advance. For every test example we need to apply the decision rule to it to find its yl+1 ; once this is done, computing the predictive region from (6) is trivial.

350


Another Way of Computing αi Definition (1) defines the strangeness of the new example as the error of the decision rule on it. A natural way to make this strangeness measure more precise is to take into account the predicted accuracy of the decision rule f found from the proper training set on a given unlabelled example from {xm+1 , . . . , xl+1 }. Hopefully this should lead to smaller prediction regions. Instead of using the strangeness measure αi = |yi − yî |, we can use yi − yî , (7) αi := σi where σi is an estimate of the accuracy of the decision rule f on xi . More specifically, we take σi := eµi , where µi is the RR prediction of the value ln(|(yi − f (xi )|) for the example xi . The use of the logarithmic scale instead of the direct one ensures that the estimate is always positive; besides, relatively more weight is given to examples with classifications close to f ’s predictions. It is easy to see that when using αi computed from (1) ICM will output predictive intervals of the same length for all test examples. This is not longer the case when (7) is used; the length of the predictive interval will be proportional to the predicted accuracy of f on the new example. What we are actually accomplishing by using (7) is that the predictive regions obtained will be smaller for points where the RR prediction is good and larger for points where it is bad. Fixed Prediction Interval There are two possible modes of using the p-values computed from (2): 1. For a given significance level δ, find a predictive region such that we can be 1 − δ confident that it covers the true label. 2. Given a fixed predictive region, find the maximum level at which we can be confident that the true label will be covered. The first mode corresponds to the regression ICM considered so far. The second mode is essentially what is usually done in classification problems, where a fixed predictive region may represent one of the possible classifications. It is clear that the maximum confidence interval at which a given predictive interval [a, b] is valid will be 1 − js /(k + 1), where s is the maximum number such that yi − a|, |ˆ yi − b|) . α(s) ≥ max (|ˆ

5


The first set of experiments check how reliable the obtained predictive regions are. We count the percentage of wrong predictive intervals; in other words, how many times the algorithm fails to give a predictive region that contains the real label of every test example. In effect this checks empirically the validity of our


351

Table 1. The average success of the predictions made, for different confidence levels using (1) as strangeness measure Kernel Type

Empirical reliability 90% 95% 99% Polynomial 93.6% 97.4% 99.3% RBF 97.5% 98.6% 99.6% ANOVA Splines 97.7% 97.2% 98.8%

Table 2. The average success of the predictions made, for different confidence levels using (7) as strangeness measure Kernel Type

Empirical reliability 90% 95% 99% Polynomial 95.2% 97.8% 99.1% RBF 97.3% 98.8% 99.6% ANOVA Splines 95% 97.6% 99.2%

algorithm, which was proven theoretically in Section 3. We expect that for a large number of examples the percentage of wrong predictions will not exceed (and perhaps will be close to) the specified significance level. A second set of experiments checks the tightness of our predictive regions by calculating the median value of the lengths of all predictive regions obtained for a specific significance level. This gives us a measure of how efficient our algorithm is. We prefer using the median value instead of the mean, because it is more robust: if a few of the predictions are extreme (either very large or very small) due to noise or due to over-fitting, the average will be affected, while the median will remain unchanged. The proposed algorithm has been tested on the Boston Housing data set, which gives the values of houses, ranging from 5K to 50K, depending on 13 attributes. In the experiments 100 splits of this data set have been used, with different examples for the proper training, calibration, and test sets each time. In every split the calibration set consisted of 99 examples, the test set of 25 examples, and the rest of 382 examples was used as the proper training set. In Tables 1 to 4 we give the widths of the predictive regions and the empirical reliability (i.e., the percentage of cases when the true label turned out to be outside the predictive region) of these bounds for specific significance levels (1%, 5%, and 10%) and for specific kernels (Polynomial, RBF, and ANOVA) used in conjunction with RR. The results in Tables 1 and 2 confirm the validity of our algorithm: the rate of successful predictions is at least equal to the desired accuracy. In Tables 3 and 4 we present results about tightness of our predictive regions for both variations of our algorithm. As we can see, in both cases the best results

352


Table 3. The median width of the predictive regions, for different confidence levels using (1) as strangeness measure Kernel Type

Median width 90% 95% 99% Polynomial 9.6 12.6 16.1 RBF 9.9 13.5 29.4 ANOVA Splines 9.5 12.2 15.4

Table 4. The median width of the predictions made, for different accuracy levels using (7) as strangeness measure Kernel Type

Median width 90% 95% 99% Polynomial 9.5 11.8 15.6 RBF 10 12.7 23.5 ANOVA Splines 9.7 11.7 15

Table 5. Comparison of the mean width of the predictive regions, for ICM and TCM Variation1 Algorithm

Mean width 90% 95% 99% ICM 10.8 12.7 17.5 TCM Variant1 12.4 16.7 28.8

Table 6. Comparison of the median width of the predictive regions, for ICM and TCM Variation2 Algorithm

Median width 90% 95% 99% ICM 9.5 11.8 15.6 TCM Variant2 7.5 9.3 18.8

were obtained when we used the ANOVA splines as our kernel function. By comparing the results for the two variations we notice that the method which uses (7) as strangeness value gives, on average, slightly better results. The difference is becoming relatively larger as we move toward higher confidence levels. Figures 1 and 2 complement the information given in Tables 3 and 4 for ANOVA splines by also giving other characteristics of the distribution of the predictive interval widths. These figures show that the distribution of the method which uses (7) as strangeness measure is more spread out, as we would expect.


353

Constant interval sizes

Region width

Anova splines of order 4 with a=10 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5

90%

95% Accuracy level

99%

Fig. 1. Medians, upper and lower quartiles, and 10th and 90th percentile of the distributions of the predictive interval widths for the method using (1) as strangeness value

Finally, in Tables 5 and 6 we compare our algorithm with two variations of TCM which are described in (Melluish, Vovk, Gammerman, 1999) and (Nouretdinov, Melluish, Vovk, 2001), using the polynomial kernel1 . It is obvious that ICM outperforms the first variation of the TCM in all hypothesis tests, while compared with the second variation the difference is small. Though the set of α values in TCM is richer than the one in ICM and the Ridge Regression rule is derived using less examples in the case of induction, this doesn’t seem to worsen the performance of the latter significantly. We also tested the algorithm on the Bank Rejection and the CPU Activity data sets both of which consist of 8192 examples split into 4096 training and 4096 test examples. This was done in order to demonstrate the algorithms ability to handle large sets. The Bank Rejection data set was generated from a simplistic simulator, which simulated the queues in a series of banks. Our task is to predict the rate of rejections(i.e., the fraction of customers that are turned away from the bank because all the open tellers have full queues) depending on 32 attributes. The CPU Activity data set is a collection of a computer systems activity measures collected from a Sun Sparcstation 20/712 with 128 Mbytes of memory running in a multi-user university department. Users would typically be doing a large variety of tasks ranging from accessing the internet, editing files or running 1

In Table 5 we compare the mean widths instead of the median as in (Melluish, Vovk, Gammerman, 1999) only mean widths are reported

354


Variable interval sizes

Region width

Anova splines of order 4 with a=10 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5

90%

95% Accuracy level

99%

Fig. 2. Medians, upper and lower quartiles, and 10th and 90th percentiles of the distributions of the predictive interval widths for the method using (7) as strangeness value Table 7. The median width of the predictive regions, for the Bank Rejection and the CPU Activity data sets Data Set Strangeness measure Bank (1) (7) CPU (1) (7)

Median width 90% 95% 99% 0.29 0.39 0.51 0.24 0.28 0.40 8.89 11.81 16.79 8.77 10.71 15.89

very cpu-bound programs. Our task is to predict the portion of time that the cpus run in tuser mode depending on 12 attributes. The median widths of the predictive regions obtained by the Bank Rejection and the CPU Activity data sets are listed in Table 7. The rate of rejections in the Bank Rejection data set ranges from 0 to 0.7 and the portion of time that the cpus run in user mode in the CPU Activity data set ranges from 0 to 99. So even for a 99% confidence level the second variation of our algorithm gives a predictive region which covers only 57% and 17% of the whole range of labels for each set respectively.


6

355

Conclusions

We have defined ICM, a computationally efficient confidence machine for the regression problem based on inductive inference. In addition to the bare prediction ICM outputs a measure of its accuracy which has a clear probabilistic interpretation. The experimental results obtained give good empirical reliability that is constantly above the specified confidence level. This confirms that the algorithm can be used for obtaining reliable predictions. Furthermore, the width of our predictive regions, is almost as tight as that of the transductive version. The tightness of our predictive regions can be seen by the fact that our best result for the Boston Housing data set, which is given by the second variation of the algorithm (using (7) as strangeness measure), predicts a region that is only 33% of the whole range of house prices at the 99% confidence level.

Acknowledgements We are grateful to David Surkov for useful discussions. This work was partially supported by EPSRC through grants GR/L35812 (“Support Vector and Bayesian learning algorithms”), GR/M14937 (“Predictive complexity: recursiontheoretic variants”), and GR/M16856 (“Comparison of Support Vector Machine and Minimum Message Length methods for induction and prediction”).

References 1. Cristianini, N., & Shawe-Taylor, J. (2000). Support Vector Machines and Other Kernel-based Learning Methods. Cambridge: Cambridge University Press. 2. Fraser, D. A. S. (1957). Non-parametric Methods in Statistics. New York: Wiley. 3. Gammerman, A., Vapnik, V., & Vovk, V. (1998). Learning by transduction. Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence (pp. 148–156). San Francisco: Morgan Kaufmann. 4. Li, M., & Vit´ anyi, P. (1997). An Introduction to Kolmogorov Complexity and Its Applications. Second edition. New York: Springer. 5. Melluish, T., Saunders, C., Nouretdinov, I., & Vovk, V. (2001). Comparing the Bayes and typicalness frameworks. ECML’01. 6. Melluish, T., Vovk, V., & Gammerman, A. (1999). Transduction for Regression Estimation with Confidence. NIPS’99. 7. Nouretdinov, I., Melluish, T., & Vovk, V. (1999). Ridge Regression Confidence Machine. Proceedings of the 18th International Conference on Machine Learning. 8. Nouretdinov, I., Vovk, V., V’yugin, V., & Gammerman, A. (2001). Transductive Confidence Machine is universal. Work in progress. 9. Proedrou, K., Nouretdinov, I., Vovk, V., & Gammerman, A. (2001). Transductive Confidence Machines for Pattern Recognition. Proceedings of the 13th European Conference on Machine Learning. 10. Saunders, C., Gammerman, A., & Vovk, V. (1999). Transduction with confidence and credibility. Proceedings of the 16th International Joint Conference on Artificial Intelligence (pp. 722–726).

356


11. Saunders, C., Gammerman, A., & Vovk, V. (2000). Computationally efficient transductive machines. ALT’00 Proceedings. 12. Vapnik, V. (1998). Statistical Learning Theory. New York: Wiley. 13. Vovk, V., Gammerman, A., & Saunders, C. (1999). Machine-learning applications of algorithmic randomness. Proceedings of the 16th International Conference on Machine Learning (pp. 444–453). 14. Vovk, V., & Gammerman, A. (2001). Algorithmic Theory of Randomness and its Computer Applications. Manuscript. 15. Vovk, V., and Gammerman, A. (1999). Statistical applications of algorithmic randomness. Bulletin of the International Statistical Institute. The 52nd Session. Contributed Papers. Tome LVIII. Book 3 (pp. 469–470).

Macro-Operators in Multirelational Learning: A Search-Space Reduction Technique Lourdes Pe˜ na Castillo and Stefan Wrobel Otto-von-Guericke-University Magdeburg {pena,wrobel}@iws.cs.uni-magdeburg.de

Abstract. Refinement operators are frequently used in the area of multirelational learning (Inductive Logic Programming, ILP) in order to search systematically through a generality order on clauses for a correct theory. Only the clauses reachable by a finite number of applications of a refinement operator are considered by a learning system using this refinement operator; ie. the refinement operator determines the search space of the system. For efficiency reasons, we would like a refinement operator to compute the smallest set of clauses necessary to find a correct theory. In this paper we present a formal method based on macro-operators to reduce the search space defined by a downward refinement operator (ρ) while finding the same theory as the original operator. Basically we define a refinement operator which adds to a clause not only single-literals but also automatically created sequences of literals (macro-operators). This in turn allows us to discard clauses which do not belong to a correct theory. Experimental results show that this technique significantly reduces the search-space and thus accelerates the learning process.

1

Introduction

Typically, a multirelational learning system takes as input background knowledge B, positive examples E + and negative examples E − , and has to find a correct theory T . A correct theory is a set of Horn clauses which implies E + and is consistent1 with respect to E − . This theory is then used to classify unseen examples E ? as positive or negative. To find T the system has to search among permitted clauses (hypothesis space) for a set of clauses with the required properties. For instance, if a e ∈ E + is not implied by T the system should search for a more general theory; on the other hand, if T is not consistent with E − the system should look for a more specific one. Refinement operators are commonly used in multirelational learning systems as a way to systematically search for T . A refinement operator is a function which computes either a set of specializations (downward operator) or generalizations (upward operator) of a clause. Thus, refinement operators allow us to search 1

Note, however, that the condition of consistency is often relaxed and the systems actually try to minimize the classification error over the training data.

T. Elomaa et al. (Eds.): ECML, LNAI 2430, pp. 357–368, 2002. c Springer-Verlag Berlin Heidelberg 2002

358

Lourdes Pe˜ na Castillo and Stefan Wrobel

step-by-step through a generality order2 on clauses for a correct theory. The search space H is then restricted to clauses that can be obtained by successively applying the refinement operator. This approach works if and only if there is a number of refinement steps to every clause in at least one correct theory. [8] For efficiency reasons, we would like a refinement operator ρ to compute the smallest set of clauses necessary to find T . Many downward operators currently used work by adding to a clause C one literal available from B. In this paper we present a method based on automatically created macro-operators [5] to reduce the hypothesis space defined by one of these refinement operators. In the search and planning literature a macro-operator (or macro for short) is a sequence of operators chosen from the primitive “operators” of a problem. We consider our primitive operators the literals available to be added to C, so a macro is a sequence of literals. By refining C using only “valid” literals and macros, we discard a significant number of clauses (hypotheses) which cannot belong to T . Macros are based on the fact that literals exist which if added alone to a clause C affect neither the coverage nor the consistency of C. For example, consider we want a system to learn the concept divisibleBy(x, y) (x is divisible by y) and as part of B we give the literals isN ull(z) and remainder(x, y, z) where x, y are input arguments and z is computed by remainder and used by isN ull. The literal remainder is true for every combination of numeric values for x and y and adding it by itself to C does not help to distinguish between E + and E − . However, the macro remainder(x, y, z), isN ull(z) is certainly more useful; thus our ρ adds to C this macro instead of adding either remainder alone or remainder together with another literal that does not use z. In this paper, we precisely develop the macro-based approach for the class of refinement operators that are lower bounded by a bottom clause, such as the one used in Progol [7]. Specifically, we adapt the refinement operator used in Mio [9] (henceforth referred to as literal-based ρ). The literal-based ρ is then used in our experiments as our comparison point. Experiments on four application domains show that the reduction of the search space obtained using the macro-based ρ produces a significant speed-up of the learning process. Using the macro-based ρ implies to the user almost no extra effort since: 1) a formal characterization of literals which helps to decide whether a literal can be included in a macro without risk of missing a solution is provided, and 2) the mode declaration language is enhanced to allow the user to easily declare these literals. In addition, the macros are domain-independent and constructed automatically by the system based on the mode declarations given by the user. The remaining of this paper is organized as follows. The next section describes the learning system used and the literal-based ρ. Section 3 explains how the search space reduction is achieved. Section 4 defines a macro, the macro-based ρ and the macro generation algorithms. Section 5 discusses our experiments. Related work is briefly surveyed in Section 6 and Section 7 concludes. 2

A generality order determines what constitutes a “specialization” or “generalization” of a clause. Subsumption and logical implication are two of the most widely used generality orders.

Macro-Operators in Multirelational Learning

359

Mio and the Literal-Based ρ

2 2.1

The Learning System

Mio is an example-driven learning system introduced in [9] which uses a Progollike declaration language and, the same as Progol [7], lower bounds the search space with a most specific clause ⊥. This ⊥ is a minimal generalization sufficient to cover an example e ∈ E + . Mio performs a general-to-specific IDA* search of the hypothesis space to find a clause C to add to T . IDA* is guided by the number of literals needed to obtain an I/O-complete clause. An I/O-complete clause has neither unbound output variables in the head nor unbound input variables in the body. In addition, Mio (contrary to Progol) selects stochastically the examples from which it learns, performs parallel search and enforces type strictness. To construct ⊥, the user has to give a set of mode declarations. These mode declarations define the literals from B which can be used as the head and in the body of the clauses in H. In the mode declarations the arguments of a literal are defined either as a constant (#cte), as an output variable (−var), or as an input variable (+var). A literal provides a value if it has a −var and consumes a value if it has a +var. We can say that a literal p is a consumer of literal q if p has at least one +var bound to an output argument value of q (q is then a provider of p). A literal p defined by the mode declarations can appear in ⊥ if and only if p has at least one provider placed before p in ⊥ for each +var ∈ p. The most specific clause is then defined as follows. Definition 1 (Most Specific Clause ⊥ [7]). Let ⊥ be the most specific definite clause (bottom clause) constructed with the literals defined by the mode declarations, background knowledge B and example e ∈ E + such that: (a) B ∧ ⊥ h e (ie. e can be derived in h resolution steps). (b) ⊥∞ ⊥ where ⊥∞ is the (potentially infinite) conjunction of ground literals which are true in all models of B ∧ ¬E (ie. B ∧ ¬E |= ⊥∞ ). The most specific clause can be seen as a sequence of literals where every literal is uniquely mapped to its position in ⊥ starting with the first body literal after the head of ⊥ (ie. h ← 1, . . . , i − 1, i, . . . , n.). We refer henceforth to every literal i (1 ≤ i ≤ n) in ⊥ as the integer corresponding to its position. The literals in ⊥ are placed according to their consumer-provider relationships. For both the literal-based and the macro-based method, we assume that the clauses are evaluated by a heuristic function eval(C) considering the length of C (| C |= number of body literals in C) and the number of positive p and negative n examples covered by C, and that eval(C)3 favours shorter clauses over longer ones with the same coverage and consistency. 3

In Mio, eval(C) = count p covered by(C)/ | C | −count n covered by(C).

360

2.2


Literal-Based ρ

The literal-based refinement operator consists in adding one literal from ⊥ to a clause C. The search space H defined by this operator allows the system to search from the empty clause (✷) to ⊥. Notice that by the definition of ρ given below the hypotheses’ literals keep the topological order of ⊥. Definition 2 (Literal-Based ρ). Let i and j be literals in ⊥ and C be a clause whose last literal is i, then: ρ(C) = {C ∪ {j} | i < j and all + var ∈ j are either bound to −var of literals already in C or to +var in the head of C}. Assume C ∈ ρ(C), then C is heuristically evaluated (eval(C )) iff C is I/O-complete. Then, the search space is the following. Definition 3 (Hypothesis Space H). The hypothesis space H consists of I/O-complete hypotheses whose literals appear in ⊥: H := {C ⊆ ⊥ | C is I/O-complete}. Mio considers a clause C ∈ H as a solution (ie. as a candidate to be included in T ) iff C maximizes eval(C) and satisfies several user-defined parameters such as maximum length allowed, minimum required coverage per clause, etc. Lemma 1. Let C ∈ ρj (Cj ). C can be a solution iff there is no C ∈ ρi (Ci ), i < j such that eval(C) ≥ eval(C ).

3

Reducing the Search Space

As explained in the introduction sometimes there are literals defined in the mode declarations that, when queried, succeed for each combination of input argument values. These literals usually are needed to introduce a variable. The ρ described in Def. 2 produces clauses C ∈ ρ(C) which might vary from C only in having one of these “non-gain”4 literals added. However, since these literals succeed for every combination of input argument values, they modify neither the coverage nor the consistency of C but they do increase its length (ie. eval(C) ≥ eval(C )). Thus C cannot be a solution and could be safely removed from H. One way to avoid that C differs from C only in having one of these “nongain” literals added is to identify such literals in the mode declarations and to use a ρ which does not add “non-gain” literals by themselves to C but only as part of a macro. Since these “non-gain” literals usually contain output variables, we can add them to C together with at least one of their consumers. To declare the “non-gain” literals, we enhance the mode declaration language with the ∗var notation. A ∗var is an output variable argument of a “non-gain” literal. To decide whether a literal should not be added by itself to C, we divide the literals in the following categories based on their consumption/providing properties. 4

The information gain of using this literal is null.


361

Definition 4. Let i be a literal in the body of ⊥. Then: 1. i is called an independent provider iff i has output variables but does not succeed for every combination of input argument values (it does not contain ∗var). (e.g. see literals b and c in Table 1) 2. i is called a dependent provider iff i has output variables and succeeds for every combination of input argument values (it contains at least one ∗var). (e.g. see literal a in Table 1) 3. i is called a head provider iff a −var in the head of ⊥ is bound to an output variable (−var or ∗var) in i. (e.g. see literals b and e in Table 1) 4. i is called an independent consumer iff each of its +var is bound to a +var in the head of ⊥ or to a −var in a provider. (e.g. see literal e in Table 1) 5. i is called a dependent consumer iff at least one of its +var is only provided by ∗var (it is not bound to a +var in the head of ⊥ or to a −var in a provider). (e.g. see literal d in Table 1) To illustrate the literal characterization given above, assume we want to learn the target predicate h(+x, +y, −u) and we are given Table 1. Since a is satisfied by all the examples and has an output variable, a is a dependent provider. On the other side a is the only provider of d, thus d is a dependent consumer. In the mode declarations a has to be declared as a(+x, ∗w). All the other mode declarations are identical to the original literal definitions. Table 2 shows how a literal has to be used based on its classification. If a literal is not a provider one should ignore the first column. Theorem 1. Let i be a dependent provider in C ∈ ρ(C). C is not a solution if there is not at least one consumer of i in C too. Proof (by contradiction). Assume that C = C ∪ {i} is a solution and i is a dependent provider and there is not a consumer of i in C . But, since i succeeds for every combination of input argument values, eval(C) ≥ eval(C ) and since C = C ∪ {i}, | C | 1. Assume again the case when b = jm . For b to be a legal subsequence given Ci−1 it is only required that at least one provider for every input argument of jm is already in Ci−1 . All the providers of jm can be added before b because ∀ providers k of jm , max prov(jm ) > max prov(k). Then a ρ -chain (ρ1 . . . ρn−1 ) can be found so that all the required providers of jm are in Ci−1 and then b is a legal subsequence given Ci−1 . Thus Ci ∈ ρn (Ci−1 ), as claimed. Assume the second case when b = ji . . . jm . All providers k (k < ji ) of jl , i ≤ l ≤ m, are required to be in Ci−1 so that b is a legal subsequence given Ci−1 and since max prov(b) > max prov(k), ∀k, a ρ -chain can be found so that Ci−1 contains all the required providers of b and b is a legal subsequence given Ci−1 . Hence Ci ∈ ρn (Ci−1 ) as claimed. This completes the proof of the induction step and thus of the theorem. Using the macro-based ρ the hypothesis space H is reduced to I/O-complete clauses which are legal subsequences of literals, ie. H := {C ⊆ ⊥ | C is I/Ocomplete and a legal subsequence}. 4.3

Algorithms to Construct the Macros

The procedure to obtain the ordered set D of all macros from a given most specific clause and mode declarations can be seen in Figure 1. The algorithm we use to compute the set A of all the macros starting with a dependent provider i is shown in Figure 2. In the second algorithm, × means a special case of Cartesian product where the resulting set’s elements are numerically ordered; A[j] represents the element j of set A; and, disaggregate(A[j]) is a function that separates the compound element A[j] into its component parts. Example 1. In this example we illustrate how the algorithm in Fig. 2 works. Suppose ⊥ = h(+x) ← p(+x, ∗y, ∗z), t(+x, −u), o(∗w), q(+x, −w), r(+w, +z), s(+u, +y), m(+z), then the algorithm has to compute the set of macros starting with the dependent provider p(+x, ∗y, ∗z). Notice that the macros are always ordered according to their position in ⊥. In this example the ordered set D is D = {t, q, [p, m], [p, t, s], [p, o, r], [p, q, r], [o, r]} 1. i = p(+x, ∗y, ∗z), Zy = {s}, Zz = {r, m} then A = {p} × {s} ∪ {p} × {r, m} = {[p, s], [p, r], [p, m]} for A [1] (2.1) Ys,+u = {t} (2.2.2) if {t} ∩ {p, s} = ∅ then T1 = {} ∪ {[p, s]} × {t} = {[p, t, s]} (2.2.3) T1 = {[p, t, s]} for A [2] (2.1) Yr,+w = {o, q} (2.2.2) if {o, q} ∩ {p, r} = ∅ then T2 = {[p, o, r], [p, q, r]} (2.2.3) T2 = {[p, o, r], [p, q, r]} for A [3], T3 = {[p, m]} 2. A = {[p, t, s], [p, o, r], [p, q, r], [p, m]}


365

– for every literal i ∈ ⊥ = h ← 1, . . . , n do: 1. if i has to be a single-literal macro according with Table 2, add i to D. 2. if i is a dependent provider, construct the set A of all possible macros starting with i (as shown in Fig. 2) and add A to D. – sort D according to Def. 11.

Fig. 1. Algorithm to obtain the macros 1. let Z1 , . . . , Zn be the sets of consumers of i for ∗var1 , . . . , ∗varn ∈ i (ie. Zj = {k ∈ ⊥ | k consumes ∗varj ∈ i}) then: A = {i} × Z1 ∪ {i} × Z2 ∪ . . . ∪ {i} × Zn . 2. for every element A [j] in A do: 2.1 obtain the providers’ set Yk,+var for each +var not provided by i in consumer k of i such that Yk,+var = {g ∈ ⊥ | g is a provider of k for +var and k ∈ A [j] and g > i}. 2.2 let Tj = {A [j]} then for every Yk,+var do: 2.2.1 Tj = ∅. 2.2.2 ∀Tj [l] if Yk,+var ∩ disaggregate(Tj [l]) = ∅ then Tj = {Tj ∪ {Tj [l] × Yk,+var }. 2.2.3 Tj = Tj . 3. A = Tj

S

Fig. 2. Algorithm to obtain the macros starting with dependent provider i

5

Performance Analysis

Let us illustrate the search tree reduction obtained with the macros. Assume that ⊥ = h ← 1, . . . , 16, that the literals 1 and 4 are dependent providers of the literals 13 and 16, and that 2 and 3 are dependent providers of 14 and 15. Let D be D = { 5, 6, 7, 8, 9, 10, 11, 12, [2, 14], [2, 15], [3, 14], [3, 15], [1, 13], [1, 16], [4, 13], [4, 16]}. Suppose that we are looking for a two-literal clause. The search-trees expanded by IDA* using both methods are shown in Figure 3. The main advantage of using the macro-based refinement operator is the reduction of the search space; however, there is a cost for obtaining the macro’s set D. To analyze the performance of the macro-based method, we perform experiments on four datasets. The first dataset contains 180 positive and 17 negative examples of valid chess moves for five pieces5 ; the second consists of 3340 positive and 1498 negative examples of “safe”6 minesweeper moves; the third one is the dataset used in [2] with 256 positive and 512 negative examples of road sections where a traffic problem has occurred; and the last one is the ILP benchmark dataset mutagenesis [12] with 125 positive and 63 negative examples. Mio was run twice with the same parameters on every dataset: once using the literal-based and once the macro-based method. For the first two datasets we created 5 folds and for the last two 10 folds. Both methods were compared 5 6

E + of this dataset are contained in the distribution package of CProgol 4.4. Tiles without a mine.

366


1

1,2 ... 1,13

2

1,16

1

1,13

3

2,3 ... 2,12

2,14

2,15

2

1,16

2,14

3,4 ... 3,12 3,14

3

2,15

3,14

4

3,15

4,5 ... 4,13

4

3,15

4,13 4,16

4,16

5

6 ... 11

5,6 ...

5, 12 ..... 11,12

5

5,6 ...

6 ...11

5,12 ...

11,12

Fig. 3. Search trees expanded by IDA* looking for two-literal clauses. The literalbased method considers 74 leaf-nodes (top) and the macro-based only 36 (bottom). The nodes surrounded by a rectangle are the solutions using the average number of nodes expanded per search (Avg. N/S) and the average run-time (Avg. RT). For these experiments Mio performs parallel search as described in [9]. The results can be seen in Table 3. Mio obtained exactly the same theory for chess, minesweeper and mutagenesis using both methods. In the traffic problem, the theory obtained with the macro-based method has an extra literal in one clause. The reason for this is that the literal defined as dependent provider in the traffic problem does not succeed for every input argument value (it is satisfied by 75.22% of the examples). However, this does not affect the accuracy of the theory. The macro-based method is in average 8 times faster than the literal-based method. However, the actual speedup on every dataset depends on problem features such as the size of the search space, how long it takes to compute eval(C), and the number of dependent providers. The macros are suitable for any domain where there exists a dependent provider; if a dependent provider is falsely declared, the macro approach obtains clauses with unnecessary literals.

6

Related Work

Mio uses mode declarations to define the hypothesis language and computes ⊥ in the same way as Progol [7] does. However, Mio differs from Progol in the search Table 3. Comparison between the literal-based and the macro-based method Dataset Method

Chess Literal- Macrobased based Avg. N/S 41.56 21.55 Avg. RT 3.33s 2.43s

Minesweeper Literal- Macrobased based 1141.90 69.90 16h33m 51m

Traffic Literal- Macrobased based 3179.47 543.41 3h17m 29m

Mutagenesis Literal- Macrobased based 58905.4 14299.5 7h45m 1h46m


367

strategy, the heuristics, and the type strictness. Additionally the literal-base ρ differs from Progol’s refinement operator in that it does not perform splitting. Macros were introduced by Korf [5] as a domain-independent weak-method for learning. He cataloged the macro-operators as a non-optimal problem solving technique. Junghanns and Schaeffer [3] used macros to reduce the search-tree for Sokoban while preserving optimality. McGovern and Sutton [6] proposed the use of macros as a method to accelerate reinforcement learning. The macro-based method presented in this work preserves optimality and reduces the search-space of a multirelational learning system. In multirelational learning the macros can be considered as a general framework for syntactic bias specification such as MOBAL‘s schemata [4] or relational clichés [11]. Relational clichés add sequences of literals to a clause as a heuristic way to solve the search myopia of a greedy algorithm. Although macros could solve too the shortsightedness of a greedy search, they are not a heuristic but a formal method to reduce the search space that guarantees that the same solutions as the base method are found. Blockeel and De Raedt [1] propose a lookahead technique to improve the evaluation of a clause when a dependent provider is added. In their technique the refinement operator is redefined to incorporate twostep-refinements; the macro-based ρ can be seen as a refinement operator which incorporates n-step-refinements. Contrarily to macros, the lookahead increases the run time and requires the user to provide a template for every providerconsumer match. In contrast to all these methods, in our approach macros are computed automatically and the user only needs to mark the output variables of every dependent provider with the ∗var notation. Quinlan included in Foil determinate literals to overcome the greedy search myopia by adding at once all the determinate literals to a clause [10]. The determinate literals are found automatically by Foil; however, since all the determinate literals are added, a clause refining (pruning) step is later needed to remove the unnecessary literals. Determinate literals must be uniquely satisfied by all the positive examples while a dependent provider must be satisfied by all the examples but can be multiple satisfied.

7


In this paper we propose a downward refinement operator (ρ) which adds macrooperators to a clause. By using this macro-based ρ a reduction in the search space is obtained which results in shorter run-times. We have proved that this refinement operator finds the same solutions as the literal-based one. In addition we present two algorithms to compute automatically a set of macros given a most specific clause and a set of mode declarations. A literal’s classification based on the literal’s consumer/provider properties is proposed to assist the user to determine whether a literal can be marked as a dependent provider. For this work we have used as base case a refinement operator similar to the one used in Progol [7]; however, we believe that macro-operators are also suitable for other refinement operators and as a lookahead technique for greedy

368


systems. In the current implementation the user has to indicate in the mode declarations which literals are dependent providers; however, it should be possible to perform an automatic analysis of the body literals and determine which ones are dependent providers. It is part of the future work to explore these ideas.

Acknowledgments This work was partially supported by a scholarship of the federal state SachsenAnhalt, Germany. We would like to thank Saˇso Dˇzeroski for providing the traffic dataset, and O. Meruvia and S. Hoche for helping to improve this paper.

References 1. Hendrik Blockeel and Luc De Raedt. Lookahead and discretization in ILP. In Saˇso Dˇzeroski and Nada Lavrac, editors, Proc. of the 7th Int. Workshop on ILP, volume 1297 of Lecture Notes in AI, pages 77–84. Springer-Verlag, 1997. 367 2. Saˇso Dˇzeroski, Nico Jacobs, Martin Molina, Carlos Moure, Stephen Muggleton, and Wim Van Laer. Detecting traffic problems with ILP. In D. Page, editor, Proc. of the 8th Int. Conference on ILP, volume 1446 of Lecture Notes in AI, pages 281–290. Springer-Verlag, 1998. 365 3. Andreas Junghanns and Jonathan Schaeffer. Sokoban: A challenging single-agent search problem. In IJCAI Workshop “Using Games as an Experimental Testbed for AI Reasearch”, pages 27–36, 1997. 367 4. J¨ org-Uwe Kietz and Stefan Wrobel. Controlling the complexity of learning in logic through syntactic and task-oriented models. In S. Muggleton, editor, Inductive Logic Programming, pages 335–359. Academic Press, 1992. 367 5. Richard E. Korf. Macro-Operators: A weak method for learning. Artificial Intelligence, 26(1):35–77, 1985. 358, 367 6. Amy McGovern, Richard S. Sutton, and Andrew H. Fagg. Roles of macro-actions in accelerating reinforcement learning. In Proc. of the Grace Hopper Celebration of Women in Computing, 1997. 367 7. Stephen Muggleton. Inverse entailment and Progol. New Generation Computing Journal, 13:245–286, 1995. 358, 359, 366, 367 8. Shan-Hwei Nienhuys-Cheng and Ronald de Wolf. Foundations of Inductive Logic Programming, volume 1228 of Lecture Notes in AI. Springer-Verlag, 1997. 358 9. Lourdes Pe˜ na Castillo and Stefan Wrobel. On the stability of example-driven learning systems: a case study in multirelational learning. In C. A. Coello Coello, A. de Albornoz, E. Sucar, and O. Cairo, editors, Proc. of MICAI’2002, volume 2313 of Lecture Notes in AI. Springer-Verlag, 2002. 358, 359, 366 10. J. Ross Quinlan. Determinate literals in inductive logic programming. In John Mylopoulos and Raymond Reiter, editors, Proc. of the 12th IJCAI, volume 2, pages 746–750. Morgan Kaufmann, 1991. 367 11. Glenn Silverstein and Michael J. Pazzani. Relational clichés: constraining constructive induction during relational learning. In L. Birnbaum and G. Collins, editors, Proc. of the 8th Int. Workshop on Machinge Learning, pages 203–207, 1991. 367 12. Ashwin Srinivasan, Stephen Muggleton, Ross D. King, and Michael J. E. Sternberg. Theories for mutagenicity: a study of first-order and feature based induction. Artificial Intelligence, 85(1–2):277–299, 1996. 365

Propagation of Q-values in Tabular TD(λ) Philippe Preux Laboratoire d’Informatique du Littoral UPRES-EA 2335, Université du Littoral Cˆ ote d’Opale BP 719, 62228 Calais Cedex, France [email protected]

Abstract. In this paper, we propose a new idea for tabular TD(λ) algorithm. In TD learning, rewards are propagated along the sequence of state/action pairs that have been visited recently. In complement to this, we propose to propagate rewards towards neighboring state/action pairs along this sequence, though unvisited. This leads to a great decrease in the number of iterations required for TD(λ) to be able to generalize since it is no longer necessary that a state/action pair is visited for its Q-value to be updated. The use of this propagation process makes tabular TD(λ) coming closer to neural net based TD(λ) with regards to its ability to generalize, while keeping unchanged other properties of tabular TD(λ).

1

Introduction

Time derivative (TD) algorithms [9] are important reinforcement learning methods. Assuming discrete time, at each time step t ∈ N, being in a certain state st ∈ S, a reinforcement learning algorithm learns which action at ∈ A(st ) to T emit in order to optimize the total amount of rewards RT = t=0 γ t rt it will receive, where rt is the reward received at time step t, γ is the discount factor (γ ∈ [0, 1]), and T can be ∞. In the sequel, we assume that conditions to apply dynamic programming techniques are not met. A key point in the design of a TD(λ) algorithm lies in the choice of a structure to store estimates of qualities (or values). One possibility is to use a look-up table in which each state/action pair is associated to one element. The access as well as the update of a quality costs a single array element access. An update only concerns one state/action pair and to obtain an estimate of all state/action pairs, all pairs should be visited once at the very least. However, the size of the table is O(| S |) which may be considerable. The other possibility is to use some sort of approximation architecture which represents the information in a much more compact form. Different architectures have been used [2]. Among them, neural networks are rather popular and well-known for their ability to generalize from their training. They have been used to tackle problems of large size, such as TDGammon which learnt to play Backgammon at a grand master level [13]. In this case, states are encoded in some way to be fed into the network. The output of the network provides the current estimate of the state value. This estimate is a function of the network weights. The number of weights to be learnt is very small T. Elomaa et al. (Eds.): ECML, LNAI 2430, pp. 369–380, 2002. c Springer-Verlag Berlin Heidelberg 2002

370

Philippe Preux

with regards to the number of possible states. In TD-Gammon, | S | is estimated to be 1020 , while the number of neurons is varying around 300-400 depending on the version of the program, resulting in O(104 ) weights to learn. These weights are typically learnt using a backpropagation process. So, the access to a state value costs the computation of the output of the network, while its update costs a backprop; clearly, these computaitonal costs are much larger than in the case of tabular TD(λ). However, when weights are updated for a given input state, the estimated value of all states are updated in the same time. There is thus some sort of implicit parallel update of the estimation of all values. This confers the neural based TD(λ) much greater ability for generalization: it is no longer required that a state is visited to have an estimate of its value. Tabular TD(λ) is appealing when the number of states is small, whereas the ability for generalization of neural networks is very attractive. So, in this paper, we propose a variation of tabular TD to enhance its ability to generalize. This variation can be embedded in Q-learning, Q(λ) [14], Sarsa, Sarsa(λ) [8, 7] and their derivatives, and each of these algorithms can benefit of it. Basically, the idea is to add a propagation process of Q-values (and we call the resulting algorithm, the Propagation-TD, or PTD, as well as PQ(λ) and PSarsa(λ)). This idea relies on the observation that two neighboring states s and s are such that, let a an action that can lead from s to s , the quality of (s, a) (denoted Q(s, a)) is likely to be closely related to the value of s (denoted V (s )): if V (s ) is high, then the quality of state/action pairs that lead to s is likely to be high too, and conversely, unless the return when transiting from s to s via action a is very large. Actually, neural network based TD(λ) faces exactly the same problem. This propagation process transforms tabular TD(λ) into something coming close to neural TD with regards to its generalization ability. Indeed, it is no longer required that a state/action pair is visited for its quality to be estimated. Except with regards to the compactness of the neural net representation of Q-values, we end-up with an algorithm that combines the advantages of both approaches to TD learning. In the sequel of the paper, we detail the propagation process in Sec. 2. Then, we evaluate its interest using an experimental approach in Sec. 3. In this evaluation, we are mainly interested in PTD as an agent behavior control learning algorithm, rather than as a value learning algorithm, although both issues are closely related. We are also interested in combining supervised learning with reinforcement learning in order to show the agent how to achieve its goal and makes learning faster. After that, we discuss and conclude in Sec. 4.

2

The Propagation Algorithm

In this section, we detail the keypoint of PTD, that is the propagation process. Before going into details, we wish to make it clear that this is a generic idea from which different strategies can be drawn for general cases, and some specific strategies can be drawn for specific cases, such as episodic tasks. So, we will not detail all these specificities. Some of them will be discussed in the experimental b discussion, in Sec. 3. We use the notation s → s to mean that the action b

Propagation of Q-values in Tabular TD(λ)

371

emitted in state s has a non null probability to lead to state s in a single transition; s is a predecessor of s. The propagation process acts as follows. Let us consider a TD(λ) process, and let us suppose that it has visited the state/action pairs (st , at )t∈[0,n] since it began. Then, the generic idea of PTD is as follows: – build S0 = {st∈[0,n]}. b

Then, Q(s , b) should be updated for any (s , b) such that s → s ∈ S0 , then – build S1 = S0 ∪ {s , such that Q(s , b) has just been updated }. b Then, Q(s , b) should be updated for any (s , b) such that s → s ∈ S1 , and consequently, – build S2 = S1 ∪ {s , such that Q(s , b) has just been updated }, – iterate by building Sd using Sd−1 until some criterion is fulfilled. Fig. 1 sketches the propagation algorithm embedded in a naive1 Q(λ) using accumulating eligibility traces. The propagation process is handled in the “Propagate” procedure. We could equally have given the propagation algorithm embedded into Sarsa(λ) by simply adding a call to “Propagate” at the end of it. The propagate procedure relies on the assumption that we know whether a transition between one state to an other state via a certain action is possible or not; however, we do not need to know the probability of this transition. Let us now discuss some details. The propagation can be iterated as long as new state/action pairs are reached by way of a saturation process. However, the discount factor involves a fast decrease of the amount of update of qualities. So, after some iterations, this amount can be neglected and the propagation can be stopped: this may avoid a huge amount of updates, thus, and may save a huge amount of computation time. Then, propagation can be stopped after a certain amount of iterations either by limiting d, or by cutting off the propagation as the update becomes smaller than a given threshold. The update of Q-values is discounted by the factor γ at each iteration of the propagation (repeat/loop of the “Procedure Propagate” in Fig. 1). This tends to create a gradient of qualities in the state/action space. Visually, in this space, the standard Q(λ) digs some sort of a very narrow furrow while PTD creates a whole gradient field. The update of Q-values is performed as follows using the update equation of Q-learning: Q(s , b) ← Q(s , b) + α[γ d max Q(s, c) − Q(s , b)] c∈A(s)

There are two differences with Q-learning update rule. First, γ appears with exponent d which is the distance from s to the closest state in the eligibility trace. Second, the current reward term is absent since the emission of b in state s has not been performed actually. This choice might be discussed. If this can 1

The word “naive” is used according to [12]. In the case of Watkins Q(λ), a naive update of eligibility traces involves that these are not reset after exploratory steps.

372

Philippe Preux

seem troublesome, it should be pointed out that this is precisely what happens in neural TD(λ). Indeed, in neural TD, the quality of a state/action pair being stored in the weights of the network, changing the weights after a state/action pair has been visited involves an alteration of the quality of many state/action pairs. However, the consequences associated with the emission of the action are not observed for these pairs. In some cases of application of PTD, a certain term could be used to approximate a systematic consequence of an action, such as an action cost. More generally, one might use a model of expected rewards. In episodic tasks, a possible variation on this basic algorithm is to propagate only at the end of the episode and not at each step. For episodic tasks where a reward is always null except when the end of the episode is reached, propagating

Procedure PQ(λ) Initialize Q(s, a) arbitrarily, for all s, a Repeat (for each episode): Initialize s, a Initialize e(s, a) = 0 for all s, a Repeat (for each step in the episode): Take action a, observe r, s Choose a from s (e.g., -greedy strategy) a∗ ← arg maxb Q(s , b) δ ← r + γQ(s , a∗) − Q(s, a) e(s, a) ← e(s, a) + 1 For all s, a Q(s, a) ← Q(s, a) + αδe(s, a) e(s, a) ← γλe(s, a) Propagate s ← s ; a ← a Until s is terminal Procedure Propagate d←0 = 0} Sd ← {(s, a), such that e(s, a) Repeat d←d+1 Sd = Sd−1 For all s ∈ Sd−1 For all (s , b) ∈ Sd−1 If s → s ∈ Sd−1 Then update Q(s , b) add (s , b) to Sd Until stopping criterion is fulfilled b

Fig. 1. Generic outline of Propagation-Q(λ). The basis is the tabular Q(λ) as expressed in [12, p. 184], using a naive strategy, and accumulating eligibility traces. We assume that the “For all” construction has a SIMD semantic, that is, all iterations of a “For all” loop are executed at the same time


373

the qualities only at that time does not alter the way the algorithm works and saves a lot of computation time.

3

Experimental Assessment

In this section, we provide an experimental assessment of propagation Q(λ) by comparing its performance with regards to Q(λ) on a test problem. The test problem consists in finding an outlet or a goal state in a 2D labyrinth. This problem can be made more complex by making it more or less random, stationary or not, ... So, this is actually a whole family of problems rather than a single problem that is used. 3.1

Experimental Setup

The labyrinth is defined in a gridworld where each cell can be either empty or filled with a piece of wall. The state space is the set of cells of the labyrinth. Each cell is numbered and the algorithm knows the cell in which it lies. Only states corresponding to empty cells can be occupied by the agent. In each state, the agent has the choice to stay in its current state, move upward, downward, leftward, or rightward if it does not hit a piece of wall; only non wall-hitting moves are allowed in any state. The agent receives a positive reward when it reaches a goal state of the labyrinth, otherwise it does not receive any reward. Clearly, this problem is markovian, and it is also fully deterministic. This is an episodic task in which a reward is given only at the end of the episode. It is not necessary to propagate at each iteration of the inner repeat/until loop of procedure PQ(λ). It is only necessary to propagate it at the end of the episode. We are mainly interested in algorithm to control an agent so that we are mainly interested in algorithms that learn to behave correctly, rather than in algorithms that learn to predict accurately state values or state/action pair qualities. So, to evaluate our approach, we are interested in the number of iterations required to learn to achieve the task, as well as the number of correct decisions that the algorithm is making regarding actions that have been emitted to reach a goal state, as well as the number of correct decisions the algorithm has learnt regarding actions that would have been emitted if the algorithm had followed other trajectories. This latter somehow measures the generalization the algorithm has made from its experience along trajectories it followed to reach goal states. With the word “iteration”, we mean here one iteration of the inner repeat/until loop in procedure PQ(λ) of Fig. 1. We compare the performance of Q(λ) and PQ(λ). Both are run on the two leftmost mazes shown at Fig. 2. The first one (called the “Pacman maze” later on) is composed of 206 states (wall cells are not counted), while the second one is composed of 2310 states. This latter maze is drawn from the partigame paper [4] and thus called the “Partigame maze”. Apart from the difference in their size, the two mazes differ greatly in that in the pacman maze, there are lots of walls and in most states, the number of possible actions is reduced to 2 (or 3 if immobility is possible). In the partigame maze, there are 4

374

Philippe Preux

(resp. 5) possible actions in the large majority of states. Finally, in the pacman maze, goal states are the two outlets while in the partigame maze, the goal state is the one used as such in the partigame paper. For each run of the algorithm, we set it into an initial cell and let it find a goal state. Then, we reset its position and run it again performing 100 reaches of the goal (without resetting Q-values along these 100 runs). To obtain correct statistics, we average the performance over 10 runs. In the pacman maze, initial states are drawn at random at each new run while in the partigame maze, initial and goal states are set to those used in the partigame paper [4]. To avoid certain biases due to the pseudo-random generator and be able to discuss experimental results more thoroughly, the algorithms can be run so that the initial states are the same for the different algorithms. Propagation is stopped whenever the amount being propagated becomes smaller than a certain threshold (10−10 ). γ is set to 0.9, α to 0.5, and λ to 0.9. Q-values are initialized to 0. The selection of action is -greedy, with set to 0.1, that is 10% of exploratory moves (exploratory moves are random moves). 3.2

Results

As expected, the average number of iterations is significantly smaller for PQ(λ). More preciselly, for both Q(λ) and PQ(λ), the naive strategy provides much better results than the non naive version. Consequently, we now use naive versions of the algorithms unless explciitely mentioned. At the least, Q(λ) needs 20% more iterations than PQ(λ) in the pacman maze, 4 times more for the partigame maze: this is clearly an effect of the size of the state/action space. Both algorithms perform approximately the same amount of backups (2 106 for the pacman maze along 100 episodes); PQ(λ) performs much more backups during the first episode than during the next ones. However, it should be said that the

G

G

G

S S

Fig. 2. Three mazes used in this paper. The leftmost maze has two goal states (the outlets) as indicated by G’s leftmost and rightmost cells at mid-heigth). In the rightmost maze, the algorithm has to find its way from an initial cell (S located in the bottom line) to a goal state (G located in the upper right corner). The rightmost maze is used at section 3.4. It has one initial state (S) and two goal states (stars), one goal being better than the other


375

Table 1. This table summarizes some results obtained on the labyrinth problem for the pacman maze and the partigame maze. Figures are averaged over 10 runs, each made of 100 episodes. The second and third lines gives the size of the problems, either as the number of states, or as the number of possible state/action pairs (this number is given considering that immobility is forbidden; when immobility is a valid action, the number of state/action pairs is the sum of the second and third lines). The “Length” column gives the average number of iterations performed to reach the goal (this number takes into account the distance between the initial state and the closest goal state), the “Backups” column gives the average number of backups per run, while the “Greedy actions” column gives the average percentage of states for which the learnt greedy action is correct Pacman maze Partigame maze states 206 2310 state/action pairs 488 8750 Algorithm Length Backups Greedy actions Length Backups Greedy actions Q(λ) 46.7 2.3 106 48% 14.8 6 106 5% 6 PQ(λ) 38.3 2.7 10 77% 3.5 3 106 60%

way we performed the comparison is unfair for PQ(λ). Indeed, for this kind of problems (deterministic), once Q-values have been propagated to all state/action pairs, exploration is no longer necessary. The amount of exploratory moves being fixed by the value of (0.1), PQ(λ) always performs 10% exploratory moves that are almost always useless: once the gradient field has been created, a mere greedy selection of action is optimal. During a single execution of PQ(λ), the first episodes require much more iterations that the others. If we do not take the 10 first episodes into account in the measure of performance, the relative performances of the compared algorithms remain unchanged. This shows that the difference of performance does not rely on a transient effect. We have performed an analysis of the role of the parameter (α, γ, and λ). The results are contrasted since Q(λ) is at its best with high values for α and λ (α = 1.0, λ = 0.75), while PQ(λ) performs at its best with small values (α = λ = 0.1); both algorithms perform better with γ = 0.5. 3.3

Capacity of Generalization

It is interesting to try to measure the capacity of generalization of the algorithms. The capacity of generalization can be assessed as follows: having learnt a good trajectory from an initial state to a final state, for what fraction of the state space have correct actions also been learnt? Clearly, after one run of Q-Learning, a one step trajectory has been learnt; for Q(λ), a several step trajectory has been learnt, according to the length of the eligibility trace when the goal state is reached. In PQ(λ), much more correct actions have been learnt. For the two

376

Philippe Preux

Table 2. Proportion of states for which the correct action has been learnt Algorithm

after 1 episode after 1000 episodes Pacman maze Partigame maze Pacman maze Partigame maze Q-Learning 0.4% 0.04% 16.9% 6.0% Watkins’ Q(λ) 0.4% 0.04% 18.7% 7.3% naive Q(λ) 1.8% 0.2% 16.9% 8.0% PQ(λ) 11.1% 7.6% 72.4% 59.0%

mazes that are used here, we obtain the results of table 2 after 1 and 100 episodes. As expected, PQ(λ) obtains the highest measures. Of course, the first episode of PQ(λ) requires larger run times than for the other algorithms (approximately 10 times with our non optimized version). However, the next runs of PQ(λ) are very efficient and very fast: as far as a gradient is already available, the algorithm has just to follow it greedily to reach the goal. It is also interesting to discuss the proportion of correct behaviors that are learnt after a certain amount of episodes, or after having used a certain amount of CPU time. After 100 episodes on the partigame maze, PQ(λ) has learnt 59% correct greedy actions (that is, in 59% of the state space, PQ(λ) greedy selection selects the correct action to perform – indeed running 100 or 1000 episodes does not increase significantly this figure); after 1 500 episodes, naive Q(λ) has only learnt 9%. Using the same CPU duration (1 minute on a Pentium III, 500 MHz running Linux), PQ(λ) performs 100 episodes, while Q(λ) performs 104 episodes. In this case, Q(λ) has only learnt 15% correct actions in the whole state space, that is one quarter of what PQ(λ) does in the same amount of time. Regarding the number of backups, PQ(λ) perfoms approximately 3 106 backups, while Q(λ) performs 5 107 within this amount of time. When plotted against the number of episodes, this proportion of correctly learnt greedy actions levels; to get closer to 100%, one has to perform more episodes so that certain yet unexplored regions of the state space get explored. From what has been reported, it is clear that, as expected, PQ(λ) is able to generalize much better than classical Q(λ). 3.4

Dealing with Local Optima

For the moment, reaching either one of the two outlets of the pacman maze provides the same positive return. We now consider a problem closely related in which one outlet is sub-optimal: reaching one of the two outlets (say, the leftmost) provides a return equal to +5.0, while reaching the other outlet provides a return of +10.0. We expect that PQ(λ) (as well as regular Q(λ)) will be able to learn to reach both outlets, though favoring the outlet associated with the largest return. Results are displayed in table 3 for two mazes, the pacman maze where one outlet is made suboptimal, and a misleading maze drawn from [3]. In the pacman maze, the two goals are equally easy to find, while in the misleading


377

Table 3. This table displays the proportion of executions that reach either the sub-optimal or the optimal goal in the pacman maze and in the misleading maze where the two goals are distinguished with regards to the return they provide when reached. Results are obtained over 103 runs of each algorithm PQ(λ) Q-learning pacman maze 91% 84% misleading maze 24.5% 3%

maze, the sub-optimal goal is much easier to find than the real optimum. PQ(λ) and Q(λ) reach the best outlet much more often than the sub-optimal one. 3.5

Combining Reinforcement Learning with Training

One can hope to greatly improve the performance of Q(λ) by showing it a trajectory (also called “training” technique), for instance, by way of a graphical interface. However, this “obvious” improvement is not so successful or, for the least, it is less successful that one would expect. Indeed, after having been shown a trajectory and subsequently having been reset to its initial position, Q(λ) begins by following the demonstrated trajectory but, after some steps and depending on the value of , it escapes from this trajectory because it has performed an exploratory move. Once Q(λ) has left the demonstrated trajectory, it is completely lost and is generally unable to return to it. Then, Q(λ) has to reach the target by its own means, the demonstrated trajectory being then totally unused. PTD solves this problem. When being shown the trajectory, PTD creates a gradient towards the taught trajectory. Then, when behaving autonomously, this trajectory is followed and exploratory moves simply lead out of the trajectory to which it is attracted back by the gradient field when performing a greedy move. Furthermore, exploratory moves naturally lead to the optimization of the taught trajectory. Indeed, when shown via the graphical interface, the trajectory is generally not perfect, that is, it is seldomly the best possible trajectory. Starting from an already good approximation of the best trajectory, PTD optimizes it little by little. To illustrate this point, on the pacman maze, during the first episode, we train the algorithm: instead of selecting itself the action to emit, the algorithm follows a training trajectory. This trajectory is voluntarily sub-optimal, being 1.94 longer than the shortest trajectory from the initial state to the closest goal (the training trajectory is the dashed line on the Pacman maze in Fig. 2). During the next episodes, Q(λ) trajectories are generally getting a little bit longer, while those of PQ(λ) are getting shorter. Averaged over 100 runs, after 1000 episodes, the length of the trajectories followed by PQ(λ) has shrunk down to an average of 1.27 times longer than the shortest one, while the average length of those followed by Q(λ) is 1.83 times longer. The shortest trajectory followed by PQ(λ) is 1.11 times longer than the shortest, the 10% extra-length being explained by

378

Philippe Preux

= 0.1 leading to 10% exploratory moves; the longest trajectory is 1.74. In the case of Q(λ), the shortest trajectory is 1.42 and the longest is 4.0. An other worthy point regards whether if the training trajectory leads towards a sub-optimal goal, the algorithm is still able to reach the optimal goal. To check this, in the pacman maze, we make the rightmost outlet sub-optimal, while the leftmost outlet is the best rewarding goal and we train the algorithm with the same trajectory as before, leading to the sub-optimal goal. Performing 105 episodes after having been trained during the first one, PQ(λ) finds the best optimum after 103 epsodes for the first time. On average, the best optimum is found 66% along these 105 episodes. More generally, we think that the use of PTD (instead of tabular TD), in combination with training, can be applied in many cases and can bring important speed-ups despite its initial overcost which is largely compensated by its ability to generalize from its own experience and from training.

4

Discussion and Conclusion

In this paper, we have proposed the propagation TD algorithm. Based on tabular TD methods, PTD tends to bridge the gap between tabular TD and neural TD with regards to its generalization capabilities. While tabular TD updates only Q-values of state/action pairs that are visited, PTD propagates the updates to state/action pairs that lead to states that have been visited. Propagation is grounded on the idea that if a state s has a high value, then state/action pairs that lead to s are likely to have a high quality. This idea is general and can be applied to all tabular TD algorithms. There are a number of nice features of PTD. First, PTD does not involve any extra parameter. Second, as a natural extension of tabular TD(λ) algorithms, propagation can be used instead of these algorithms in many places and applications. For example, it can take advantage of techniques to speed-up Q(λ), or be used in hierarchical Q(λ) to solve POMDPs [15], ... Third, though not studied here, existant convergence proofs should be able to be adapted from other algorithms to PTD2 . Fourth, PTD is worthy when combining reinforcement learning with training (or supervised learning) by avoiding tabular TD to be unable to come back to the taught trajectory after an exploratory move. In some sense, the rather blind tabular TD method becomes far sighted. Fifth, the experimental assessment has shown that, even though it has been implemented very crudely and we have not spent any time to optimize neither the propagation of Q-values, nor the number of backups, the run time of PTD is very reasonable and the trade-off between the run time and the number of learning iterations is not bad for PTD. These last three points make us think that PTD is worthy for applications based on Q-Learning that require generalization abilities. The fact that PTD only needs to know which transitions are possible is also a nice point with regards to real time dynamic programming [1] for which a complete model is necessary. 2

As a matter of fact, the idea of PTD has been proposed very recently and independently by other authors, accompagnied with such a convergence proof [16]


379

With regards to existing work, PTD shares some similarities with Dyna, Priˆ oritized sweeping, and queue-Dyna. All three algorithms build a model (Tˆ, R) ˆ where T (s, a, s ) is the estimated probability that taking action a in state s leads ˆ a) is the estimated return of taking action a in state s. Dyna to state s , and R(s, was introduced by [10, 11]. In complement to regular Q-learning, Dyna mainˆ at each iteration and uses it to update its tains and updates the model (Tˆ, R) estimates of the quality of k other state/action pairs drawn at random among those that have been visited. Prioritized sweeping and queue-Dyna are two similar techniques that have been proposed independently, respectively by [3] and [5]. They are both derived from Dyna from which they differ in that state values are estimated instead of state/action pair qualities, and updated estimates are not drawn at random. Each state is characterized by its predecessors, as well as a priority. Value estimates are updated according to the priority of states: the value of the states having the highest priority is updated. For each updated value V (s), the priority of s is reset to 0, while the priority of the predecessors s of s is updated proportionaly to the change in V (s) and Tˆ (s, a, s ). Thus, PTD is yet an other strategy. First, PTD does not make use of such a thorough model: it solely relies on whether a transition between two states is possible or not. Second, the updated state/action pairs are all those that have been visited since the beginning of the episode as PTD uses eligibility traces, as well as neighboring state/action pairs of updated state/action pairs. In the near future, we wish to optimize the propagation process. More fundamentally, we wish to compare more precisely the ability to generalize of PTD with regards to neural TD. We also wish to evaluate the generalization abilities of PTD with regards to the size of the state/action space, and the performance of PTD in a non deterministic environment. We are also currently evaluating the usefulness of using PTD instead of Q-learning to control the animat MAABAC [6]: this is a multi-segmented artefact in which multiple reinforcement agents learn to collectively solve a task. In this application, the environment is no longer markovian, nor stationary.

Acknowledgements The author would like to thank anonymous reviewers for their constructive remarks, as well as pointing towards the recently published and very closely related paper [16].

References [1] A. G. Barto, S. J. Bradtke, and S. P. Singh. Learning to act using real-time dynamic programming. Artificial Intelligence, 72:81–138, 1995. 378 [2] D. P. Bertsekas and J. N. Tsitsiklis. Neuro-dynamic programming. Athena Scientific, 1996. 369 [3] A. W. Moore and C. G. Atkeson. Prioritized sweeping: reinforcement learning with less data and less real time. Machine Learning, 13:103–130, 1993. 376, 379

380

Philippe Preux

[4] A. W. Moore and C. G. Atkeson. The parti-game algorithm for variable resolution reinforcement learning in multidimensional state-spaces. Machine Learning, 21, 1995. 373, 374 [5] J. Peng and R. J. Williams. Efficient learning and planning within the dyna framework. Adaptive Behavior, 1(4):437–454, 1993. 379 [6] Ph. Preux, Ch. Cassagnabère, S. Delepoulle, and J-Cl. Darcheville. A non supervised multi-reinforcement agents architecture to model the development of behavior of living organisms. In Proc. European Workshop on Reinforcement Learning, October 2001. 379 [7] G. A. Rummery. Problem Solving with Reinforcement Learning. PhD thesis, Cambridge University, 1995. 370 [8] G. A. Rummery and M. Niranjan. On-line Q-learning using connectionist systems. Technical Report TR 166, Cambridge University, Enginerring Department, September 1994. 370 [9] R. S. Sutton. Learning to predict by the method of temporal difference. Machine Learning, 3:9–44, 1988. 369 [10] R. S. Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Proc. Seventh Int’l Conf. on Machine Learning, pages 216–224. Morgan Kaufmann, 1990. 379 [11] R. S. Sutton. Planning by incremental dynamic programming. In Proc. Eighth Int’l Conf. on Machine Learning, pages 353–357. Morgan Kaufmann, 1991. 379 [12] R. S. Sutton and A. G. Barto. Reinforcement learning: an introduction. MIT Press, 1998. 371, 372 [13] G. Tesauro. Temporal difference learning and TD-Gammon. Communications of the ACM, 38:58–68, 1995. 369 [14] C. J. C. H. Watkins. Learning from delayed rewards. PhD thesis, King’s college, Cambridge, UK, 1989. 370 [15] M. Wiering and J. Schmidhuber. HQ-Learning. Adaptive Behavior, 6(2):219–246, 1997. 378 [16] W. Zhu and S. Levinson. PQ-learning: an efficient robot learning method for intelligent behavior acquisition. In Proc. 7th Int’l Conf. on Intelligent Autonomous Systems, March 2002. 378, 379

Transductive Confidence Machines for Pattern Recognition Kostas Proedrou, Ilia Nouretdinov, Volodya Vovk, and Alex Gammerman Department of Computer Science, Royal Holloway, University of London Egham, Surrey TW20 0EX, England {konstant,ilia,vovk,alex}@cs.rhul.ac.uk

Abstract. We propose a new algorithm for pattern recognition that outputs some measures of “reliability” for every prediction made, in contrast to the current algorithms that output “bare” predictions only. Our method uses a rule similar to that of nearest neighbours to infer predictions; thus its predictive performance is close to that of nearest neighbours, while the measures of confidence it outputs provide practically useful information for individual predictions.

1

Introduction

Current machine learning algorithms usually lack measures that can give an indication of how “good” the predictions are. Even when such measures are present they have certain disadvantages, such as: – They cannot be applied to individual test examples. – They often are not very useful in practice (PAC theory). – They often rely on strong underlying assumptions (Bayesian methods). In our case none of these disadvantages are present. Our only assumption is that data items are produced independently by the same stochastic mechanism (iid assumption), our measures of confidence are applicable to individual examples, while experimental results show that they produce good results for benchmark data sets (and so potentially are useful in practice). The iid assumption that we make is a very natural one for most applications of pattern recognition, as it only implies that – all our examples are produced by the same underlying probability distribution and – they are produced independently of each other; so the order in which they appear is not relevant. Many algorithms have been proposed in the past, both in the Bayesian and in the PAC settings, that provide additional information of the “quality” of the predictions. Bayesian algorithms usually provide useful confidence values but when the underlying distribution is not known these values are “misleading”. Experiments T. Elomaa et al. (Eds.): ECML, LNAI 2430, pp. 381–390, 2002. c Springer-Verlag Berlin Heidelberg 2002

382

Kostas Proedrou et al.

in (Melluish et al., 2001) have shown that in Bayesian algorithms, when the underlying probability distribution of the examples is not known, the deviation from the expected percentage of misclassified examples is too large to give any practical meaning to the confidence values. For example, we expect that from all examples with a confidence value of 90% the percentage of those wrongly classified will be close to 10%. Bayesian algorithms instead, can produce a much higher percentage of error at the above confidence level; in experiments in (Melluish et al., 2001) this error is between 20% and 40%. PAC theory doesn’t make any assumptions about the underlying probability distribution, but its results are often not useful in practice. To demonstrate crudeness of the usual PAC bounds, we reproduce an example from (Nouretdinov, Vovk et al., 2001). Littlestone and Warmuth’s theorem stated in (Cristianini et al., 2000) is one of the tightest results of PAC theory, but still usually does not give practically meaningful results. The theorem states that for a two-class Support Vector classifier f the probability of mistakes is l el 1 d ln + ln err(f ) ≤ l−d d δ with probability 1−δ, where l is the training size and d is the number of Support Vectors. For the US Postal Service (USPS) database (described below and in Vapnik, 1998, Section 12.2), the error bound given by that theorem for one out of ten classifiers is close to 7291e 1 274 ln ≈ 0.17, 7291 − 274 274 even if we ignore the term ln δl (274 is the average number of support vectors for polynomials of degree 3, which give the best predictive performance; see Table 12.2 in Vapnik, 1998). Since there are ten classifiers, the upper bound on the total probability of mistakes becomes 1.7, which is not helpful at all. Our prediction method is based on the so called algorithmic theory of randomness. A description of this theory is the subject of Section 2. Then, in Section 3, we describe our algorithm, and in the next section we give some experimental and comparison results for our algorithm as applied to the USPS and other data sets. Our algorithm follows the transductive approach, as for the classification of every new example it uses the whole training set to infer a rule for that particular example only. In contrast, in the inductive approach a general rule is derived from the training set and then applied to each training example. For this reason we shall call our algorithm transductive confidence machine for nearest neighbours (TCM-NN). It is also possible to use the inductive approach to obtain confidence measures for our predictions (see e.g. (Papadopoulos et al., 2002) for an example of how to obtain confident predictions in the case of regression using the inductive approach).

Transductive Confidence Machines for Pattern Recognition

2

383

Algorithmic Theory of Randomness

According to classical probability theory if we toss a fair coin n times, all sequences {0, 1}n will have the same probability 21n of occuring. We would be much more surprised, however, to see a sequence like 111111111 . . . 1 than a sequence like 011010100 . . . 1. The classical approach to probabilty theory can only give probabilites of different outcomes, but cannot say anything about the typicalness of sequences. Intuitively, sequences that don’t seem to have any specific pattern in their elements would be more typical than sequences in which one can easily find regularities. An important result of the theory of algorithic randomness is that there exists a universal method of finding regularities in data sequences. This result is due to Martin-L¨ of, who was the first to introduce the notion of a randomness test. A slightly modified definition of Martin-Löf’s test1 states that a function t : Z ∗ → [0, 1] is a test for randomness with respect to a class of probability distributions Q in Z if – for all n ∈ N, for all s ∈ [0, 1] and for all probability distributions P in Q, P n {z ∈ Z n : t(z) ≤ s} ≤ s,

(1)

– t is semi-computable from above. Here Z is a space that possesses some computability properties; in our application, Z is the set of all possible examples. Every randomness test creates a series of nested subsets. Each subset is associated with a number s that bounds the value t(z) that the test takes. We can expect that every randomness test will detect only some of the non-random patterns occuring in each sequence. Martin-Löf proved, however, that we can merge all such tests to obtain a universal test for randomness2 . Such a test would be able to find all non-random patterns in a sequence of elements. Unfortunately, universal tests for randomness are not computable. Thus we have to approximate them using valid (in the sense of satisfying (1)) non-universal tests. In the next section, we will give a valid randomness test for finite sequences of real numbers produced under the iid assumption and we shall use what we call a strangeness measure to map each example into a single real value in order to utilize that test for obtaining confident predictions using the nearest neighbours algorithm. 1

2

The definition stated here is equivalent with Martin-L¨ of’s original definition; the only difference being the use of the ‘direct scale’ (randomness values from 0 to 1), instead of the ‘logarithmic scale’ (randomness values from 0 to +∞). A proof of the existence of universal randomness tests can be found in (Li & Vit´ anyi, 1997), Chapter 2.4.

384

3 3.1


Nearest Neighbours and Randomness Formal Setting of the Problem

We have a training set {(x1 , y1 ), . . . , (xm , ym )}, of m elements, where xi = (x1i , . . . , xni ) is the set of feature values for example i and yi is the classification for example i, taking values from a finite set of possible classifications, which we identify as {1, 2, . . . , c}. We also have a test set of r examples similar to the ones in the training set, only this time the actual classifications are withheld from us. Our goal is to assign to every test example one of the possible classifications. For every classification we also want to give some confidence measures, valid in the sense of (1), that will enable us to gain more insight in the predictions that we make. 3.2

Nearest Neighbours Transductive Confidence Machine

Let us denote the sorted sequence (in ascending order) of the distances of exy ample i from the other examples with the same classification y as Diy . Also, Dij −y will stand for the jth shortest distance in this sequence and Di for the sorted sequence of distances containing examples with classification different from y. We assign to every example a measure called the individual strangeness measure. This measure defines the strangeness of the example in relation to the rest of the examples. In our case the strangeness measure for an example i with label y is defined as k y j=1 Dij , (2) αi = k −y j=1 Dij where k is the number of neighbours used. Thus, our measure for strangeness is the ratio of the sum of the k nearest distances from the same class to the sum of the k nearest distances from all other classes. This is a natural measure to use, as the strangeness of an example increases when the distance from the examples of the same class becomes bigger or when the distance from the other classes becomes smaller. Now let us return to algorithmic randomness theory. In (Melluish et al., 2001) it is proved that the function p(αnew ) =

#{i : αi ≥ αnew } , m+1

(3)

where αnew is the strangeness value for the test example (assuming there is only one test example, or that the test examples are processed one at a time), is a valid randomness test in the iid case. The proof takes advantage of the fact that since our distribution is iid all permutations of a sequence have the same probability of occuring. If we have a sequence α1 , . . . , αm and a new element αnew is introduced then αnew can take any place in the new (sorted) sequence with the same probability, as all permutations of the new sequence are equiprobable.


385

Thus, the probability that αnew is among the j largest occurs with probability j of at most m+1 . The values taken by the above randomness test will be called p-values. The p-value for the sequence {α1 , . . . , αm , αnew }, where {α1 , . . . , αm } are the strangeness measures for the training examples and αnew is the strangeness measure of a new test example with a possible classification assigned to it, is the value p(αnew ). We can now give our algorithm.

TCM-NN Algorithm Choose k, the number of nearest neighbours to be used for i = 1 to m do Find and store Diy and Di−y end for Calculate alpha values for all training examples for i = 1 to r do Calculate the dist vector as the distances of the new example from all training examples for j = 1 to c do for every training example t classified as j do j > dist(t) recalculate the alpha value of example t if Dtk end for for every training example t classified as non-j do −j > dist(t) recalculate the alpha value of example t if Dtk end for Calculate alpha value for the new example classified as j Calculate p-value for the new example classified as j end for Predict the class with the largest p-value Output as confidence one minus the 2nd largest p-value Output as credibility the largest p-value end for

For each possible classification of a test example we construct the sequence of strangeness values of the training set augmented by the strangeness value of the new test example3 . The prediction for each example is the classification that gives the most typical completion of the sequence of strangeness measures of the training set under the iid assumption. Each prediction is accompanied by two other measures. The most important of them is the confidence measure. Since, by equation (1), the second largest 3

Note that some of the strangeness values of the training set may be different for different test examples or different possible classifications assigned to a test example. In this sense our algorithm is transductive, as the training set is being reused for each test example.

386


p-value is an upper bound on the probability that the excluded classifications will be correct, the confidence measure indicates how likely the predicted classification is the correct one. The credibility measure gives the typicalness of the predicted classification. This value indicates how well suited the training set is for the classification of a particular test example. Low credibility would mean that the test example is strange with respect to the training examples, e.g. trying to classify a letter using a training set that consists of digits. In principle, we would want for each prediction all p-values to be close to 0, apart from the one that gives the correct classification, that we would want to be close to 1.

4


The standard comparison criterion in classification problems is the percentage of incorrectly classified examples. Here we shall also use a second one. We fix a specific significance level δ, say 1%, and we accept as possible classifications the ones whose p-value is above that level. In this way we can determine how many test examples can be classified with a confidence of at least 1 − δ. We have tested our algorithm on the following datasets: – USPS. It consists of handwritten digits from 0 − 9. The training set consists of 7291 examples and the test set of 2007 examples. Each example has 256 attributes (pixels) that describe the given digit. All data were pre-processed as follows. As any image from the USPS data set was represented as 256 numbers (x1 , ..., x256 ), we replaced it by (y1 , ..., y256 ), where 256 xi xi − S , S = i=1 , yi = D 256 256 2 i=1 (xi − S) D= 256 The aim of this preprocessing is to normalise the level of brightness. After the preprocessing, the mean value of each image becomes 0 and the standard deviation becomes 1. – Satellite. These are 6435 satellite images(4435 for train and 2000 for test). The classification task is to identify between 6 different soil conditions that are represented in the images. – Shuttle. The classes of this dataset are the appropriate actions that should be taken under certain conditions(described by 9 attributes) in a space shuttle. There are 43500 train examples, 14500 test examples and 7 different classes. – Segment. 2310 outdoor images described by 9 attributes each. The classifications are : brick-face, sky, foliage, cement, window, path, grass. The last three datasets are used in the Statlog project (King et al., 1995). For comparison purposes we followed the same testing procedure. For the satellite


387

Table 1. Comparison of the error rate of TCM-NN with other learning algorithms Algorithm C4.5 CART NB k-nn CASTLE Discrim Neural TCM Satellite 15.1 13.8 30.7 9.4 19.4 17.1 13.9 10.6 Shuttle 0.04 4.55 0.44 3.77 4.83 4.9 0.11 Segment 4 4 26 7.7 11.2 11.6 3.68 Dataset

Table 2. Comparison of the error percentage of TCM-NN with other algorithms on the USPS dataset Learning Nearest TCM-NN Support Vector Five layer Algorithms Neighbours Machine Neural Network % of error 4.29% 4.29% 4.04% 5.1%

and shuttle datasets we used the same training and test set, while for the segment one we used 10 fold cross-validation. In Table 1 we compare the performance of our algorithm4 with 7 others, all taken from the Statlog project, on the satellite, shuttle and segment datasets. The algorithms are two decision tree algorithms, C4.5 and CART, the Naive Bayes classifier (NB), the k-nearest neighbours algorithm, a Bayesian network algorithm (CASTLE), a linear discriminant algorithm (Discrim) and a backpropagation neural network. Two of the values from Table 1 are missing as these results are not mentioned in (King et al., 1995). Table 2 contains experimental and comparison results on the USPS dataset. The error percentage for the Five Layer Neural Network was obtained from (Vapnik, 1998), while for the other three algorithms the results were produced by the authors on the same training and test set. It is clear from both tables that TCM-NN’s performance is almost as good as the performance of the best algorithms for all datasets used. Next, in Figure 1 we compare the error rate of TCM with that of the original nearest neighbours algorithm on the USPS dataset using a different number of neighbours each time. Though the perfromace of both algorithms is decreasing as the number of neighbours is increasing it seems that TCM is more robust as its error rate is increasing much slower. When the second comparison criterion is used, our algorithm makes ‘region’ predictions (outputs a set of classifications) instead of point predictions. For a specified significance level δ the correct classification will be in the predicted set of classifications with a probability of at least 1 − δ, since the set of rejected classifications can occur with probability of at most δ. In Figure 2 we demonstrate this relationship between error classification and confidence level using 50 random instances of the USPS dataset. 4

We normally use one nearest neighbour for testing TCM-NN. When this is not the case, the number of neighbours used will be stated explicitly.

388


9 TCM-NN Error % NN Error % +

8 7 % 6 + + + + +

5 4

+

+ + +

+

+ + +

+ + +

+ + + +

3 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Number of nearest neighbours

Fig. 1. Error percentage of TCN-NN and NN on the USPS dataset using 1-20 nearest neighbours In Table 3 we detail the results of ‘region’ predictions for significance levels of 1% and 5%, giving the percentage of examples for which the predicted set contains one label, more than one label and no labels. For the shuttle and USPS datasets we predict a set containing one classification for 99.17% and 94.77% of the examples respectively with great certainty (confidence of 99% or more). We can also note that as the overall error rate is increasing the number of examples that can be given a single classification is decreasing. Since greater error percentages mean more difficult classification problems it is natural that more examples will be assigned more than one possible classifications. Finally, the last column in Table 3 gives the percentage of examples of the ‘one class’ column that were correctly classified. These percentages are very close and in most cases higher than the corresponding confidence levels; thus indicating the practical usefulness of TCM’s confidence measure5 .

5

Conclusion

The TCM-NN algorithm presented here has the advantage of giving probabilistic measures for each individual prediction that we make. In this way we gain more insight into how likely a correct classification is for an example when given a specific training set. Furthermore, the percentage of errors of TCM-NN seems to be as good as that of other learning algorithms. 5

Note that choosing a smaller significance level doesn’t necessarily guarantee a greater rate of success, as we only consider the examples that are assigned one classification. The former holds only when we consider all test examples (see Figure 2).

369

Transductive Confidence Machines for Pattern Recognition

90 80 70 60 -

Correct Predictions 50 -

-

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 Confidence Level

Fig. 2. Percentage of correct 'region1 predictions for different confidence levels using 50 random instances of the TJSPS dataset

The scheme we have proposed can be used on top of every classification algorithm and not only nearest neighbours, by defining the individual strangeness measure (2) in a different way. For example, the method can be applied to the Support Vector Machine algorithm using as a strangeness measure the distance of each example from the hyper-plane that separates the different classes. Finally, as an approximation to the universal test defined in Section 2 we have used the

Table 3. TCM-NN Performance. The column ''One class" gives the number of examples for which a confident prediction is made, the column ''22 classes" gives the number of examples for which two or more possible classifications were not excluded at the given significance level, and the column ''No class" gives the number of examples for which all possible classifications were excluded at the given significance level. The last column shows the percentage of correct predictions for the examples we could confidently predict at each significance level

390


statistical p-test (3). It remains an open problem though whether one can find valid tests for randomness (under the general iid assumption) that are better approximations to the universal tests for randomness than the one used here.

Acknowledgements This work was partially supported by EPSRC through grants GR/L35812 (“Support Vector and Bayesian learning algorithms”), GR/M14937 (“Predictive complexity: recursion-theoretic variants”), and GR/M16856 (“Comparison of Support Vector Machine and Minimum Message Length methods for induction and prediction”). We are grateful to the Program Committee for useful comments.

References 1. Cristianini, N., & Shawe-Taylor, J. (2000). An Introduction to Support Vector Machines and Other Kernel-based Methods. Cambridge: Cambridge University Press. 2. Fraser, D. A. (1976). Non-parametric methods in statistics. New York: Wiley. 3. King, R. D., Feng, C., & Sutherland, A. (1995). Statlog: Comparison of classification algorithms on large real-world problems. Applied Artificial Intelligence, 9(3), pp 259–287. 4. Li, M., & Vit´ anyi, P. (1997). An introduction to Kolmogorov complexity and its applications (2nd edn.). New York: Springer. 5. Melluish, T., Saunders, C., Nouretdinov, I., & Vovk, V. (2001). Comparing the Bayes and typicalness frameworks. In Proceedings of ECML’2001. 6. Nouretdinov, I., Melluish, T., Vovk V. (2001). Ridge Regression Confidence Machine. In Proceedings of the 18th International Conference on Machine Learning. 7. Nouretdinov, I., Vovk, V., Vyugin, M., & Gammerman, A. (2001). Pattern recognition and density estimation under the general iid assumption. In Proceedings of the Fourteenth Annual Conference on Computational Learning Theory and Fifth European Conference on Computational Learning Theory. 8. Papadopoulos, H., Proedrou, K., Vovk, V., Gammerman, A. (2002). Inductive Confidence Machines for Regression.. In Proceedings of ECML’2002. 9. Vapnik, V. (1998). Statistical Learning Theory. New York: Wiley. 10. Vovk, V., & Gammerman, A. (2002). Algorithmic Theory of Randomness and its Computer Applications. Manuscript.

Characterizing Markov Decision Processes Bohdana Ratitch and Doina Precup McGill University, Montreal, Canada {bohdana,dprecup}@cs.mcgill.ca http://www.cs.mcgill.ca/{~sonce,~dprecup} Abstract. Problem characteristics often have a significant influence on the difficulty of solving optimization problems. In this paper, we propose attributes for characterizing Markov Decision Processes (MDPs), and discuss how they affect the performance of reinforcement learning algorithms that use function approximation. The attributes measure mainly the amount of randomness in the environment. Their values can be calculated from the MDP model or estimated on-line. We show empirically that two of the proposed attributes have a statistically significant effect on the quality of learning. We discuss how measurements of the proposed MDP attributes can be used to facilitate the design of reinforcement learning systems.

1

Introduction

Reinforcement learning (RL) [17] is a general approach for learning from interaction with a stochastic, unknown environment. RL has proven quite successful in handling large, realistic domains, by using function approximation techniques. However, the properties of RL algorithms using function approximation (FA) are still not fully understood. While convergence theorems exist for some valuebased RL algorithms using state aggregation or linear function approximation (e.g., [1, 16]), examples of divergence of some RL methods combined with certain function approximation architectures also exist [1]. It is not known in general which combinations of RL and FA methods are guaranteed to produce stable or unstable behavior. Moreover, when unstable behavior occurs, it is not clear if it is a rare event, pertinent mostly to maliciously engineered problems, or if instability is a real impediment to most practical applications. Most efforts for analyzing RL with FA assume that the problem to be solved is a general stochastic Markov Decision Process (MDP), while very little research has been devoted to defining or studying sub-classes of MDPs. This generality of the RL approach makes it very appealing. This is in contrast with prior research in combinatorial optimization (e.g., [13, 6]), which showed that the performance of approximate optimization algorithms can be drastically affected by characteristics of the problem at hand. For instance, the performance of local search algorithms is affected by characteristics of the search space for a given problem instance, such as the number of local optima, the sizes of the regions of attraction, and the diameter of the search space. Recent research (e.g., [7, 10]) has shown that such problem characteristics can be used to predict T. Elomaa et al. (Eds.): ECML, LNAI 2430, pp. 391–404, 2002. c Springer-Verlag Berlin Heidelberg 2002

392

Bohdana Ratitch and Doina Precup

the behavior of local search algorithms, and improve algorithm selection. In this paper, we show that a similar effect is present in algorithms for learning to control MDPs: the performance of RL algorithms with function approximation is heavily influenced by characteristics of the MDP. Our focus is to identify relevant characteristics of MDPs, propose ways of measuring them, and determine their influence on the quality of the solution found by value-based RL algorithms. Prior theoretical [14] and empirical results [3, 9] suggest that the amount of stochasticity in an MDP can influence the complexity of finding an optimal policy. We propose quantitative attributes for measuring the amount of randomness in an MDP (e.g., entropy of state transitions, variance of immediate rewards and controllability), and for characterizing the structure of the MDP. These attributes can be computed exactly for MDPs with small discrete state space, and they can be approximated for MDPs with large or continuous state spaces, using samples from the environment. Our research builds on the work of Kirman [9], who studied the influence of stochasticity on dynamic programming algorithms. In this paper, we redefine some of his attributes and propose new ones. We treat both discrete and continuous state space, while Kirman focused on discrete problems. We also focus on on-line, incremental RL algorithms with FA, rather than off-line dynamic programming. We discuss the potential for using MDP attributes for choosing RL-FA algorithms suited for the task at hand, and for automatically setting user-tunable parameters of the algorithms (e.g., exploration rate, learning rate, and eligibility). At present, both the choice of an algorithm and the parameter setting are usually done by a very time-consuming trial-and-error process. We believe that measuring MDP attributes can help automate this process. We present an empirical study focused on the effect of two attributes, state transition entropy and controllability, on the quality of the behavior learned using RL with FA. The results show that these MDP characteristics have a statistically significant effect on the quality of the learned policies. The experiments were performed on randomly generated MDPs with continuous state spaces, as well as randomized versions of the Mountain Car task [17]. The paper is organized as follows. In Sect.2, we introduce basic MDP notation and relevant reinforcement learning issues. In Sect.3, we introduce several domain-independent attributes by which we propose to characterize MDPs, and give intuitions regarding their potential influence on learning algorithms. Sect.4 contains the details of our empirical study. In Sect.5 we summarize the contributions of this paper and discuss future work.

2

Markov Decision Processes

Markov Decision Processes (MDPs) are a standard, general formalism for modeling stochastic, sequential decision problems [15]. At every discrete time step t, the environment is in some state st ∈ S, where the state space S may be finite or infinite. The agent perceives st and performs an action at from a discrete, finite action set A. One time step later, the agent receives a real-valued numerical

Characterizing Markov Decision Processes

393

reward rt+1 and the environment transitions to a new state, st+1 . In general, both the rewards and the state transitions are stochastic. The Markov property means that the next state st+1 and the immediate reward rt+1 depend only on the current state and action, st , at . The model of the MDP consists of the trana a sition probabilities Ps,s and the expected values of the immediate rewards Rs,s , ∀s, a, s . The goal of the agent is to find a policy, a way of behaving, that maximizes the cumulative reward over time. A policy is a mapping π : S × A → [0, 1], where π(s, a) denotes the probability that the agent takes action a when the environment is in state s. The long-term reward received by the agent is called the return, and is defined as an additive function ∞ of the reward sequence. For instance, the discounted return is defined as t=0 γ t rt+1 , where γ ∈ [0, 1). Many RL algorithms estimate value functions, which are defined with respect to policies and reflect the expected value of the return. The actionvalue function of policy π represents the expected discounted return obtained whenstarting from state s, taking a, and henceforth following π: Qπ (s, a) = ∞ Eπ { k=0 γ k rt+k+1 |st = s, at = a} , ∀s ∈ S, a ∈ A. The optimal action-value function is Q∗ (s, a) = maxπ Qπ (s, a), ∀s ∈ S, a ∈ A. An optimal policy is one for which this maximum is attained. If the optimal action-value function is learned, then an optimal policy can be implicitly derived as a greedy one with respect to that value function. A policy is called greedy with respect to some actionvalue function Q(s, a) if in each state it selects one of the actions that have the maximum value: π(s, a) > 0 iff a ∈ arg maxa ∈A Q(s, a ). Most RL algorithms iteratively improve estimates of value functions based on samples of transitions obtained on-line. For example, at each time step t, the tabular Sarsa learning algorithm [17] updates the value of the current stateaction pair (st , at ) based on the observed reward rt+1 and the next state-action pair st+1 , at+1 , as: Q( st , at ) ← Q(st , at ) + αt [rt+1 + γQ(st+1 , at+1 ) −Q(st , at )], αt ∈ (0, 1). Input Target

(1)

Unlike supervised learning, RL is a trial-and-error approach. The learning agent has to find which actions are the best without the help of a teacher, by trying them out. This process is called exploration. The quality and the speed of learning with finite data can depend dramatically on the agent’s exploration strategy. In MDPs with large or continuous state spaces, value functions can be represented by function approximators (e.g., CMACs or neural networks [17]). In that case, RL methods sample training data for the approximator, which consist of inputs (e.g., state-action pairs) and targets (e.g., estimates of the action-value function). Equation (1) shows an example of inputs and targets for the SARSA algorithm. The approximator generalizes the value estimates gathered for a subset of the state-action pairs to the entire S × A space. The interplay between the RL algorithm and the function approximator has an iterative, interleaved manner, as shown in Fig. 1. Function approximation in the context of RL is harder than in the classical, supervised learning setting. In supervised learning, many techniques assume a

394

Bohdana Ratitch and Doina Precup a subset ✗state-action values ✔ of S × A✲ (targets)✲✗ ✔ Q(s,✲ a) generalization of the value function ✲ RL ✲ FA to the entire S × A space ✖✕ (inputs) ✖ ✕

Fig. 1. Interaction of RL and FA learning Table 1. Factors contributing to the noise in the target of the FA training data rt1 +1 = rt2 +1 rt1 +1 = rt2 +1 Q(st1 +1 , at1 +1 ) Q(st2 +1 , at2 +1 ) =⇒ Q(st1 +1 , at1 +1 ) Q(st2 +1 , at2 +1 )

Factor 1: Stochastic immediate rewards =⇒ Factor 2: Stochastic transitions: st1 +1 = st2 +1 =⇒ Factor 3: Stochastic transitions: st1 +1 = st2 +1 =⇒

=

Factor 4: Different action choices: at1 +1 = at2 +1

=

static training set. In RL on the other hand, the estimates of the value function (which are the targets for the FA) evolve and improve gradually. Hence, the FA’s target function appears to be non-stationary. Moreover, the stochasticity in the environment and the exploration process may introduce variability into the training data (i.e., variability in the targets for a fixed input, which we will call “noise” from now on). To identify the potential sources of this noise, let us examine (1) again. Suppose that the same state-action pair (ˆ s, a ˆ) is encountered at time steps t1 and t2 during learning, and the FA is presented with the corresponding targets [rt1 +1 + γQ(st1 +1 , at1 +1 )] and [rt2 +1 + γQ(st2 +1 , at2 +1 )]. Table 1 shows four factors that can contribute to the noise. Note that these factors arise from the particular structure of the estimated value functions, from the randomized nature of the RL algorithm and, most of all, from the inherent randomness in the MDP. We will now introduce several attributes that help differentiate these sources of randomness, and quantify their effect.

3

MDP Attributes

We present six domain-independent attributes that can be used to quantitatively describe an MDP. For simplicity, we define them assuming discrete state and action spaces and availability of the MDP model. Later, we discuss how these assumptions can be lifted. State transition entropy (STE) measures the amount of stochasticity due to the environment’s state dynamics. Let Os,a denote a random variable representing the outcome (next state) of the transition from state s when the agent performs action a. This variable takes values in S. We use the standard informationtheoretic definition of entropy to measure the STE for a state-action pair (s, a) (as defined in [9]):


ST E(s, a) = H(Os,a ) = −

a a Ps,s log Ps,s

395

(2)

s ∈S

A high value of ST E(s, a) means that there are many possible next states s a (with Ps,s = 0) which have about the same transition probabilities. In an MDP with high STE values, the agent is more likely to encounter many different states, even by performing the same action a in some state s. In this case, state space exploration happens naturally to some extent, regardless of the exploration strategy used by the agent. Since extensive exploration is essential for RL algorithms, a high STE may be conducive to good performance of an RL agent. At the same time, though, a high STE will increase the variability of the state transitions. This suggests that the noise due to Factors 2 and 3 in Table 1 may increase, which can be detrimental for learning. The controllability (C) of a state s is a normalized measure of the information gain when predicting the next state based on knowledge of the action taken, as opposed to making the prediction before an action is chosen (note that a similar, but not identical, attribute is used by Kirman [9]). Let Os denote a random variable (with values from S) representing the outcome of a uniformly random action in state s. Let As denote a random variable representing the action taken in state s. We consider As to be chosen from a uniform distribution. Now, given the value of As , information gain is the reduction in the entropy of Os : H(Os ) − H(Os |As ), where H(Os ) = −

Ps,s log Ps,s = −

s ∈S

s ∈S

H(Os |As ) = −

(

1 a 1 a Ps,s ) log( Ps,s ) |A| |A| a∈A

a∈A

1 a a Ps,s log(Ps,s ) |A|

a∈A

s ∈S

The controllability in state s is defined as: C(s) =

H(Os ) − H(Os |As ) H(Os )

(3)

If H(Os ) = 0 (deterministic transitions for all actions), then C(s) is defined to be 1. It may also be useful (see Sect. 5) to measure the forward controllability (FC) of a state-action pair, which is the expected controllability of the next state: a F C(s, a) = Ps,s (4) C(s ) s ∈S

High controllability means that the agent can exercise a lot of control over which trajectories (sequences of states) it goes through, by choosing appropriate actions. Having such control enables the agent to reap higher returns in environments where some trajectories are more profitable than others. Similar to the STE, the level of controllability in an MDP also influences the potential exploration of the state space. Because in a highly controllable state s the outcomes of

396


different actions are quite different, the agent can choose what areas to explore. This can be advantageous for the RL algorithm, but may be detrimental for function approximation, because of the noise due to Factor 4 in Table 1. The variance of immediate rewards, V IR(s, a), characterizes the amount of stochasticity in the immediate reward signal. High VIR causes an increase in the noise due to Factor 1 in Table 1, thus making learning potentially more difficult. The risk factor (RF) measures the likelihood of getting a low reward after the agent performs a uniformly random action. This measure is important if the agent has to perform as well as possible during learning. Let rsau denote the reward observed on a transition from a state s after performing a uniformly random action, au . The risk factor in state s is defined as: RF (s) = P r[rsau < E{rsau } − (s)],

(5)

where (s) is a positive number, possibly dependent on the state, which quantifies the tolerance to lower-than-usual rewards. Note that low immediate rewards do not necessarily mean low long-term returns. Nevertheless, knowledge of RF may help minimize losses, especially during the early stages of learning. The final two attributes are meant to capture the structure in the state transitions. The transition distance, T D(s, a), measures the expected distance between state s and its successor states, according to some distance metric on S. We are currently investigating what distance metric would be appropriate. One candidate is a (weighted) Euclidean distance, but this is not adequate for all environments. The transition distance may affect RL when using global function approximators, such as neural networks. In the case of incremental learning with global approximators, training on two consecutive inputs that are very different may create mutual interference of the parameter updates and impede learning. Hence, such MDPs may benefit from using local approximators (e.g. Radial Basis Networks). The transition variability, T V (s, a), measures the average distance between possible next states. With a good T V metric, a high value of T V (s, a) would indicate that the next states can have very different values, and hence introduce noise due to Factor 3 in Table 1. In continuous state spaces, the attributes are defined by using integrals instead of sums. If the model of the process is not available, these attributes can be estimated from the observed transitions, both for discrete and continuous state spaces. The attributes can be measured locally (for each state or state-action pair) or globally, as an average over the entire MDP. Local measures are most useful for tuning the parameters of the RL algorithm. For example, in Sect.5, we suggest how they can be used to adapt the exploration strategy. In the experiments presented in the next section, we use global measures - sample averages of the attribute values, computed under the assumption that all states and actions are equally probable. This choice of the sampling distribution is motivated by our intention to characterize the MDP before any learning takes place. Note, however, that under some circumstances, weighted averages might be of more interest (e.g., if we want to compute these attributes based on behavior generated by a restricted class of policies). If the sample averages were estimated


397

on-line, during learning, they would naturally reflecting the state distribution that the agent is actually encountering.

4

Empirical Study

In this section we focus on studying the effect of two of state-transition entropy (STE) and controllability (C), on the quality of the policies learned by an RL algorithm using linear function approximation. The work of Kirman suggests that these attributes influence the performance of off-line dynamic programming (DP) algorithms, to which RL approaches are related. Hence, it seems natural to start with studying these two attributes. Our experiments with the other attributes are still in their preliminary stages, and the results will be reported in future work. In Sect. 4.1 we present the application domains used in the experiments. The experimental details and the results are described in Sect.4.2. 4.1

Tasks

In order to study empirically the effect of MDP characteristics on learning, it is desirable to consider a wide range of STE and C values, and to vary these two attributes independently. Unfortunately, the main collection of currentlypopular RL tasks1 contains only a handful of domains, and the continuous tasks are mostly deterministic. So our experiments were performed on artificial random MDPs, as well as randomized versions of the well-known Mountain-Car task. Random discrete MDPs (RMDPs) have already been used for experimental studies with tabular RL algorithms. In this paper, we use as a starting point a design suggested by Sutton and Kautz for discrete, enumerated state spaces2 , but we extend it in order to allow feature-vector representations of the states. Fig. 2 shows how transitions are performed in an RMDP. The left panel shows the case of a discrete, enumerated state space. For each state-action pair (s, a) the next state s is selected from a set of b possible next states, according to a the probability distribution Ps,s , j = 1, ..., b. The reward is then sampled from j a a a normal distribution with mean Rs,s and variance Vs,s . Such MDPs are easy to generate automatically. Our RMDPs are a straightforward extension of this design. A state is described by a feature vector: v1 , ..., vn , with vi ∈ [0, 1]. State transitions are governed by a mixture of b multivariate normal distributions (Gaussians) N (µj , σj ), with means µj = µ1j , . . . µnj and variances σj = σj1 , . . . σjn . The means µij = Mji (s, a) and variances σji = Vji (s, a) are functions of the current state-action pair, (s, a). Sampling from this mixture is performed hierarchically: first one of the b Gaussian components is selected according to probabilities Pj (s, a), j = 1, . . . b, then the next state s is sampled from the selected 1 2

Reinforcement learning repository at the University of Massachusetts, Amherst www-anw.cs.umass.edu/rlr www.cs.umass.edu/˜rich/RandomMDPs.html

398

Bohdana Ratitch and Doina Precup s a s1'

sj'

s

Taken with P(s,a,sj')

sb'

s' r~N(R(s,a,s'),V(s,a,s'))

a N1

Nj

Taken with Pj(s,a)

Nb

v1~N(Mj1(s,a),Vj1(s,a))... vn~N(Mjn(s,a),Vjn(s,a)) s' r~N(R(s,a,s'),V(s,a,s'))

Fig. 2. Random MDPs component, N (µj , σj )3 . Once the next state s is determined, the reward for the transition is sampled from a normal distribution with mean R(s, a, s ) and variance V (s, a, s ). The process may terminate at any time step according to a probability distribution P (s ). Mixtures of Gaussians are a natural and nonrestrictive choice for modeling multi-variate distributions. Of course, one can use other basis distributions as well. We designed a generator for RMDPs of this form4 , which uses as input a textual specification of the number of state variables, actions, branching factor, and also some constraints on the functions mentioned above. In these experiments we used piecewise constant functions to represent Pj (s, a), Mji (s, a), Vji (s, a), R(s, a, s ) and V (s, a, s ), but this choice can be more sophisticated. Mountain-Car[17] is a very well-studied minimum-time-to-goal task. The agent has to drive a car up a steep hill by using three actions: full throttle forward, full throttle reverse, or no throttle. The engine is not sufficiently strong to drive up the hill directly, so the agent has to build up sufficient energy first, by accelerating away from the goal. The state is described by two continuous state variables, the current position and velocity of the car. The rewards are -1 for every time step, until the goal is reached. If the goal has not been reached after 1000 time steps, the episode is terminated. We introduced noise in the classical version of the Mountain Car task by perturbing either the acceleration or the car position. In the first case, the action that corresponds to no throttle remained unaffected, while the other two actions were perturbed by zero-mean Gaussian noise. This is done by adding a random number to the acceleration of +1 or -1. The new value of the acceleration is then applied for one time step. In the second case, the car position was perturbed on every time step by zero-mean Gaussian noise. 4.2

Effect of Entropy and Controllability on Learning

We performed a set of experiments to test the hypothesis that STE and C have a statistically significant effect on the quality of the policies learned with finite 3 4

By setting the variances σji to zero and using discrete feature values, one can obtain RMDPs with discrete state spaces. The C++ implementation of the RMDP generator and the MDPs used in these experiments will be available from www.cs.mcgill.ca/˜sonce/


399

amounts of data. We used a benchmark suite consisting of 50 RMDPs and 10 randomized versions of the Mountain-Car task (RMC). All the RMDPs had two state variables and two actions. In order to estimate the average STE and C values for these tasks, each state variable was discretized into 10 intervals; then, 100 states were chosen uniformly (independently of the discretization) and 150 samples of state transitions were collected for each of these states.5 The STE and C values were estimated for each of these states (and each action, in the case of STE) using counts on these samples. Then the average value for each MDP was computed assuming a uniform state distribution.6 The RMDPs formed 10 groups with different combinations of average STE and C values, as shown in the left panel of Fig. 3. Note that it is not possible to obtain a complete factorial experimental design (where all fixed levels of one attribute are completely crossed with all fixed levels of the other attribute), because the upper limit on C is dependent on STE. However, the RMDP generator allows us to generate any STE and C combination in the lower left part of the graph, up to a limiting curve. For the purpose of this experiment, we chose attribute values distributed such that we can still study the effect of one attribute while keeping the other attribute fixed. Note that each group of RMDPs contains environments that have similar STE and C values, but which are obtained with different parameter settings for the RMDP generator. The RMDPs within each group are in fact quite different in terms of state transition structure and rewards. The average values of STE and C for the RMC tasks are shown in the right panel of Fig. 3. We used two tasks with acceleration noise (with variances 0.08 and 0.35 respectively) and eight tasks with position noise (with variances 5·10−5 , 9 · 10−5, 17 · 10−5, 38 · 10−5, 8 · 10−4, 3 · 10−3, 9 · 10−3 and 15 · 10−3). These tasks were chosen in order to give a good spread of the STE and C values. Note that for the RMC tasks, STE and C are anti-correlated. We used SARSA as the RL algorithm and CMACs as function approximators [17] to represent the action-value functions. The agent followed an -greedy exploration strategy with = 0.01. For all tasks, the CMAC had five 9×9 tilings, each offset by a random fraction of a tile width. For the RMDPs, each parameter w of the CMAC architecture had an associated learning rate which followed 1.25 , where nt is the number of updates to w a decreasing schedule αt = 0.5+n t performed by time step t. For the RMCs, we used a constant learning rate of α = 0.0625. These choices were made for each set of tasks (RMDPs and RMCs) based on preliminary experiments. We chose settings that seemed acceptable for all tasks in each set, without careful tuning for each MDP. 5 6

Note that by sampling states uniformly, we may get more than one state in one bin of the discretization, or we may get no state in another bin. For the purpose of estimating STE and C beforehand, the choice of either uniform state distribution or the distribution generated by a uniformly random policy are the only natural choices. Preliminary experiments indicate that the results under these two distributions are very similar.

400

Bohdana Ratitch and Doina Precup RMDPs

4

1.6

3.5

1.4

3

1.2

2.5

STE

STE

0.8

1.5

0.6

1

0.4

0.5

0.2

0

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Controllability (C)

Position noise Acceleration noise

1

2

0

RMC Tasks

0

0

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Controllability (C)

Fig. 3. Values of the Attributes for MDPs in the Benchmark Suite

For each task in the benchmark suite, we performed 30 learning runs. Each run consisted of 10000 episodes, where each episode started in a uniformly chosen state. For the RMDPs, the termination probability was set to 0.01 for all states. Every 100 trials, the current policy was evaluated on a fixed set of 50 test states, uniformly distributed across the state space. The best policy on a particular run r is the one with the maximum average return (M ARr ) over the states in the test set. The learning 30quality is measured as the average returns of the best policies found: LQ = r=1 M ARr . Note that it is not always possible to compare the LQ measure directly for different MDPs, because their optimal policies may have different returns (thus the upper limits on M ARr and LQ are different). So we need to normalize this measure across different MDPs. Ideally, we would like to normalize with respect to the expected return of the optimal policy. Since the optimal policy is not known, we normalize instead by the average return of the uniformly random policy over the same test states (RURP). The normalized LQ , learning quality (NLQ) measure used in the experiments is N LQ = RURP RURP if rewards are positive (as is the case of the RMDPs), and N LQ = LQ otherwise (for the RMCs). We conducted some experiments with RMDPs for which we knew the optimal policy and the results for the optimally and RURPnormalized LQ measures were very similar. Note that learning quality of RL algorithms is most often measured by the return of the final policy (rather than the best policy). In our experiments, the results using the return of the final policy are very similar to those based on the best policy (reported below), only less statistically significant. The within-group variance of the returns is much larger for the final policies, due to two factors. First, final policies are more affected by the learning rate: if the learning rate is too high, the agent may deviate from a good policy. Since the learning rate is not a factor in our analysis, it introduces unexplained variance. Secondly, SARSA with FA can exhibit an oscillatory behavior [5], which also increases the variance of the returns if they are measured after a fixed number of trials. We plan to study the effect of the learning rate more in the future. To determine if there is a statistically significant effect of STE and C on NLQ, we performed three kinds of statistical tests. First, we used analysis of


401

variance [2], to test the null hypothesis that the mean NLQ for all 10 groups of RMDPs is the same. We performed the same test for the 10 RMC tasks. For both domains, this hypothesis can be rejected at a significant confidence level (p < 0.01). This means that at least one of the two attributes has a statistically significant effect on NLQ. We also computed the predictive power (Hay’s statistic [2]) of the group factor, combining STE and C, on NLQ. The values of this statistic are 0.41 for the RMDPs and 0.57 for the RMCs. These values indicate that the effect of STE and C is not only statistically significant but also practically usable: the mean squared error in the prediction of the NLQ is reduced by 41% and 57% respectively for the RMDPs and RMC tasks, as a result of knowing the value of these attributes for the MDP. This result is very important because our long-term goal is to use knowledge about the attributes for making practical decisions, such as the choice of the algorithm or parameter settings for the task at hand. For the RMDP domains, the combination of STE and C values has the most predictive power (41%), whereas STE alone has only 4% prediction power and C alone has none. This suggests that both attributes have an effect on NLQ and should be considered together. Figure 4 shows the learning quality as a function of STE and C for the RMDPs (left panel) and for the RMC tasks (middle and right panels). Note that for the RMC tasks, we cannot study the effects of STE and C independently, because their values are anti-correlated. For ease of comparing the results to those obtained for RMDPs, we include two graphs for the RMC tasks, reflecting the dependency of NLQ on STE and C (middle and right panels of the figure 4). We emphasize that both graphs reflect one trend: as STE increases (and C decreases correspondingly), NLQ decreases. The reader should not conclude that STE and C exhibit independent effects in the case of the RMC tasks. As can be seen in the figure, for both domains (RMDPs and RMCs) the quality decreases as the entropy increases. We also conducted Least Significant Difference (LSD) tests [2] to compare the mean NLQ of the different pairs of RMDP groups and different pairs of RMC tasks. These tests (conducted at a conventional 0.05 confidence level), show that there is a statistically significant difference in the mean NLQ for all groups of RMDPs with different STE values, but the effect of STE becomes less significant as the value of the STE increases (potentially due to a floor effect). The trend is the same for the RMC tasks. As discussed in Sect. 3, high entropy is associated with the amounts of noise in the training data due to Factors 2 and 3 in Table 1, which makes learning more difficult. As we discussed in Sect. 2, the amount of noise also depends on the shape of the action-value functions. For example, if the action-value function is constant across the state-action space, then there will be no noise due to Factor 3 (see Table 1). Additional experiments with RMDPs that have relatively smooth and flat action-value functions7 showed that in this case, the learning quality increased as the STE increased. This is due to the positive effect of extensive state-space exploration in high-entropy MDPs. Thus, the effect of STE on learning quality is 7

In those RMDPs, one action has higher rewards than the other in all states and the functions R(s, a, s ) have a small range.

402

Bohdana Ratitch and Doina Precup RMDPs

1.5

16

1.45

1.4

STE~0.5

1.35

NLQ

16

14

14

12

12

NLQ

NLQ

1.25

1.2

1.15

STE~1.5 STE~3.5

1.1

0

0.2

0.4

0.6

Controllability (C)

0.9

8

8

6

6

4

4

2

RMC Tasks

10

10

1.3

1.05

RMC Tasks

2

0 0.25 0.37

STE

0.8

1.29 1.52

0.06 0.24

0.45 0.68 0.81

1

Controllability (C)

Fig. 4. Learning Quality a tradeoff between the negative effect of noise and the positive effect of natural state space exploration. The LSD tests also show differences in NLQ for the groups of RMDPs with different C values. The differences are significant between some of the groups with STE≈ 0.5 and STE≈ 1.5 levels. They appear when C changes by about 0.4. As can be seen from the left panel of Fig.4, the learning quality increases as controllability increases. As discussed in Sect. 3, high controllability means that the agent can better exploit the environment, and has more control over the exploration process as well.

5


In this paper, we proposed attributes to quantitatively characterize MDPs, in particular in terms of the amount of stochasticity. The proposed attributes can be either computed given the model of the process or estimated from samples collected as the agent interacts with its environment. We presented the results of an empirical study confirming that two attributes, state transition entropy and controllability, have a statistically significant effect on the quality of the policies learned by a reinforcement learning agent using linear function approximation. The experiments showed that better policies are learned in highly controllable environments. The effect of entropy shows a trade-off between the amount of noise due to environment stochasticity, and the natural exploration of the state space. The fact that the attributes have predictive power suggests that they can be used in the design of practical RL systems. Our experiments showed that these attributes also affect learning speed. However, statistically studying this aspect of learning performance is difficult, since there is no generally accepted way to measure and compare learning speed across different tasks, especially when convergence is not always guaranteed. We are currently trying to find a good measure of speed that would allow a statistically meaningful study. We are also currently investigating whether the effect of these attributes depends on the RL algorithm. This may provide useful information


403

in order to make good algorithmic choices. We are currently in the process of studying the effect of the other attributes presented in Sect. 3. The empirical results we presented suggest that entropy and controllability can be used in order to guide the exploration strategy of the RL agent. A significant amount of research has been devoted to sophisticated exploration schemes (e.g., [11], [4], [12]). Most of this work is concerned with action exploration, i.e. trying out different actions in the states encountered by the agent. Comparatively little effort has been devoted to investigating state-space exploration (i.e. explicitly reasoning about which parts of the state space are worth exploring). The E 3 algorithm [8] uses state-space exploration in order to find near-optimal policies in polynomial time, in finite state spaces. We are currently working on an algorithm for achieving good state-space exploration, guided by local measures of the attributes presented in Sect. 3. The agent uses a Gibbs (softmax) exploration policy [17]. The probabilities of the actions are based on a linear combination of the action values, local measures of the MDP attributes and the empirical variance of the FA targets. The weights in this combination are timedependent, in order to ensure more exploration in the beginning of learning, and more exploitation later.

Acknowledgments This research was supported by grants from NSERC and FCAR. We thank Ricard Gavald` a, Ted Perkins, and two anonymous reviewers for valuable comments.

References [1] Bertsekas, D. P., Tsitsiklis, J. N.: Neuro-Dynamic Programming. Belmont, MA: Athena Scientific (1996) 391 [2] Cohen, P. R.: Empirical Methods for Artificial Intelligence. Cambridge, MA: The MIT Press (1995) 401 [3] Dean, T., Kaelbling, L., Kirman, J., Nicholson, A.: Planning under Time Constraints in Stochastic Domains. Artificial Intelligence 76(1-2) (1995) 35-74 392 [4] Dearden, R., Friedman, N., Andre, D.: Model-Based Bayesian Exploration. In Uncertainty in Artificial Intelligence: Proceedings of the Fifteenth Conference (UAI1999) 150-159 403 [5] Gordon, J. G.: Reinforcement Learning with Function Approximation Converges to a Region. Advances in Neural Information Processing Systems 13 (2001) 10401046 400 [6] Hogg, T., Huberman, B. A., Williams, C. P.: Phase Transitions and the Search Problem (Editorial). Artificial Intelligence, 81 (1996) 1-16 391 [7] Hoos, H. H., Stutzle, T. : Local Search Algorithms for SAT: An Empirical Evaluation. Journal of Automated Reasoning, 24 (2000) 421-481. 391 [8] Kearns, M., Singh, S.: Near-Optimal Reinforcement Learning in Polynomial Time. In Proceedings of the 15th International Conference on Machine Learning (1998) 260-268 403 [9] Kirman, J.: Predicting Real-Time Planner Performance by Domain Characterization. Ph.D. Thesis, Brown University (1995) 392, 394, 395

404


[10] Lagoudakis, M., Littman, M. L. : Algorithm Selection using Reinforcement Learning Proceedings of the 17th International Conference on Machine Learning (2000) 511-518 391 [11] Meuleau, N., Bourgine, P.: Exploration of Multi-State Environments: Local Measures and Back-Propagation of Uncertainty. Machine Learning 35(2) (1999) 117154 403 [12] Moore, A. W., Atkeson, C. G.: Prioritized Sweeping: Reinforcement Learning with Less Data and Less Time. Machine Learning, 13 (1993) 103-130 403 [13] Papadimitriou, C. H., Steiglitz, K: Combinatorial Optimization: Algorithms and Complexity. Prentice Hall (1982) 391 [14] Papadimitriou, C. H., Tsitsiklis, J. N.: The Complexity of Markov Chain Decision Processes. Mathematics of Operations Research 12(3) (1987) 441-450 392 [15] Puterman, M. L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley (1994) 392 [16] Singh, S. P., Jaakkola, T., Jordan, M. I.: Reinforcement Learning with Soft State Aggregation. Advances in Neural Information Processing Systems, 7 (1995) 361368 391 [17] Sutton, R. S., Barto, A. G.: Reinforcement Learning. An Introduction. Cambridge, MA: The MIT Press (1998) 391, 392, 393, 398, 399, 403

Phase Transitions and Stochastic Local Search in k-Term DNF Learning Ulrich R¨ uckert, Stefan Kramer, and Luc De Raedt Machine Learning and Natural Language Lab, Institute of Computer Science, University of Freiburg Georges-K¨ ohler-Allee, Geb¨ aude 079, D-79110 Freiburg i. Br., Germany {rueckert,skramer,deraedt}@informatik.uni-freiburg.de

Abstract. In the past decade, there has been a lot of interest in phase transitions within artificial intelligence, and more recently, in machine learning and inductive logic programming. We investigate phase transitions in learning k-term DNF boolean formulae, a practically relevant class of concepts. We do not only show that there exist phase transitions, but also characterize and locate these phase transitions using the parameters k, the number of positive and negative examples, and the number of boolean variables. Subsequently, we investigate stochastic local search (SLS) for k-term DNF learning. We compare several variants that first reduce k-term DNF to SAT and then apply well-known SLS algorithms, such as GSAT and WalkSAT. Our experiments indicate that WalkSAT is able to solve the largest fraction of hard problem instances.

1

Introduction

The study of phase transitions of NP-complete problems [15, 3, 6, 7] has become quite popular within many subfields of artificial intelligence in the past decade. However, phase transitions have not yet received a lot of attention within the field of machine learning. Indeed, so far, there are only a few results that concern inductive logic programming [11, 8, 9]. The existence of phase transitions in inductive logic programming is not surprising because inductive logic programming is known to be computationally expensive, especially due to the use of θ-subsumption tests [21], which are NP-complete. In this paper, we study an important class of boolean formulae, i.e. k-term DNF, and show that phase transitions also occur in this propositional framework. The task in k-term DNF learning is to induce a DNF formula with at most k disjuncts (or terms) that covers all positive and none of the negative examples (this is the consistency requirement). Examples are boolean variable assignments. Learning k-term DNF is of practical relevance because one is often interested in finding the smallest set of rules that explains all the examples. This criterion is motivated by the principle of William of Ockham. Various practical machine learning systems employ the covering algorithm in the hope of finding small rule-sets. Moreover, k-term DNF has some interesting computational properties. It has polynomial sample complexity (in the PAC-learning sense) but T. Elomaa et al. (Eds.): ECML, LNAI 2430, pp. 405–417, 2002. c Springer-Verlag Berlin Heidelberg 2002

406

Ulrich R¨ uckert et al.

the consistency problem is hard [14]. The polynomial sample complexity implies that only a polynomial number of examples is needed in order to converge with high probability to a good approximation of the concept. On the other hand, the computation of complete and consistent concepts (the consistency problem) cannot be done in polynomial time (unless RP = N P ). The combination of these properties makes k-term DNF the ideal class of formulae to start an investigation of phase transitions in boolean learning. The contributions of this paper are as follows. First, we show that phase transitions exist for learning k-term DNF. This result is not surprising because of the hardness of the consistency problem. Secondly, we locate the phase transitions that arise in k-term DNF. Thirdly, we introduce the use of stochastic local search methods for learning hard k-term DNF problems. Stochastic local search algorithms approximate the optimal solution at much lower computational costs. Well-known examples of stochastic local search algorithms for SAT include GSAT and WalkSAT. Finally, our experiments demonstrate that these stochastic local search algorithms are effective. This paper is organized as follows: Section 2 introduces k-term DNF learning, and Section 3 identifies and localizes the phase transition in k-term DNF learning. Subsequently, Section 4 presents stochastic local search that is based on the reduction of k-term DNF learning to the satisfiability problem (SAT) and compares variants thereof on test sets of hard problem instances. Finally, Section 5 discusses further work, related work and concludes.

2

K-term DNF Learning

A k-term DNF formula is a disjunction of k terms, where each term is a conjunction of literals. E.g. (a1 ∧ ¬a2 ∧ a3 ) ∨ (a1 ∧ a4 ∧ a5 ) is a 2-term DNF with the terms (a1 ∧ ¬a2 ∧ a3 ) and (a1 ∧ a4 ∧ a5 ). The k-term DNF learning problem can now be formalized as follows [14]: Given – – – –

a a a a

set of Boolean variables V ar, set P os of truth value assignments pi : V ar → {0, 1}, set N eg of truth value assignments ni : V ar → {0, 1}, and natural number k,

Find a k-term DNF formula that is consistent with P os and N eg, i.e. that evaluates to 1 (true) for all variable assignments in P os and to 0 (f alse) for all variable assignments in N eg. We can make a few observations about this problem. First, for k = |P os|, we have a trivial solution F , where each term in F covers exactly one positive example. Obviously, we are only interested in problem instances with 1 ≤ k < |P os|. Second, if we know a solution F for a given kmin , we can easily derive solutions for any k > kmin . That means we can safely weaken the condition

Phase Transitions and Stochastic Local Search in k-Term DNF Learning

407

“formula needs to have exactly k terms” to “formula needs to have at most k terms”. Finally, assume that we have discovered a solution F for a given problem instance. Upon closer inspection we might discover that some literals are redundant in F , i.e. removing or adding these literals from or to F would still yield a solution. To examine which literals might be added to F , we can compute the least general specialization of F . The least general specialization (lgs) of F is a formula, that covers the same positive examples as F , but as few other instances as possible. To construct the lgs, we determine, which positive examples are covered by the individual terms in the solution: Covi =def {p ∈ P os| the ith term of F is satisfied by p}. We can then compute the lgs using the least general generalization (lgg) of those positive examples. The least general generalization of a set of examples e (over the variables V ari , 1 ≤ i ≤ n) can be efficiently computed by merging the literals:   V ari if all examples in e set V ari to 1 merge(i, e) =def ¬V ari if all examples in e set V ari to 0  1 otherwise lgg(e) =def

merge(i, e)

1≤i≤n

The least general specialization is then: lgs(F ) =def lgg(Covi (F )) 1≤i≤k

One can show that lgs(F ) is a solution if F is a solution. As a consequence, a problem instance P has a solution if and only if it has a solution, that is a least general specialization (proof omitted). We can leverage these considerations to construct a complete algorithm for solving the k-term DNF learning problem. Instead of searching through the space of all possible formulae, we only search for least general solutions. More precisely: 1. Recursively enumerate all possible partitionings of P os into k pairwise disjoint subsets Pi . This can be done by starting with an empty partitioning and adding one positive example per recursion step. 2. In every recursion step, build the formula F =def 1≤i≤k lgg(Pi ), which corresponds to the current (incomplete) partitioning. 3. Whenever F is satisfied by a negative example, backtrack, otherwise continue the recursion. 4. When all positive examples have been added and the resulting F did not cover any negative examples, F is a solution. Here is a short example: consider the learning problem with three variables V ar = {V1 , V2 , V3 }, three positive examples P os = {001, 011, 100}, two negative examples N eg = {101, 111} and k = 2. The algorithm will start with the empty

408


partitioning {∅, ∅} and recursively add the positive examples, thereby calculating the formula F = lgg(P1 ) ∨ lgg(P2 ). As soon as the algorithm reaches the partitioning {{001, 100}, {011}}, F will be ¬V2 ∨ (¬V1 ∧ V2 ∧ V3 ) and will cover the first negative example. Thus, the algorithm backtracks. However, when generating the partitioning {{001, 011}, {100}}, F is (¬V1 ∧ V3 ) ∨ (V1 ∧ ¬V2 ∧ ¬V3 ), which is consistent with N eg. The algorithm outputs F as a solution. Note that the terms of F are always satisfied by the positive examples in the corresponding subsets of the partitioning. The size of this search space for a k-term DNF learning problem with n positive examples is the number of possible partitionings of P os into k pairwise disjoint nonempty subsets. This is the Stirling number of the second kind k−1 1 i k n S(n, k) = k! i=0 (−1) i (k−i) [28]. For large n, S(n, k) grows approximately exponentially to the base k. For most practical settings this is considerably lower than 3|V ar|·k , the size of the space of all k-term formulae. Additionally, one can prune the search whenever a negative example is covered during formula construction. Note, however, that searching the partitioning space is redundant: two or more partitionings of P os might lead to the same formula. For that reason it might be more efficient to search the k-term formula space in some settings, especially for low k and high |P os|.

3

The Phase Transition

To identify the location and size of the phase transition for k-term DNF learning, we examined the solubility and search costs for randomly generated problem instances. K-term DNF learning problem instances can be classified by four parameters: the number of Boolean variables n, the size of the set of positive examples |P os|, the size of the set of negative examples |N eg| and k, the maximal number of terms in the desired formula. We generated the positive and negative examples of the problem instances by choosing either V ari = 1 or V ari = 0 with the same probability for each variable i, 1 ≤ i ≤ n. The search costs were measured by counting the number of partitionings generated by the complete algorithm sketched in Section 2. The search costs for finding a solution using the complete algorithm for such a randomly generated problem instance obviously depend on all of the four parameters. For instance, when keeping n, |P os|, and k fixed and varying |N eg|, one would expect to have – on average – low search costs for very low or very high |N eg|. With only a few negative examples, almost any formula covering P os should be a solution, hence the search should terminate soon. For very large |N eg|, we can rarely generate formulae covering even a small subset of P os without also covering one of the many negative examples. Consequently, we can prune the search early and search costs should be low, too. Only in the region between obviously soluble and obviously insoluble problem instances, the average search costs should be high. Similar considerations can be made about n and |P os|, but it is not obvious, to which degree each parameter affects solubility and average search costs.

Phase Transitions and Stochastic Local Search in k-Term D N F Learning

409

Fig, 1.Ps,!(ab eve) m d seas-& ~ d s t s(helow) pltitted as SD p p h grid m5tou.r ploi.foi the $ ~ o b l e m s e t t ~ nwihh,h .~ = 3, IPm[ = 15) 1 5 INq] 5 128,and ?.=;.a

.=l@

To examine this further, we calculated t h e probability PsOl of a problem instance being soluble a n d t h e search costs for a b r o a d range of problem settings. For instance, Figure 1shows the plots of these two quantities for fixed lPosl = 10 a n d k = 2, and varying Negl and n. Each d a t a point represents t h e average over 100 probleminstances. As expected, search costs are especially high in t h e region of Ps,l = 0.5. If t h e methods from s t a t i s t i c d mechanics c a n b e applied t o k-term DNF learning, we should b e able t o identify a "control parameter" a describing t h e location of t h e phase transition 16, 151. If finite-size scaling methods hold, we should b e able t o express PsOl as a function of this control parameter around some critical point a , [E, 21:

I

a-a, a, The t e r m

7mimics t h e

) .N?)

"reduced temperature"

9i n physical s y s t e m :

while t h e t e r m N ? provides t h e change of scale. As w i t h many other NPcomplete problems [TI, we expect f t o b e t h e cumulative distribution function of a normal distribution. Figure 2 shows t h a t PsOlincreases rapidly with t h e n u d e r of variables n . Thus, choosing n a s t h e control parameter seems t o b e a reasonable idea. W e (arbitrarily) choose the critical point n , s o t h a t PsOl(n,) = 0.5. Unlike w i t h

410

Ulrich R¨ uckert et al. 1 0.9

0.8

0.8

0.7

0.7 Normalized Search Costs

1 0.9

PSol

0.6 0.5 0.4

0.6 0.5 0.4

0.3

0.3

0.2

0.2

0.1

0.1

0

0

50

100

150

200 n

250

300

350

400

0

0

50

100

150

200 n

250

300

350

400

Fig. 2. PSol and search costs for problem settings with k = 2 and (|P os|, |N eg|) being – from left to right – (10, 100), (12, 60), (16, 20), (16, 60), and (16, 100)

some other NP-complete problems, in k-term DNF learning nc is not simply a constant. Instead, its value depends on |P os|, |N eg|, and k. We will now try to express nc as a function of |P os|, |N eg|, and k so that PSol (nc ) = 0.5. First of all, note that there is an inherent asymmetry in the constraints imposed upon a term by the positive and negative examples: assume we have a term c containing l literals. Assume further that c is consistent with some positive examples P osc ⊆ P os and all negative examples in N eg. If we require the term to cover a new (random) positive example p, we have to replace c with lgg(c, p). On average, we would expect the number of literals in lgg(c, p) to be half the number of literals in c. Since a formula contains more than one term, we will expect that c needs to cover only “suitable” examples, so we expect the number of literals in c to decrease slightly slower than by factor 0.5. Still, the number of literals decreases exponentially with the number of covered positive examples. On the other hand, if we add a new negative example e, the term has to differ only in one literal in order to be consistent with the new negative example. If l ≥ 1, c is already consistent with e in most cases. Only with probability 0.5l we do have to add one new literal to c. Thus, the number of literals in c will increase considerably slower than the number of negative examples consistent with c. This leads to two observations about nc : – Observation 1: nc grows exponentially with the number of positive examples |P os|. Assume, we found parameters n, |P os|, |N eg|, and k so that PSol (n, |P os|, |N eg|, k) = 0.5. If we add a new positive example e, a formula F has to additionally cover e in order to remain consistent with all positive examples. That means we have to replace at least one term c of F with lgg(c, e), effectively reducing the number of literals in F by some unknown factor. Then, F more likely covers a negative example and this in turn decreases PSol . In order to


411

350

300

250

n

200

150

100

50

0

0

20

40

60

80

100

120

140

|Neg|

Fig. 3. The location of nc for k = 2 depending on |N eg| for |P os| being – from bottom to top – 8, 9, 10, 12, 14, and 16

keep PSol constant we have to increase n by a factor β, thus restoring the previous level of literals in c. Since formulae have more than one term, the size of the exponent is an (unknown) function γ, depending on |P os| and k. This yields: (2) nc ≈ β γ(|P os|,k) – Observation 2: In fact, the value of β depends on the number of negative examples. Adding a new variable only increases PSol , if it increases the number of literals in F . The more negative examples are present, the more variables we have to add on average until we can add a new literal to F without making it inconsistent. As indicated by figure 1, nc grows with log |N eg|. This seems to be reasonable given the fact that – on average – we need 2l negative examples to increase the number of literals in term c by one. We would therefore expect that nc ∝ a · log2 (|N eg|), with the factor a depending on k. Assuming β = a · log2 (|N eg|) as described above, we obtain: nc ≈ (a log2 |N eg|)γ(|P os|,k)

(3)

a is the growth rate for fixed |P os| and variable |N eg|, while γ describes the growth rate of nc with increasing |P os|. To identify a and γ, we calculated nc for a set of problem settings in the range of 2 ≤ k ≤ 5, 1 ≤ |N eg| ≤ 120, and 7 ≤ |P os| ≤ 25. From the resulting graphs, we estimated a and γ(|P os|, k) using non-linear least square function regression (Nelder-Mead). We found that γ can be approximated very well by a linear function of |P os|: γ(x) =def b · x + c. Figure 3 shows the computed and the approximated value of nc for k=3 and |P os| ∈ {9, 12, 15, 18}. Table 1 shows the values of a, b, and c for 2 ≤ k ≤ 5. Finally, these considerations lead us to our hypothesis about nc .

412


Table 1. The values of a,b, and c for determining nc depending on the number of terms k Number of terms a b c 2 3.6995 0.080602 0.49471 3 1.8072 0.056234 0.68868 4 1.4542 0.041363 0.76301 5 1.3927 0.026572 0.85334

Hypothesis: nc ≈ (a · log2 |N eg|)(b·|P os|+c) a seems to converge for larger values of k. Unfortunately, k-term DNF learning requires huge computational resources for k > 5, so we could not examine this further. To verify the correctness of the approximation, we predicted nc at |P os| = 30, |N eg| = 30, and k = 3 to be about 178. We then computed 100 random problem instances and indeed found that PSol (30, 30, 3, 178) = 0.51, with an average search cost of 50 million recursions. In order to put our hypothesized nc to the test, we now check whether equation 1 adequately describes the phase transition. We computed PSol for k = 3, |P os| ∈ {10, 12, 14, 16, 18}, and |N eg| ∈ {20, 40, 60, 80, 100, 120}. We varied n between 1 and 384 and solved 1000 randomly generated problem instances per parameter setting to determine PSol . Figure 4 shows PSol for some selected probc lem settings, plotted against n and α(n) =def n−n nc . As can be seen, the selected problem settings can be adequately described by α, even though we did not introduce a “change of scale” parameter N 1/ν . Further investigations showed, that problem settings with the same |N eg| are virtually indistinguishable when plotted against α. Only for small |N eg|, the slope of PSol (α) is slightly smaller than predicted. However, similar anomalies for small control parameter values are known for other NP-complete problems as well [7, 18].

4

Stochastic Local Search

Most NP-complete problems can be formulated as a search through the space of possible solutions. For the hard problem instances the size of the search space is extremely large, and, as a consequence, complete algorithms require huge computational resources. Thus, for most practical problems, we are gladly willing to sacrifice completeness (i.e. the certainty of finding a solution, if there is one) for adequate runtime behavior. Though being incomplete, stochastic local search (SLS) algorithms have been shown to find solutions for many hard NP-complete problems in a fraction of the time required by the best complete algorithms. Since the introduction of GSAT [23], there has been a lot of research on SLS


0.8

0.8

0.7

0.7

0.6

0.6 PSol

1 0.9

PSol

1 0.9

0.5

0.5

0.4

0.4

0.3

0.3

0.2

0.2

0.1

0.1

0

413

0

100

200 n

300

400

0 −0.5

0 α =def (n − nc) / nc

0.5

Fig. 4. PSol for k = 2 and (|P os|, |N eg|) being – from left to right – (10, 80), (10, 100), (10, 120), (12, 80), (12, 100), (12, 120), (16, 80), (16, 100), and (16, 120), plotted against n and α

algorithms, and a couple of different variants have been proposed and evaluated [12, 24, 25, 17]. There are two main properties of SLS algorithms: first, instead of doing a systematic search through the whole instance space, an SLS algorithm starts a local search at a randomly selected location and restarts at different random locations, if it does not find a solution within a given time frame. Second, the search is local in a sense that it steps only to neighboring instances. During its search an SLS algorithm usually favors those instances, that minimize some predefined global evaluation function. To be successful, an SLS algorithm needs to have some way of escaping or avoiding local optima. Often, this is achieved by performing randomized steps from time to time (the so called “noise”1). An easy way to apply SLS algorithms to k-term DNF learning is to reduce a given k-term DNF learning problem to a satisfiability (SAT) problem and apply one of the many published SLS algorithms to the resulting SAT problem. In [13] Kamath et al. introduced a reduction of k-term DNF learning to SAT. They generated a test set of 47 k-term DNF learning problem instances ranging from problems with eight variables up to 32 variables. The reduction of this test set to SAT has been widely used as a benchmark for SAT SLS algorithms [26, 24, 23, 25]. Unfortunately, the test set seems to be very easy to solve. Kamath et al. used a target concept driven approach for constructing the problem instances. For each problem instance they built a random target formula. Then they uniformly generated random examples and labeled them according to the target formula. We reproduced the largest problem instances from the test set and found that even our complete algorithm solved all of them within a few 1 2

Note that this is different from the “noise” we use in machine learning! For GSAT+Tabu we state the size of the tabu table instead of a noise level

414


Table 2. The success rates (i.e. fraction of tries that found a solution) for various SLS algorithms running on the reduced test sets Noise Level2 Success Rate Success Rate Success Rate Test Set 1 Test Set 2 Test Set 3 GSAT n/a 78.5% 0% 0% 0.25 87.2% 0% 0% GSAT+RandomWalk 0.5 89.3% 1.7% 0% 0.75 56.8% 0.8% 0% 5 92.9% 0% 0% GSAT+Tabu 10 93.5% 0% 0% 15 84.4% 0% 0% 0.25 100% 97.5% 76.0% WalkSAT 0.5 100% 98.2% 62.6% 0.75 100% 90.5% 19.4% 0.25 93.1% 2.8% 0% Novelty 0.5 97.7% 4.2% 0% 0.75 98.1% 6.7% 0% Algorithm

seconds. Even worse, a propositional version of FOIL [19] was able to solve them in less than a second. Obviously, the information gain heuristic works especially well for problem instances, which were generated by sampling over the uniform distribution. Clearly, this test set is too easy to be used as a hard benchmark for k-term DNF learning. In order to evaluate SLS algorithms on harder problem instances, we generated three test sets, all taken from the phase transition region of the problem setting space. Each test set contains one hundred soluble (for k=3) problem instances. The first test set was generated with |P os| = 10, |N eg| = 10, n = 10, the second one with |P os| = 20, |N eg| = 20, n = 42 and the third one with |P os| = 30, |N eg| = 30, n = 180. We reduced the test sets to SAT using the reduction from [13]. The resulting SAT problems describe the desired solution F using 2 · |V ar| · k variables. They use another |P os| · k auxiliary variables to express, which positive example is covered by which term. The constraints put on those variables by the positive and negative examples are encoded in k · (|V ar| · (|P os| + 1) + |N eg|) + |P os| clauses. We tested a range of known SLS algorithms on the SAT-encoded problems of the test sets (see [12] for a description of the algorithms). We ran ten tries per problem instance and counted the number of successful tries (i.e. tries that found a solution) for each algorithm. For WalkSAT and Novelty each try was cut off after 100000 flips, for the GSAT based algorithms, we chose a cutoff value of 20 times the number of variables in the corresponding SAT problem. Table 2 shows the fraction of successful tries for each algorithm. On the hardest test set only WalkSAT yielded reasonable results. GSAT and its derivatives failed even on the second test set. Though WalkSAT and GSAT+RandomWalk are conceptually very similar, their results


415

differ strongly (similar results have been found in [26]). It seems that k-term DNF learning especially benefits from WalkSAT’s bias towards steps, which do not break any currently satisfied clause. This behavior ensures that the structural dependencies between the already satisfied clauses remain intact once they are found.

5

Conclusion

In the preceding sections we examined the NP-complete problem of k-term DNF learning. In Machine Learning we are not so much interested in the decision problem, but much more in the corresponding optimization problem: DNF minimization. As with many other NP-complete problems, an algorithm for the decision problem can be easily generalized to solving the optimization problem (usually by adding branch and bound techniques) and vice versa [7]. In fact, we found that the presented SLS algorithm was able to quickly find a solution for all k > kmin . This is also supported by the results in Section 2, where we identified the location of the phase transition: if a problem instance is located in the phase transition for a given k, the corresponding decision problem instances for all k > k are in the “obviously soluble” region. The DNF minimization problem settings seems to be at the core of most propositional concept learning settings, in the sense that: 1. Learning problems with discrete (and even with continuous-valued) attribute sets can be easily reformulated into a form that uses only two-valued (i.e. Boolean) attributes. This form corresponds exactly to our problem description. 2. Most propositional concept learners use representations that are subsets of or equivalent to DNF formulae, e.g. decision trees or disjunctive sets of rules. 3. Most concept learning algorithms include a bias towards a short representation of the hypotheses. While this might not necessarily increase the predictive accuracy, it is commonly considered as a desirable property [27, 4]. We showed that SLS algorithms can be successfully applied to hard randomly generated problem instances. Problem instances that are sampled from a uniform (or near uniform) distribution, can usually be solved in less than a few seconds by the presented SLS algorithm. The examples in the test cases were generated randomly; they do not follow a particular distribution. We would therefore expect that “real world” problems, which are obtained according to an (unknown) distribution are much easier to solve than the presented hard problem instances. However, it is not yet clear, whether or not SLS algorithms can efficiently deal with more structured problems. We are currently evaluating SLS algorithms for problem sets in the domain of chess endgames, such as the problem of predicting the minimum number of moves before a win on the Rook’s side in the KRK (King-Rook-King) endgame. The domain of chess endgames provides an ideal testbed for k-term DNF learning, since here we deal with noise-free datasets with discrete attributes only, and we are more interested in compression than in

416


predictivity. Finding a minimum theory for such endgame data was also a goal for previous research in this area [1, 22, 20] and is of continuing interest [5], but has not been tackled since then, partly due to the complexity of the task. Another field of interest is the application of SLS algorithms in different learning settings. SLS algorithms can easily be adapted to tolerating noise by introducing some noise threshold for the score. Whenever the score of a formula falls below this threshold, the remaining uncovered examples are considered as noise. Finally, we have to emphasize that stochastic search in general has been used before in Machine Learning (see, e.g., [10, 16]). However, to the best of our knowledge, this is the first attempt to introduce algorithms from stochastic local search (SLS) into propositional Machine Learning. Reducing the problem of k-term DNF to SAT, we can draw from a huge body of results from the area of satisfiability algorithms. In this sense, we hope that this work can stimulate further work along these lines in Machine Learning.

References [1] Bain, M. E. (1994) Learning logical exceptions in chess. PhD thesis. Department of Statistics and Modelling Science, University of Strathclyde, Scotland 416 [2] Barber, M. N. (1983) Finite-size scaling. Phase Transitions and critical phenomena, Vol. 8, 145-266, Academic Press 409 [3] Cheeseman, P., Kanefsky, B., and Taylor, W. M. (1991). Where the really hard problems are. Proceedings of the 12th IJCAI, 331-337 405 [4] Domingos, P. (1999) The role of Occam’s razor in knowledge discovery. Data Mining and Knowledge Discovery, Vol. 3, Nr. 4, 409-425 415 [5] F¨ urnkranz, J. (2002) Personal communication 416 [6] Gent, I. P., and Walsh, T. (1995). The number partition phase transition. Research report 95-185, Department of Computer Science, University of Strathclyde 405, 409 [7] Gent, I. P., and Walsh, T. (1996). The TSP phase transition, Artificial Intelligence, 88, 1-2, 349-358 405, 409, 412, 415 [8] Giordana, A., Saitta, L., Sebag, M., and Botta, M. (2000) Analyzing Relational Learning in the Phase Transition Framework. Proc. 17th International Conf. on Machine Learning, 311-318 405 [9] Giordana, A., Saitta, L. (2000) Phase Transitions in Relational Learning. Machine Learning, 41(2), 217-25 405 [10] Giordana, A., Saitta, L., and Zini, F. (1994) Learning disjunctive concepts by means of genetic algorithms. Proc. 11th International Conf. on Machine Learning, 96-104 416 [11] Giordana, A., Botta, M., and Saitta, L. (1999) An experimental study of phase transitions in matching. IJCAI 1999, 1198-1203 405 [12] Hoos, H. H. (1998) Stochastic local search - methods, models, applications, PhD Thesis, Technische Universit¨ at Darmstadt 413, 414 [13] Kamath, A. P., Karmarkar, N. K., Ramakrishnan, K. G., and Resende, M. G. C. (1991). A continous approach to inductive inference. Mathematical Programming, 57, 1992, 215-238. 413, 414


417

[14] Kearns, M. J., and Vazirani, U. V. (1994). An introduction to computational learning theory. Cambridge, MA: MIT Press 406 [15] Kirkpatrick, S., and Selman, B. (1994). Critical behavior in the satisfiability of random boolean expressions. Science, 264, 1297-1301 405, 409 [16] Kovacic, M. (1994) MILP – a stochastic approach to Inductive Logic Programming. Proc. 4th International Workshop on Inductive Logic Programming, 123–138 416 [17] McAllester, D., Selman, B., and Kautz, H. (1997) Evidence for invariants in local search. Proceedings of the 14th National Conference on Artificial Intelligence, 321326 413 [18] Mitchell, D., Selman, B., and Levesque, H. (1992) Hard and easy distributions of SAT problems. Proceedings of the 10th National Conference on Artificial Intelligence, AAAI Press/ MIT Press, San Jose, CA, 459-465 412 [19] Mooney, R. J. (1995) Encouraging experimental results on learning CNF. Machine Learning, Vol. 19, 1, 79-92 414 [20] Nalimov, E. V., Haworth, G.McC., and Heinz, E. A. (2000) Space-efficient indexing of chess endgame tables. ICGA Journal, Vol. 23, Nr. 3, 148-162 416 [21] Plotkin, G. D. (1970) A note on inductive generalization. Machine Intelligence 5, Edinburgh University Press, 153-163. 405 [22] Quinlan, J. R., and Cameron-Jones, R. M. (1995) Induction of logic programs: FOIL and related systems. New Generation Computing, Vol. 13, 287-312 416 [23] Selman, B., and Kautz, H. A. (1992) A new method for solving hard satisfiability problems. Proceedings of the 10th National Conference on Artificial Intelligence, San Jose, CA, 440-446 412, 413 [24] Selman, B., Kautz, H. A., and Cohen, B. (1993) Local search strategies for satisfiability testing. Proceedings of the 10th National Conference on Artificial Intelligence, San Jose, CA 413 [25] Selman, B., and Kautz, H. A. (1993) Domain-independent extensions to GSAT: solving large structured satisfiability problems. Proceedings of IJCAI 93, 290-295 413 [26] Selman, B., Kautz, H. A. and Cohen, B. (1994) Noise strategies for improving local search. Proceedings of the 14th National Conference on Artificial Intelligence, 337343 413, 415 [27] Webb, G. J. (1996) Further evidence against the utility of Occam’s razor. Journal of Artificial Intelligence Research, Vol. 4, 397-417 415 [28] Weisstein, E. W. (2002) Stirling number of the second kind, http://mathworld.wolfram.com/StirlingNumberoftheSecondKind.html 408

Discriminative Clustering: Optimal Contingency Tables by Learning Metrics Janne Sinkkonen, Samuel Kaski, and Janne Nikkil¨ a Helsinki University of Technology, Neural Networks Research Centre P.O. Box 9800, FIN-02015 HUT, Finland {Janne.Sinkkonen,Samuel.Kaski,Janne.Nikkila}@hut.fi http://www.cis.hut.fi/projects/mi

Abstract. The learning metrics principle describes a way to derive metrics to the data space from paired data. Variation of the primary data is assumed relevant only to the extent it causes changes in the auxiliary data. Discriminative clustering finds clusters of primary data that are homogeneous in the auxiliary data. In this paper, discriminative clustering using a mutual information criterion is shown to be asymptotically equivalent to vector quantization in learning metrics. We also present a new, finite-data variant of discriminative clustering and show that it builds contingency tables that detect optimally statistical dependency between the clusters and the auxiliary data. A finite-data algorithm is demonstrated to outperform the older mutual information maximizing variant.

1

Introduction

The metric of the data space determines the goodness of the results of unsupervised learning: clustering, nonlinear projection methods, and density estimation. The metric, in turn, is determined by feature extraction, variable selection, transformation, and preprocessing of the data. The principle of learning metrics aims at automating part of the process of metric selection, by learning the metric from data. It is assumed that the data comes in pairs (x, c): during learning, the primary data vectors x ∈ Rn are paired with auxiliary data c which in this paper are discrete classes. Important variation in x is supposed to be revealed by variation in the the conditional density p(c|x). The distance d between two close-by data points x and x + dx is defined to be the difference between the corresponding distributions of c, measured by the Kullback-Leibler divergence DKL . It is well known (see e.g. [3]) that the divergence is locally equal to the quadratic form with the Fisher information matrix J, i.e. d2L (x, x + dx) ≡ DKL (p(c|x)p(c|x + dx)) = dxT J(x)dx .

(1)

The Fisher information matrix has classically appeared in the context of constructing metrics for probabilistic model families. A novelty here is that the data T. Elomaa et al. (Eds.): ECML, LNAI 2430, pp. 418–430, 2002. c Springer-Verlag Berlin Heidelberg 2002

Discriminative Clustering

419

vector x is considered as the parameters of the Fisher information matrix, the aim being to construct a new metric into the data space. The Kullback-Leibler divergence defines a metric locally, and the metric can in principle be extended to an information metric or Fisher metric to the whole data space. We call the idea of measuring distances in the data space by approximations of (1) the learning metrics principle [1, 2]. The principle is presumable useful for tasks in which there is suitable auxiliary data available, but instead of merely predicting the values of auxiliary data the goal is to analyze, explore, or mine the primary data. Charting companies based on financial indicators is one example; there the bankruptcy risk (whether the company goes bankrupt or not) is natural auxiliary data. Learning metrics is similar to supervised learning in that the user has to choose proper auxiliary data. The difference is that in supervised learning the sole purpose is to predict the auxiliary data, whereas in learning metrics the metric is supervised while the rest of the analysis can be unsupervised, given the metric. In this paper we analyze clustering in learning metrics, or discriminative clustering (earlier also called semisupervised clustering) [2]. In general, a goal of clustering is to minimize within-cluster distortion or variation, and to maximize between-cluster variation. We apply the learning metrics by measuring distortions within each cluster by a kind of within-cluster Kullback-Leibler divergence. This causes the clusters to be internally as homogeneous as possible in conditional distributions p(c|x) of the auxiliary variable. The mutual differences between the distributions p(c|x) of the clusters are then automatically maximized, giving a reason to call the method discriminative. We have earlier derived and analyzed discriminative clustering with information-theoretic methods, assuming infinite amount of data. In this paper we will derive a finite-data variant and theoretical context for it, in the limit of “hard” clusters (vector quantization). It is not possible to use gradientbased algorithms for hard clusters, and hence we derive optimization algorithms for a smooth variant for which standard fast optimization procedures are then applicable.

2 2.1

Discriminative Clustering Is Asymptotically Vector Quantization in Fisher Metrics Discriminative Clustering

We will first introduce the cost function of discriminative clustering by applying the learning metrics principle to the classic vector quantization or K-means clustering. In vector quantization the goal is to find a set of prototypes or codebook vectors mj that minimizes the average distortion E caused when the data are

420

Janne Sinkkonen et al.

represented by the prototypes: E=

Vj

j

D(x, mj ) p(x) dx ,

(2)

where D(x, mj ) is the distortion caused by representing x by mj , and Vj is the Voronoi region of the cell j. The Voronoi region Vj consists of all points that are closer to mj than to any other model, that is, x ∈ Vj if D(x, mj ) ≤ D(x, mk )

(3)

for all k. The learning metrics principle is applied to (2) by introducing a set of distributional prototypes ψ j , one for each partition j, and by measuring distortions of representing the distributions p(c|x) by the prototypes ψ j . The average distortion is EKL = DKL (p(c|x), ψj ) p(x) dx , (4) j

Vj

where distortion between distributions has been measured by the KullbackLeibler divergence. Note that the Voronoi regions Vj are still kept local in the primary data space by defining them with respect to the Euclidean distortion (3). The cost (4) is minimized with respect to both sets of prototypes, mj and ψ j . The optimization is discussed further in Section 5. It can be shown that minimizing (4) maximizes the mutual information between the auxiliary data and the clusters, considered as a random variable [2]. This holds even for the soft variant discussed in Section 5. 2.2

Asymptotic Connection to Learning Metrics

In this section we aim to clarify the motivation behind discriminative clustering, by deriving a connection between it and the learning metrics principle of using (1) as the distance measure. The connection is only theoretical in that it holds only for the asymptotic limit of a large number of clusters, whereas in practice the number of clusters will be small. The asymptotic connection can be derived under some simplifying assumptions. It is assumed that almost all Voronoi regions become increasingly local when their number increases. (In singular cases, the data samples are identified with their equivalence classes having zero mutual distance.) There are always some non-compact and therefore inevitably non-local Voronoi regions at the borders of the data manifold, but it is assumed that the probability mass within them can be made arbitrarily small by increasing the number of regions. Assume further that the densities p(c|x) are differentiable. Then the class distributions p(c|x) can be made arbitrarily close to linear within each region Vj by increasing the number of Voronoi regions.


421

Let EVj ´ adenote the expectation over the Voronoi region Vj with respect to the probability density p(x). At the optimum of the cost EKL , we have ψj = EVj [p(c|x)], i.e. the parameters ψ j are equal to the means of the conditional distribution within the Voronoi regions (see [2]; this holds even for the soft clusters). Since p(c|x) is linear within each Voronoi region, there exists a linear operator Lj for each Vj , for which p(c|x) = Lj x. The distributional prototypes then become ˜ j = p(c|m ˜ j) , ψ j = EVj [p(c|x)] = EVj [Lj x] = Lj EVj [x] ≡ Lj m and the cost function becomes ˜ j(x) )) p(x) dx . EKL = DKL (p(c|x), p(c|m j

Vj

˜ j = EVj [x] for each That is, given a locally linear p(c|x), there exists a point m Voronoi region such that the Kullback-Leibler divergence appearing in the cost ˜ j(x) ) instead of function can be measured with respect to the distribution p(c|m the average over the whole Voronoi region. Since the Kullback-Leibler divergence is locally equal to a quadratic form of ˜ j to get the Fisher information matrix, we may expand the divergence around m ˜ j(x) )T J(m ˜ j(x) )(x − m ˜ j(x) ) p(x) dx , EKL = (x − m (5) j

Vj

˜ j(x) . ˜ j(x) ) is the Fisher information matrix evaluated at m where J(m Note that the Voronoi regions Vj are still defined by the parameters mj and in the original, usually Euclidean metric. In summary, discriminative clustering or maximization of mutual information asymptotically finds a partitioning from the family of local Euclidean Voronoi partitionings, for which the within-cluster distortion in the Fisher metric is minimized. In other words, discriminative clustering asymptotically performs vector quantization in the Fisher metric by Euclidean Voronoi regions: Euclidean metrics define the family of Voronoi partitionings {Vj }j over which the optimization is done, and the Fisher metric is used to measure distortion inside the regions.

3 3.1

Estimation from Finite Data Maximum Likelihood

Note that for finite data minimizing the cost function (4) is equivalent to maximizing log ψj,c(x) , (6) L= j

x∈Vj

where c(x) is the index of the class of the sample x. This is the log likelihood of a piece-wise constant conditional density estimator. The estimator predicts

422


the distribution of C to be ψ j within the Voronoi region j. The likelihood is maximized with respect to both the ψ j and the partitioning, under the defined constraints. 3.2

Maximum a Posteriori

The natural extension of maximum likelihood estimation is to introduce a prior and to find the maximum a posterior (MAP) estimate. The Bayesian framework is particularly natural for discriminative clustering since we are actually interested only on the resulting clusters, not the distribution of the auxiliary data within them. The class distributions can therefore be conveniently integrated out from the posterior (although seemingly paradoxical, the auxiliary data of course guides the clustering). Denote the observed auxiliary data set by D(c) , and the primary data set by D(x) . We then wish to find the set of clusters {m} which maximizes the posterior p({m}, {ψ}|D(c) , D(x) )d{ψ} , p({m}|D(c) , D(x) ) = {ψ}

or equivalently log p({m}|D(c) , D(x) ). Here the integration is over all ψ j . Denote the number of classes by Nc , the number of clusters by k, and the total number of samples by N . Denote the part of the data assigned to cluster j (c) by Dj , and the number of data samples of class i in cluster j by nji . Further denote Nj = i nji . Assume the improper and separable prior p({m}, {ψ}) ∝ p({ψ}) = j p(ψ j ). Then, (c) (x) p(D(c) |{m}, {ψ}, D(x) )p({ψ})d{ψ} p({m}|D , D ) ∝ {ψ} (c) p(Dj |{m}, ψ j , D(x) )p(ψ j ) dψ j = j

=

ψj

j

ψj

i

n

ψjiji p(ψ j ) dψ j ≡

Qj .

j

n0 −1 We will use a conjugate (Dirichlet) prior, p(ψ j ) ∝ i ψjii , where n0 = {n0i }i are the prior parameters common to all j, and N 0 = i n0i . Then the “partition(c) specific” density p(Dj |{m}, ψ j )p(ψ j ) is Dirichlet with respect to ψ and the factors Qj of the total posterior become

Qj =

Z

(c)

p(Dj |{m}, j

j, D

(x)

)p(

j) d

j

∝

Z Y ψ

n0 i +nji −1 ij

j

The log of the posterior probability then is

i

d

j

=

Q Γ(n

0 i

+ nji ) . Γ (N 0 + Nj ) i


log p({m}|D(c) , D(x) ) =

log Γ(n0i + nji ) −

ij

log Γ(N 0 + Nj ) .

423

(7)

j

In MAP estimation this function needs to be maximized. 3.3

Asymptotic Connection to Maximization of Mutual Information

It is shown that for a fixed number of clusters, the cost function (7) of the new method approaches mutual information as the number of data samples increases. Denote sji ≡ n0i + nji − 1, Sj ≡ i sji = N 0 + Nj − Nc , and S = j Sj . Then, log Γ(sji + 1) − log Γ[(Sj + Nc − 1) + 1] . (8) log p({m}|D(c) , D(x) ) = ij

j

It is straightforward to show using the Stirling approximation and Taylor approximations (Appendix A), that Nc k(log S + 1) sji /S log p({m}|D(c) , D(x) ) = +O sji /S log , (9) S Sj /S S ij where sji /S approaches pji , the probability of class i in cluster j, and Sj /S approaches pj as the number of data samples increases. Hence, (9) approaches the mutual information, added by a constant.

4

Discriminative Clustering Optimizes Contingency Tables

Contingency tables (see [4]) are classical methods for measuring statistical dependency between discrete-valued (categorical) random variables. The categories are fixed before the analysis, and for two variables the co-occurrences of the categories in a sample are tabulated into a two-dimensional table. A classic example due to Fisher is to measure whether the order of adding milk and tea affects the taste. The first variable indicates the order of adding the ingredients, and the second whether the taste is better or worse. In medicine the other variable could indicate health status and the other one demographic groups. The resulting contingency table is tested for dependency between the row and column variables. The literature for various kinds of tests and uses of contingency tables is extensive, see for example [4, 5, 6, 7]. The effect of small sample sizes and/or small cell frequencies has been the subject of much controversy. Bayesian methods are principled means for coping with small data sets; below we will derive a connection between the Bayesian approach presented in [7], and our discriminative clustering method. Given discrete-valued auxiliary data, the result of any clustering method can be analyzed as a contingency table. The possible values of the auxiliary variable correspond to columns and the clusters to rows. Clustering compresses

424


a potentially large number of multivariate continuous-valued observations into a manageable number of categories, and the contingency table can, at least in principle, be tested for dependency. Note that the difference from the traditional use of contingency tables is that the row categories are not fixed but clustering tries to find a suitable categorization. The question here is, is discriminative clustering a good way of constructing such contingency tables? The answer is that it is optimal in the sense introduced below. Good [7] derived a “Bayesian test” for dependency in contingency tables by computing the Bayes factor against H, ¯ P ({nij }|H) , P ({nij }|H)

(10)

where H is the hypothesis of statistical independence of the row and column categories. The probabilities are derived assuming mixtures of Dirichlet distributions as priors. In the special case of one fixed margin (the auxiliary data) in the contingency table, and the prior defined in Section 3.21 , the Bayes factor is ¯ P ({nij }|{n(ci )}, H) P ({nij }|{n(ci )}, H) 0 Γ (N 0 )k Γ (kn0 )Nc Γ (N + kN 0 ) i,j Γ (nji + n ) × = 0 Γ (n0 )Nc k i Γ (n(ci ) + kn0 )Γ (kN 0 ) j (Nj + N ) = p({m}|D(c) , D(x) ) × const. , (11) where the constant does not depend on Nj or nij . Here n(ci ) denotes the number of samples in the (auxiliary) class ci . MAP estimation for discriminative clustering is thus equivalent to constructing a dependency table that results in a maximal Bayes factor, under the constraints of the model.

5

Algorithms

Optimization of both variants of discriminative clustering, the finite data version (7) and the infinite-data version (4), is hard since the gradient is zero except on the Voronoi borders. Hence gradient-based optimization algorithms are not applicable. We have earlier [2] proposed a “smoothed” infinite-data variant which can be optimized by an on-line algorithm, reviewed below. A similar smoothed variant will be introduced for MAP estimation as well. 5.1

Algorithm for Large Data Sets

Smooth parameterized membership functions yj (x; {m})) were introduced to the cost function (4). Their values vary between 0 and 1, and j yj (x) = 1. The 1

In contrast to [7], we used priors with equal total amount of “prior data” for both hypotheses.


425

smoothed cost function is EKL

=

yj (x; {m})DKL (p(c|x), ψj ) p(x) dx .

(12)

j

The membership functions can be for instance normalized Gaussians, 2 2 yj (x) = Z −1 (x)e−x−mj /σ , where Z normalizes the sum to unity for each x. The cost function can be minimized by the following stochastic approximation algorithm. Denote the i.i.d. data pair at the on-line step t by (x(t), c(t)) and index the (discrete) value of c(t) by i, that is, c(t) = ci . Draw two clusters, j and l, independently with probabilities given by the membership functions {yk (x(t))}k . Reparameterize the distributional prototypes by the “soft-max”, log ψji = γji − log m exp(γjm ), to keep them summed up to unity. Adapt the prototypes by mj (t + 1) = mj (t) − α(t) [x(t) − mj (t)] log

ψli (t) ψji (t)

(13)

γjm (t + 1) = γjm (t) − α(t) [ψjm (t) − δmi ] ,

(14)

where δmi is the Kronecker delta. Due to the symmetry between j and l, it is possible (and apparently beneficial) to adapt the parameters twice for each t by swapping j and l in (13) and (14) for the second adaptation. Note that no updating takes place if j = l, i.e. then mj (t + 1) = mj (t). During learning the parameter α(t) decreases gradually toward zero according to a schedule that fulfills the conditions of the stochastic approximation theory. 5.2

MAP Algorithm for Finite Data Sets

In an analogous fashion to the infinite-data variant we postulate smooth membership functions yj (x; {m}) that govern the assignment of the data x to the clusters. Then the smoothed “number” of samples of class i within cluster j becomes nij = c(x)=i yj (x), and the MAP cost function (7) becomes log p({m}|D(c) , D(x) ) =

 log Γn0i +

ij

c(x)=i

 yj (x)−

log Γ Nj0 +

j

yj (x) .

x

(15) For normalized Gaussian membership functions the gradient of the cost function with respect to the jth model vector is (Appendix B) σ2

∂ log p({m}|D(c) , D(x) ) = (x − mj )yl (x)yj (x)(Lc(x),j − Lc(x),l ) , (16) ∂mj x,l

where Lij ≡ Ψ(nji + n0i ) − Ψ(Nj + Nj0 ) .

426

Jmne Sinkkonen et a1

Here Q is the digamma function, derivative of the logarithm of r. The MAP estimate can then be solved with general-purpose nonlinear optimization methods. We have used the conjugate gradient algorithm. Note that Q approaches the logarithm when its argument grows, and hence for large data sets the gradient approaches the average of (13) over the d a t a and the lth membership function, with $ji nij/Nj.

--

6

Empirical Results

The algorithm is first demonstrated with a toy example in Figure 1. The data (10,000 samples) comes from a two-dimensional spherically symmetric Gaussian distribution. The two-class auxiliary data changes only in the vertical dimension, indicating that only the vertical dimension is relevant. The algorithm learns to model only the relevant dimension. As far as we know there do not exist alternative methods for precisely the same task, partitioning the primary data space to clusters that are homogeneous in terms of the auxiliary data. We have earlier compared the older mutual information maximizing variant (section 5.1) with two clustering methods: the plain mixture of Gaussians a n d MDA2 [S,91, a mixture model for the joint distribution of primary and auxiliary data. For gene expression data our algorithm outperformed the alternatives [2]. Here we will add the new variant (section 5.2) to the comparison. A random half of the Landsat satellite data set from the UCI Machine Learning Repository (36 dimensions, six classes, and 6435 samples) was partitioned into 2-10 clusters, using the six-fold class indicator as the auxiliary data. For each number of clusters, solutions were computed for 30 values of the smoothing parameter u , ranging from two to 100 on the logarithmic scale. All the prior parameters ny were set to unity. The models were evaluated by computing the log-posterior probability (7) of the left-out data.

Fig. 1.A demonstration of the MAP algorithm. The probability density function of the data is shown in shades of gray and the cluster centers with circles. The conditional density of one of the two auxiliary classes is shown in the inset. (Here u = 0.4)

Discriminative Clustering 5 clusters

6 clusters

7 clusters

-2000

-2000

-2000

-3000

-3000

-3000

-4000

-4000

-4000

-5000

-5000

-5000

2.

5.

10.0

20.

50.

2.

5.

8 clusters

10.0

20.

50.

2.

-2000

-3000

-3000

-3000

-4000

-4000

-4000

-5000

-5000

-5000

10.0

20.

50.

2.

5.

10.0

20.

10.0

20.

50.

10 clusters

-2000

5.

5.

9 clusters

-2000

2.

427

50.

2.

5.

10.0

20.

50.

Fig. 2. The performance of the conjugate-gradient MAP algorithm (solid line) compared to the older discriminative clustering algorithm (dashed line), plain mixture of Gaussians (dotted line) and MDA2, a mixture model for the joint distribution of primary and auxiliary data (dash-dotted line). Sets of clusters were computed with each method with several values of the smoothing parameter σ, and the posterior log-probability (7) of the validation data is shown for a hard assignment of each sample to exactly one cluster. Results measured with empirical mutual information (not shown) are qualitatively similar. The smallest visible value corresponds to assigning all samples to the same cluster

The log-posterior probabilities of the validation set are presented in Figure 2. For all numbers of clusters the new algorithm performed better, having a larger edge at smaller numbers of clusters. Surprisingly, in constrast to earlier experiments with other data sets, for this data set the alternative clustering methods seem to outperform the older variant of discriminative clustering. For 4–7 clusters, the models were compared by ten-fold cross-validation. The best value for σ was chosen with validation data, in preliminary tests. The new model was significantly better for all cluster numbers (paired t test, p< 0.001).

7

Conclusions

In summary, we have applied the learning metrics principle to clustering, and coined the approach discriminative clustering. It was shown that discriminative clustering is asymptotically, in the limit of a large number of clusters, equivalent

428


to clustering in Fisher metrics, with the additional constraint that the clusters are (Euclidean) Voronoi regions in the primary data space. In the earlier work [1] Fisher metrics were derived from explicit conditional density estimators for clustering with Self-Organizing Maps; discriminative clustering has the advantage that the (arbitrary) density estimator is not required. We have derived a finite-data discriminative clustering method that maximizes the posterior probability of the cluster centroids. There exist related methods for infinite data, proposed by us and others, derived by maximizing the mutual information [2, 10, 11]. For discrete primary data there exist also finite-data generative models [12, 13]; the main difference in our methods is the ability to derive a metric to continuous primary data spaces. Finally, we have shown that the cost function is equivalent to the Bayes factor of a contingency table with the marginal distribution of the auxiliary data fixed. The Bayes factor is the odds of the data likelihood given the hypothesis that the rows and columns are independent, vs. the alternative hypothesis of dependency. Hence, discriminative clustering can be interpreted to find a set of clusters that maximize the statistical dependency with the auxiliary data.

Acknowledgment This work was supported by the Academy of Finland, in part by the grants 50061 and 52123.

References [1] Kaski, S., Sinkkonen, J., Peltonen, J.: Bankruptcy analysis with self-organizing maps in learning metrics. IEEE Trans. Neural Networks 12 (2001) 936–947 419, 428 [2] Sinkkonen, J., Kaski, S.: Clustering based on conditional distributions in an auxiliary space. Neural Computation 14 (2002) 217–239 419, 420, 421, 424, 426, 428 [3] Kullback, S.: Information Theory and Statistics. Wiley, New York (1959) 418 [4] Agresti, A.: A survey of exact inference for contingency tables. Statistical Science 7 (1992) 131–153 423 [5] Fisher, R. A.: On the interpretation of χ2 from the contingency tables, and the calculation of p. J. Royal Stat. Soc. 85 (1922) 87–94 423 [6] Freeman, G. H., Halton, J. H.: Note on an exact treatment of contingency, goodness of fit and other problems of significance. Biometrika 38 (1951) 141–149 423 [7] Good, I. J.: On the application of symmetric Dirichlet distributions and their mixtures to contingency tables. Annals of Statistics 4 (1976) 1159–1189 423, 424 [8] Hastie, T., Tibshirani, R., Buja, A.: Flexible discriminant and mixture models. In Kay, J. Titterington, D. (eds): Neural Networks and Statistics. Oxford University Press (1995) 426 [9] Miller, D. J., Uyar, H. S.: A mixture of experts classifier with learning based on both labelled and unlabelled data. In Mozer, M., Jordan, M., Petsche, T. (eds): Advances in Neural Information Processing Systems 9. MIT Press, Cambridge, MA (1997) 571–577 426


429

[10] Becker, S.: Mutual information maximization: models of cortical self-organization. Network: Computation in Neural Systems 7 (1996) 7–31 428 [11] Tishby, N., Pereira, F. C., Bialek, W.: The information bottleneck method. In: 37th Annual Allerton Conference on Communication, Control, and Computing. Urbana, Illinois (1999) 428 [12] Hofmann, T., Puzicha, J., Jordan, M. I.: Learning from dyadic data. In: Kearns, M. S., Solla, S. A., Cohn, D. A. (eds): Advances in Neural Information Processing Systems 11. Morgan Kaufmann Publishers, San Mateo, CA (1998) 466–472 428 [13] Hofmann, T.: Unsupervised learning by probabilistic latent semantic analysis. Mach. Learning (2001) 177–196 428

A

Connection of MAP Estimation to Maximization of Mutual Information

The Stirling approximation log Γ(s + 1) = s log s − s + O(log s) applied to (8) yields log p({m}|D(c) , D(x) ) =

sji log sji −

ij

− (Nc − 1)

Sj log(Sj + Nc − 1)

j

log(Sj + Nc − 1) + O(Nc k(log S + 1)) .

j

The zeroth-order Taylor expansion log(S + n) = log S + O Sn gives after rearrangements, for Sj > 1, log p({m}|D(c) , D(x) ) = sji log sji − Sj log Sj + O(Nc k(log S + 1)) . ij

j

Division by S then gives (9).

B

Gradient of the MAP Cost Function

Denote for brevity tji = nji + n0i and Tj = respect to mj is

i tji .

The gradient of (15) with

∂ ∂ ∂ log p({m}|D(c) , D(x) ) = yl (x)Ψ(tli ) − yl (x)Ψ(Tl ) ∂mj ∂mj ∂mj il c(x)=i

x,l

∂ yl (x)[Ψ(tl,c(x) ) − Ψ(Tl )] . = ∂mj x,l

It is straightforward to show that for normalized Gaussian membership functions 1 ∂ yl (x) = 2 (x − mj )(δlj − yl (x))yj (x) . ∂mj σ

430


Substituting this to the gradient gives σ2

∂ log p({m}|D(c) , D(x) ) = (x−mj )(δlj −yl (x))yj (x)[Ψ(tl,c(x) )−Ψ(Tl)] . ∂mj x,l

(17) The final form (16) for the gradient results from applying the identity (δlj − yl )yj Ll = yl yj (Lj − Ll ) , l

to (17).

l

Boosting Density Function Estimators Franck Thollard, Marc Sebban, and Philippe Ezequel EURISE, Department of Computer Science Université Jean Monnet de Saint-Etienne {franck.thollard,marc.sebban,ezequel}@univ-st-etienne.fr

Abstract. In this paper, we focus on the adaptation of boosting to density function estimation, useful in a number of fields including Natural Language Processing and Computational Biology. Previously, boosting has been used to optimize classification algorithms, improving generalization accuracy by combining many classifiers. The core of the boosting strategy, in the well-known Adaboost algorithm [4], consists in updating the learning instance distribution, increasing (resp. decreasing) the weight of misclassified (resp. correctly classified) examples by the current classifier. Except in [17, 18], few works have attempted to exploit interesting theoretical properties of boosting (such as margin maximization) independently of a classification task. In this paper, we do not take into account classification errors to optimize a classifier, but rather density estimation errors to optimize an estimator (here a probabilistic automaton) of a given target density. Experimental results are presented showing the interest of our approach.

1

Introduction

Most of the machine learning algorithms in supervised learning aim at providing efficient classification rules, often by optimizing the success rate of a given classifier. However, in some other machine learning areas, such as in Natural Language Processing, the main objective rather consists in correctly estimating probability densities over strings. In such a context, the algorithms aim at modelling a target density from a learning sample, in order to assess the occurrence probability of a new instance. This way to proceed is particularly useful in machine learning areas such as shallow parsing, spelling correction, speech recognition [7, 8, 12, 19] and computational biology [2, 10]. Many algorithms are available for estimating these densities from learning data: Hidden Markov Models [7], probabilistic automata [9, 13, 21], Markov Models and their smoothing techniques [5], etc. Recently, some work has dealt with density estimation by combining some of these models [3, 20]. In this paper, we also use such a strategy by introducing a boosting approach to density estimation. Although during the last decade many papers have shown the interest of voting classification algorithms (such as boosting or bagging) (see e.g. [1, 11, 14]), to the best of our knowledge this is the first attempt to use the optimization properties of boosting in such a context. However, we think that this way to proceed T. Elomaa et al. (Eds.): ECML, LNAI 2430, pp. 431–443, 2002. c Springer-Verlag Berlin Heidelberg 2002

432

Franck Thollard et al.

deserves further investigation. Thanks to the margin maximization principle, it not only would allow the optimisation of estimation performance, but also would avoid the tricky use of smoothing techniques, often crucial in density estimation. By combining many weak hypotheses (usually a hypothesis is a classifier), the algorithm Adaboost [4] generates a relevant final weighted classifier. Recently, many theoretical results have justified the relevance of the boosting in machine learning [15]. But so far, its use has been limited to optimize classification tasks, despite recent original extensions to prototype selection [18] and feature selection [17]. Using boosting for estimating density functions is much more difficult than a simple classification optimization. It requires modification of the weight update rule, the core of the Adaboost algorithm, that we recall in section 2. Here, we do not aim to generate an efficient classifier which minimizes the rate of misclassified examples, but rather we aim to automatically and correctly estimate target density function. We show in section 3 that the final combined estimator must then minimize estimation errors (i.e. over and under estimations) regarding the original distribution. For the experimental study, a probabilistic automaton is used during the boosting step as a weak estimator. We present it in section 4. Finally, its performance is tested on a language modelling task, and results are presented in section 5.

2

Properties of Boosting

Boosting consists in combining many (T ) weak hypotheses produced from various distributions Dt (e) over the learning set (LS). The pseudo-code of the original boosting algorithm, called Adaboost [4], is described by the Algorithm 1. At the beginning of the process, each instance e is initially distributed according to an uniform density D1 (e) (note that a given example can of course occur many times in LS, resulting in a higher density at this point). At each stage t, Adaboost decreases (resp. increases) the weight of the training instances, a priori labeled y(e), correctly (resp. incorrectly) classified by the current weak hypothesis ht . Boosting thus forces the weak learner to learn the hardest examples. The weighted combination H(e) of all the weak hypotheses results in a better performing model. Schapire and Singer [16] proved that, in order to minimize the training error, one must seek to minimize Zt (the normalization factor, i.e. the sum of the updates) on each round of boosting. It is easy to show that for minimizing the objective function Zt , the confidence αt of each weak hypothesis t (used in the final combined classifier H) is 12 log( 1− t ). In order to introduce our boosting approach to density estimation, we recall here notations already proposed in Schapire and Singer [16]. Suppose that y(e) ∈ {−1, +1} and that the range of each weak hypothesis ht is restricted to −1, 0, +1.

Boosting Density Function Estimators

433

Algorithm 1: Pseudo-code for AdaBoost. Data : A learning sample LS, a number of iterations T, a weak learner WL Result : An aggregated classifier H Initialize distribution: ∀e ∈ LS, D1 (e) = for t = 2 to T do ht = WL (LS,Dt );

t = Dt (e) ; αt =

e:y(e) =ht (e) 1−t 1 log( ) 2 t

1 |LS|

;

;

Distribution Update Return H s.t. H(e) =

1 T

(

T t=1

/* ∀e ∈ LS, Dt+1 (e) =

Dt (e)e−αt y(e)ht (e) Zt

*/;

αt ht (e)) ;

Let W −1 , W 0 and W +1 be defined by Wb =

Dt (e)

e∈LS:y(e)ht (e)=b

Using symbols + and - for +1 and -1, the following property is then satisfied: W+ + W− + W0 = 1 W + (resp. W − ) describes then the sum of the weights of the correctly (resp. incorrectly) classified instances. W 0 describes the part of the instances unclassified by the current classifier (for example, a point located on a linear separator).

3

Boosting a Density Estimator

In our framework, the weak hypothesis ht is not a classifier which labels an example e, by giving it a negative or positive label (−1 or +1). In Natural Language modelling, the examples are not split into two negative and positive classes. e is usually described by a symbol and its context, i.e. the beginning of the string in which it appears. Hence, ht is now a model which must provide an occurrence probability for a given example e. The objective is then to compare the current inferred distribution Dt with the original density D1 which is the target distribution. D1 is not the uniform distribution anymore (as in Adaboost), but rather the distribution over the data. D1 (e) describes in fact a conditional probability, to observe in the learning sample a symbol given its context. We aim to fit the original distribution D1 and the distribution Dt estimated by ht . In such a context, we cannot use the instance classes (y(e) in Adaboost). The use of the weight update rule according to the correct or incorrect prediction of h(t), is then impossible. To allow this adaptation of boosting to density estimation, we must solve the following problems:

434


1. We must redefine W + and W − which describe respectively the proportion of correctly and incorrectly classified examples in the standard Adaboost. What is now a good or a bad prediction of the weak hypothesis ? 2. Are there examples not modelled by a given weak hypothesis (corresponding to the quantity W 0 in Adaboost) ? 3. We must redefine the weight update rule, taking into account the quality of the current estimator. As to the first problem, we can enumerate three mutually exclusive cases for each learning instance e (see the example described on figure 1): 1. The weak hypothesis ht provides a good estimate of the probability of e. + The weights of such points will be describe by W in our approach, defined + as follows: W = e/Dt (e)=D1 (e) Dt (e) 2. The weak hypothesis ht under-estimates the probability to have e. The weights of such points will be described by W1− , the first part of W − , the weighted sum of instances incorrectly treated by the weak hypothesis. W1− − is then defined as follow: W1 = e/(Dt (e)−D1 (e))0 Dt (e). Contrary to Adaboost, which accepts instances unclassified (those described by W 0 ) by the weak hypothesis, the estimator provides a given density for each learning example, resulting in W 0 = 0. That deals with the second problem. We then handle three quantities, which satisfy the following properties: W + + W1− + W2− = 1 Finally, in order to correct the estimation error of ht , we will increase (resp. decrease) the density of examples under-estimated (resp. over-estimated) by the

Fig. 1. Estimation errors


435

hypothesis. The weight of the correctly estimated examples remains the same. Then, we will use the following general weight update rule: Dt (e).e−αt (Dt (e)−D1 (e)) Zt where Zt is the normalization factor: Zt = e Dt (e).e−αt (Dt (e)−D1 (e)) . Dt+1 (e) =

According to Adaboost, the confidence level αt of the current weak hypothesis can be assessed by minimizing Zt . Actually, minimizing Zt results in minimizing the error rate of the weak hypothesis, since the main part of the Zt quantity is due to misclassified instances. In our adaptation to density estimation, misestimated instances can be either under or over-estimated. Minimizing Zt would attribute more relevance to under-estimated instances than to over-estimated ones. In such a context, a better optimization would consist in minimizing the following objective function, Dt (e).e−αt |Dt (e)−D1 (e)| Zt∗ = e

The confidence αt of the weak hypothesis is determined by minimizing Zt∗ . ∂Zt∗ =− |Dt (e) − D1 (e)|.Dt (e)e−αt |Dt (e)−D1 (e)| ∂αt e Replacing e−αt |Dt (e)−D1 (e)| by its power series, we obtain −(−αt )n |Dt (e) − D1 (e)|n+1 Dt (e) ∂Zt∗ e = ∂αt n! n≥0

Since |Dt (e) − D1 (e)| ∈ [0, 1], we can assume that, for n ≥ 2, |Dt (e) − D1 (e)|n+1 is negligible, and then ∂Zt∗ − |Dt (e) − D1 (e)|.Dt (e) + αt (|Dt (e) − D1 (e)|2 .Dt (e) ∂αt e e t The value of αt for which ∂Z ∂αt = 0 is then |Dt (e) − D1 (e)|.Dt (e) E(δt (e)) = αt = e 2 E(δt (e)2 ) e |Dt (e) − D1 (e)| .Dt (e)

where δt (e) = |Dt (e) − D1 (e)|, E(δt (e)) is its first statistical moment, and E(δt (e)2 ) its second statistical moment. Despite our approximation, we can note that αt keeps the following interesting properties: 1. αt tends to ∞ when the estimated density tends towards the initial distribution. We then highly weight such a good weak hypothesis. 2. αt tends to 0 for estimated densities which do not have overlap with the initial distribution. The pseudo-code of our boosted-algorithm, called PdfBoost, is presented in the Algorithm 2.

436


Algorithm 2: PdfBoost. D1 (e) is the probability to observe e in the learning sample, ∀e ∈ LS for t = 2 to T do Build an automaton ht using Dt−1 ; Get an estimation Dt of the probability density; E(δt (e)) where δt (e) = |Dt (e) − D1 (e)|; Compute the confidence αt = E(δ 2 t (e) ) −αt (Dt (e)−D1 (e))

Update: ∀e ∈ LS: Dt+1 (e) = Dt (e)e /∗Zt is a Normalization Factor∗/;

Zt

;

Return the final model aggregating the T weighted-distributions: 1 D∗ (e) = t

4

αt

(

T

αt Dt (e))

t=1

Probabilistic Automaton as Weak Hypothesis

We recall that boosting can aggregate many weak hypothesis, i.e. many models. This feature induces two kind of constraints on the type of model used: on the one hand, each model must be compact, on the other hand its use must be very efficient. We decided to use, for the experimental study, Probabilistic Deterministic Finite States Automata, since they are, on the one hand, more compact than, say the N-gram model, and on the other hand, the determinism makes them faster than non deterministic probabilistic automata (i.e. Hidden Markov Models) when used in real applications. We present here the formal definition of the model and the inference algorithm. A Probabilistic Finite Automaton (PFA) A is a 7-tuple (Σ, QA , qIA , ξ A ,δ A ,γ A ,F A ) where Σ is the alphabet, i.e. a finite set of symbols QA is the set of states, qIA ∈ Q is the initial state, ξ A ⊂ Q × Σ × Q × (0,1] is a set of probabilistic transitions. F A : QA → [0,1] is the “end of parsing” probabilistic function. Functions δ A and γ A , from Q × Σ to Q and (0, 1] respectivelly are defined as: δ A (qi , σ) = qj iff ∃ p ∈ (0, 1] : (qi , σ, qj , p) ∈ ξ and γ A (qi , σ) = p iff ∃ qj ∈ Q : (qi , σ, qj , p) ∈ ξ. These functions can be trivially extended A to Q × Σ . A We require that for all states q, σ ξ (q, σ, q ) + F (q) = 1. We assume that all states are reachable from the start state with non-zero probability, and that the automaton terminates with probability one. This then defines a distribution over Σ . P rA (x) = γ A (qI , x) × F A (δ(qI , x)) will be the probability of x w.r.t. the automaton A. These automata differ from the one used in many papers in that they define a probability distribution over Σ and not over Σ n with n a constant. Let LS denote a positive sample, i.e. a set of strings belonging to the probabilistic language we are trying to model. Let P T A(LS) denote the prefix tree

Boosting Density Function Estimators a (9/11) 0 (2/11)

b (4/9)

2 (1/4)

1 (1/9) a (4/9)

c (3/4)

437

4 (3/3)

c (3/4) 5 (3/3)

3 (0/4) a (1/4)

3 (1/1)

Fig. 2. PPTA built with LS = {aac,λ, aac, abd, aac, aac, abd, abd, a, ab, λ} acceptor built from a positive sample LS. The prefix tree acceptor is an automaton that only accepts the strings in the sample and in which common prefixes are merged together resulting in a tree-shaped automaton. Let P P T A(LS) denote the probabilistic prefix tree acceptor. It is the probabilistic extension of the P T A(LS) in which each transition has a probability related to the number of times it is used while generating, or equivalently parsing, the positive sample. Let C(q) denote the count of state q, that is, the number of times the state q was used while generating LS from P P T A(LS). Let C(q, #) denote the number of times a string of LS ended on q. Let C(q, a) denote the count of the transition (q, a) in P P T A(LS). The P P T A(LS) is the maximal likelihood estimate built from LS. In particular, for P P T A(LS) the probability estimates are ∧ γ (q, a) = C(q,a) C(q) , a ∈ Σ ∪ {#}. Figure 2 exhibits a P P T A and the learning set it is built from. We now present the second tool used by the generic algorithm: the state merging operation. This operation induces two modifications to the automaton: (i) it modifies the structure (figure 3, left) and (ii) the probability distribution (figure 3, right). It applies to two states. Merging two states can lead to non-determinism. The states that create non-determinism are then recursively merged. When state q results from the merging of the states q and q , the following equality must hold in order to keep an overall consistent model: γ(q, a) =

C(q ,a)+C(q ,a) C(q )+C(q )

, ∀a ∈ Σ ∪ {#}

One can note two properties of the update of the probabilities: (i)

C(q ,a)+C(q ,a) C(q )+C(q )

,a) C(q ,a) is included in [ C(q C(q ) , C(q ) ] which means that the probability of an after merge transition has a value bounded by the two values of the transitions it comes from; (ii) the merge naturally weights more the probability of the transition that holds 20+100 120 20 = 1105 is closer to 1000 than to 100 the more information. For instance, 1000+105 105 . These remarks hold for each pair of transitions that takes part in the merge. Let us merge states qi and qj and define Pqi (resp. Pqj ) as the probability distribution defined by considering state qi (resp. qj ) as the initial state. Since the merge is recursively applied (see figure 3), the probability distribution after merging states qi and qj will be a kind of weighted mean between the distributions Pqi and Pqj .

438


A a 0

a b

2

b

9

b 4

a a

10

0

2

a b

4

2 (12/40) c (p=20/40)

6

0 (0/50)

a b

1 (10/50) a (p=50/50)

Merge of 9 and 4. a

a (p=8/40) a (p=40/50)

3 (20/20)

10

a 6

a (p=98/140)

Mergea of 10 and a 6 0

b

2

b

A’ 4

a

6

3’ (20/20)

0’ (22/140)

c (p=20/140)

Fig. 3. Merging states 5 and 2

We are now in a position to present the MDI algorithm itself for which we recall in the next section the main features. The MDI algorithm [21] (algorithm 3) takes two arguments: the learning set LS and a tuning parameter β. It looks for an automaton, that is the result of a tradeoff between a small size and a small distance to the data. The distance measure used is the Kullback-Leibler divergence. The data are represented by the PPTA as it is the maximum likelihood estimate of the data. While merging two states, the distance between the automaton and the data, in general, increases and, at the same time, the number of states and the number of transitions, in general, decreases. Two states will be set compatible if the impact in terms of divergence of their merge divided by the gain of size is smaller than the parameter β. Algorithm 3: MDI (LS, β). A ← Numbering in Breadth First Order(PPTA); for qi = 1 to Nb State (A) do for qj = 0 to i − 1 do if Compatible (A, qi , qj ) < β then Merge (A,qi ,qj ) ; Return A ;

The MDI algorithm tries to infer a small automaton that is close to the data. In fact, the bigger β, the more general (and small with respect to the number of states) the resulting automaton should be. The parameter β hence controls the level of generalization of the algorithm. Since boosting is known to overfit


439

Table 1. Probability updates, where Di (”to”) = Di (”to|I d like...C”) e D1 C|I’d .. D 1 to| I’d..C 1 Atlanta|I’d..to 1

D2 .4961 .4304 .1682

D3 .8754 .5931 .0910

D4 D5 D1 D2 .9012 1 1 .7407 .9254 .9265 .6061 .5157 .0621 .5640 .2791 .2220

D3 .7864 .5419 .1775

D4 .8155 .6391 .1483

D5 .8532 .6977 .2331

with noisy data, having such a controlling parameter seems to be crucial in this context. Usually, a cross validation protocol is used to assess the right value for the parameter β. The learner is then considered as the algorithm with β estimated on a held-out development set. Another point of view is to consider that each value of β defines a particular algorithm that belongs to a family of learner parameterized by β. Note that the point here is not to optimally define the value of the parameter β, because the algorithm is still a heuristic method. Moreover the boosting just needing a weak learner, we will consider the MDI algorithm, used with a fixed value of the parameter β, as a given weak learner. We will see in the next section that it deserves further investigation.

5

Experimental Issues

We used for this experimental study a very noisy database coming from a language modeling task, i.e. the ATIS task. The Air Travel Information System (ATIS) corpus [6] was developped under a DARPA speech and natural language program that focused on developing language interfaces for information retrieval systems. The corpus consists of speakers of American English making information requests such as: ‘‘I’d like to find the cheapest flight from Washington D C to Atlanta’’. Since the probabilities of the whole sentences are here very small (as a product of all the conditional probabilities), we decided to deal with the conditional probabilities. Since, the PPTA represents the maximum likelihood of the data, it will describe the target density D1 in PdfBoost. At each step t, Dt+1 is not an update of LS, but rather an update of the PPTA transition probabilities. Before studying the PdfBoost behavior on the whole database, we decided to test it on some conditional probabilities to see whether our intuitions are corroborated. The PdfBoost behavior is shown on table 1 where we provide the evolution of some conditional probabilities taken from the example sentence given above (”... D C to Atlanta)”. In this table, Di is the probability provided at iteration i; Di is the probability provided by the first i-aggregated models. As one can see, the probabilities tend rather quickly to the value of the training sample (column D1 in the table). This is true either for the non aggregate models (columns Di ) and for the aggregate ones (columns Di ). One can note

440


that aggregate models converge slower, hopefully leading to less overfit than the models taken alone.

6

Behavior on the Full Task

The criterion used to estimate the quality of a model is the perplexity. Probabilistic models cannot be evaluated by classification error rate, as the fundamental problem has become the estimation of a probability distribution over the set of possible strings. The quality of the model is measured by the per symbol log-likelihood of strings x belonging to a test sample according to the distribution defined by the hypothesis PA (x) computed on a test sample S : |S| |x|

LL = −

1 log P (xji |q i ) S j=1 i=1

where P (xji |q i ) denotes the probability of generating xji , the i-th symbol of the jth string in S, given that the generation process was in state q i . The test sample perplexity P P is most commonly used for evaluating language models of speech applications. It is given by P P = 2LL . The minimal perplexity P P = 1 is reached when the next symbol xji is always predicted with probability 1 from the current state q i (i.e. P (xji |q i ) = 1) while P P = |Σ| corresponds to random guessing from an alphabet of size |Σ|. We aim here to check if the behavior of PdfBoost is coherent with the behavior of the standard AdaBoost algorithm. With the standard algorithm, the classifier gets closer and closer to the training set. Figure 4 shows the trainingset perplexity of the aggregated automata. As expected, the perplexity goes down as the number of iterations grows. It stabilizes at around 100 iterations. This means that the update function chosen is adequate with respect to the perplexity.

Fig. 4. Behavior of the aggregated model on the training set


441

Fig. 5. Behavior of the aggregated model on the development set

Figure 5 shows the behavior of the method on the development set, i.e. its behavior in generalization. As one can see, the two curves are rather different depending on the value of the tuning parameter β. It is interesting to notice that the parameter which performs best on the training set performs worse on the development one. From our point of view, this means that having a parameter that prevents generalization will cause over-fitting. Actually, during the four first boosting steps the development-set perplexity goes down, which shows the interest of our approach. The curves then raise, which, from our point of view, proves over-fitting. We think that tuning the MDI parameter β at each boosting step could prevent over-fitting and thus lead to better results in generalization.

7

Conclusion and Further Work

The preliminary results presented in Figure 5 seem promising to us in that they tend to show that the behavior of PdfBoost is coherent with the one usually observed in the classical boosting. The next step will be to have a complete study of the behavior on the development set, e.g. tuning the inference algorithm at each boosting step in order to prevent overfitting. Since the boosting has been already applied to prototype selection [18], another further work should be to see if there exists a unified boosting that could include the three frameworks known at the moment, i.e. the classic boosting, the boosting applied to protototype selection, and the boosting of density function estimation.

Acknowledgments We wish to thank Alexander Clark for useful remarks on preliminary version.

442


References [1] E. Bauer and R. Kohavi. An empirical comparison of voting classification algorithms: Bagging, boosting, and variants. Machine Learning, 36(1-2):105–139, 1999. 431 [2] P. Bladi, Y. Chavin, and T. Hunkapillerand McClure. Hidden markov models of biological primary sequence information. In National Academy of Science, pages 1059–1063, USA, 1991. 431 [3] Andrew Brown and Geoffrey Hinton. Products of hidden markov models. In T. Jaakkola and T. Richardson, editors, Artificial Intelligence and Statistics, pages 3–11. Morgan Kaufmann, 2001. 431 [4] Y. Freund and R. E. Shapire. A decision theoretic generalization of online learning and an application to boosting. Intl. Journal of Computer and System Sciences, 55(1):119–139, 1997. 431, 432 [5] Joshua Goodman. A bit of progress in language modeling. Technical report, Microsoft Reserach, 2001. 431 [6] L. Hirschman. Multi-site data collection for a spoken language corpus. In DARPA Speech and Natural Language Workshop, pages 7–14, 1992. 439 [7] Frederick Jelinek. Statistical Methods for Speech Recognition. The MIT Press, Cambridge, Massachusetts, 1998. 431 [8] D. Jurafsky and J. H. Martin. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Prentice Hall, Englewood Cliffs, New Jersey, 2000. 431 [9] M. J. Kearns, Y. Mansour, D. Ron, R. Rubinfeld, R. E. Schapire, and L. Sellie. On the learnability of discrete distributions. In Proc. of the 25th Annual ACM Symposium on Theory of Computing, pages 273–282, 1994. 431 [10] Anders Krogh, Michael Brown, I. Saira Mian, Kimmen Sjolander, and David Haussler. Hidden markov models in computational biology: Applications to protein modeling. Journal of Molecular Biology, 235:1501–1531, 1994. 431 [11] R. Maclin and D. Opitz. An empirical evaluation of bagging and boosting. In Proc. of the Fourteenth Natl. Conf. on Artificial Intelligence, pages 546–551, 1997. 431 [12] E. Roche and Yves Schabes. Finite-State Language Processing. MIT Press, 1997. 431 [13] D. Ron, Y. Singer, and N. Tishby. On the learnability and usage of acyclic probabilistic finite automata. In ACM, pages 31–40, Santa Cruz, 1995. COLT’95. 431 [14] E. Tjong Kim Sang. Text chunking by system combination. In CoNLLL-2000 and LLL-2000, pages 151–153, Lisbon, Portugal, 2000. 431 [15] R. E. Schapire, Y. Freund, P. Bartlett, and W. Sun Lee. Boosting the margin: a new explanation for the effectiveness of voting methods. Annals of Statistics, 1998. 432 [16] R. E. Schapire and Y. Singer. Improved boosting algorithms using confidencerated predictions. In Proceedings of the Eleventh Annual Conference on Computational Learning Theory, 1998. 432 [17] M. Sebban and R. Nock. Contribution of boosting in wrapper models. In Proc. of the Thirth European Conf. on Principles and Practice of Knowledge Discovery in Databases, pages 214–222, 1999. 431, 432 [18] M. Sebban, R. Nock, and S. Lallich. Boosting neighborhood-based classifiers. In Proc. of the Seventeenth Intl. Conf. on Machine Learning, 2001. 431, 432, 441


443

[19] A. Stolcke and S. Omohundro. Inducting probabilistic grammars by bayesian model merging. In Lecture Notes in Artifitial Intelligence, editor, Second Intl Collo. on Gramatical Inference, 862, pages 106–118. ICGI-94, 1994. 431 [20] F. Thollard. Improving probabilistic grammatical inference core algorithms with post-processing techniques. In Eighth Intl. Conf. on Machine Learning, pages 561–568, Williams, July 2001. Morgan Kauffman. 431 [21] F. Thollard, P. Dupont, and C. de la Higuera. Probabilistic dfa inference using kullback-leibler divergence and minimality. In Pat Langley, editor, Seventh Intl. Conf. on Machine Learning, San Francisco, June 2000. Morgan Kaufmann. 431, 438

Ranking with Predictive Clustering Trees Ljupˇco Todorovski1, Hendrik Blockeel2 , and Saˇso Dˇzeroski1 1

2

Department of Intelligent Systems, Joˇzef Stefan Institute Jamova 39, SI-1000 Ljubljana, Slovenia {Ljupco.Todorovski,Saso.Dzeroski}@ijs.si Department of Computer Science, Katholieke Universiteit Leuven Celestijnenlaan 200A, B-3001 Heverlee, Belgium {Hendrik.Blockeel}@cs.kuleuven.ac.be

Abstract. A novel class of applications of predictive clustering trees is addressed, namely ranking. Predictive clustering trees, as implemented in Clus, allow for predicting multiple target variables. This approach makes sense especially if the target variables are not independent of each other. This is typically the case in ranking, where the (relative) performance of several approaches on the same task has to be predicted from a given description of the task. We propose to use predictive clustering trees for ranking. As compared to existing ranking approaches which are instance-based, our approach also allows for an explanation of the predicted rankings. We illustrate our approach on the task of ranking machine learning algorithms, where the (relative) performance of the learning algorithms on a dataset has to be predicted from a given dataset description.

1

Introduction

In many cases, running an algorithm on a given task can be time consuming, especially when the algorithm is complex and complex tasks are involved. It is therefore desirable to be able to predict the performance of a given algorithm on a given task from a description (set of properties of the task) and without actually running the algorithm. The term “performance of an algorithm” is often used to denote the quality of the solution provided, the running time of the algorithm or some combination of the two. When several algorithms are available to solve the same type of task, the problem of choosing an appropriate algorithm for the particular task at hand arises. An appropriate algorithm would be an algorithm with a good performance on the given task. Being able to predict the performance of the algorithms, without actually starting them on a given task, will make the problem of choosing easier and less time consuming. We can view performance prediction as a multitarget prediction problem, where the same input (the task description) is used to predict several related targets (the performances of the different algorithms). In this context, it is the relative performance of the different algorithms that matters, and not so much the absolute performance of each of them. We are T. Elomaa et al. (Eds.): ECML, LNAI 2430, pp. 444–455, 2002. c Springer-Verlag Berlin Heidelberg 2002

Ranking with Predictive Clustering Trees

445

thus interested in obtaining an ordering of the algorithms (called also ranking) in terms of their expected relative performance. Within the area of machine learning, many learning algorithms have been developed, especially for classification tasks. A classification task is specified by giving a table of data and indicating the target column: the pair is often referred to as a dataset. The task of predicting the performance of learning algorithms from dataset properties has been addressed within the StatLog project [6], while the task of ranking learning algorithms has been one of the major topics of study of the METAL project [1]. Both are treated as learning problems, where the results of applying selected learning algorithms on selected datasets (baselevel learning) constitute a dataset for meta-level learning. A typical meta-level dataset for ranking thus consists of two parts. The first set of columns (attributes) contains a description of the task at hand. In the case of ranking learning algorithms, it typically contains general and statistical properties of datasets (such as the number of examples and class value and the average kurtosis per numerical attribute). The second set of columns (class values) contains the performance figures for the learning algorithms on the given datasets (e.g., the classification error of C5.0, RIPPER, etc.). Many different variants of ranking have been studied within the METAL project. A prototypical ranker uses a case-based (nearest neighbor) approach. To produce a ranking of the learning algorithms on a new dataset, the most similar datasets from the meta-level dataset are chosen and the performances (rankings) of the algorithms on these datasets are averaged to obtain a prediction of the performance (ranking) on the new dataset [11]. In an alternative approach to ranking, proposed in [2], regression methods are used to estimate the (absolute) performance of each of the learning algorithms on a given task. These individual predictions are then used to obtain the ranking of the algorithms. In this paper, instead of using regression methods for predicting performances of individual algorithms, we propose the use of predictive clustering trees for ranking. In this case, a single predictive clustering tree has the ability to predict performances of all the learning algorithms at once. Thus, in addition to obtaining a ranking, we also obtain an explanation for it. The remainder of this paper is organized as follows. Section 2 describes in more detail the task of ranking of learning algorithms. Section 3 briefly describes predictive clustering trees and describes the particular formulation of the multitarget (relative) performance prediction used in our experiments. Section 4 describes the experimental setup and the results of evaluating our approach to ranking learning algorithms. Finally, Section 5 concludes with a summary and possible directions for future work.

2

Ranking of Learning Algorithms

This section describes in more detail the task of ranking of learning algorithms. This includes the machine learning algorithms ranked, the base-level datasets, the descriptions of the datasets, and the performance evaluation methodology.

446

Ljupˇco Todorovski et al.

Table 1. Eight machine learning algorithms for classification tasks used in our study Acronym c50tree (c50t) c50rules (c50r) c50boost (c50b) ltree (lt) ripper (rip) mlcnb (nb) mlib1 (nn) lindiscr (ld)

2.1

Brief description C5.0 - decision trees based classifier decision rules extracted from a C5.0 tree boosting C5.0 decision trees linear discriminant decision trees decision rules based classifier naive Bayes classifier (MLC++) 1-NN nearest neighbor classifier (MLC++) linear discriminant classifier

The Machine Learning Algorithms and Datasets

In this study, we analyze the relative performance of eight machine learning algorithms for classification tasks. The same set of classifiers and algorithms has been used in the related study of estimating the predictive performance of individual classifiers [2]. The set of algorithms is presented in Table 1: this is a subset of the set of ten algorithms used within the METAL project [1]. Representatives of different classification approaches are included in this set, such as decision trees, decision rules, naive Bayes, nearest neighbor and linear discriminant classifiers. The performance of these eight algorithms has been measured on a set of sixty-five classification tasks (datasets) from the UCI repository [3] and from the METAL project. The list of datasets is given in Table 2. 2.2

Dataset Descriptions

Finding a dataset characterization method that would provide a solid basis for prediction of performance of learning algorithms is probably most important

Table 2. Sixty-five classification datasets used in our study . abalone, acetylation, agaricus-lepiota, allbp, allhyper, allhypo, allrep, australian, balance-scale, bands, breast-cancer-wisconsin, breast-cancer-wisconsin nominal, bupa, car, contraceptive, crx, dermatology, dis, ecoli, flag language, flag religion, flare c, flare c er, flare m, flare m er, flare x, flare x er fluid, german numb, glass, glass2, heart, hepatitis, hypothyroid, ionosphere, iris, kp, led24, led7, lymphography, monk1, monk2, monk3-full, mushrooms, new-thyroid, parity5 5, pima-indiansdiabetes, processed.cleveland 2, processed.cleveland 4, processed.hungarian 2, processed.hungarian 4, processed.switzerland 2, processed.switzerland 4, quisclas, sickeuthyroid, soybean-large, tic-tac-toe, titanic, tumor-LOI, vote, vowel, waveform40, wdbc, wpbc, yeast


447

Table 3. DCT dataset properties DCT nr examples nr num attributes nr classes missvalues total missvalues relative nr sym attributes lines with missvalues total lines with missvalues relative countattr count all value ndiscrimfunct fract cancor meanskew meankurtosis classentropy entropyattributes mutualinformation equivalent nr of attrs minattr multicorrel noisesignalratio avgattr multicorrel sdratio avgattr gini sym maxattr multicorrel maxattr gini sym avgattr relevance minattr gini sym maxattr relevance numattrswithoutliers minattr relevance minattr gfunction maxattr gfunction avgattr gfunction

aspect of meta-learning.1 Several different dataset descriptions have been used for meta-learning. One approach to dataset characterization, proposed within the StatLog project [6], is to use a set of general, statistical and information theory based measures of the dataset. The general properties include properties such as number of examples, classes and (symbolic and numeric) attributes in the dataset. Statistical properties are used to characterize numeric attributes in the dataset and they include measures such as average skewness and kurtosis of numeric attributes. Characteristics of discrete attributes are measured with information theory based measures such as average entropy and average mutual information between discrete attributes and the class. The StatLog approach gave rise to the development of the Data set Characterizing Tool (DCT) [9] within the METAL project. The set of DCT properties extends the initial set of StatLog properties. Table 3 presents the set of DCT properties used in this study. The DCT properties include also properties of the individual attributes in the dataset, such as kurtosis of each numerical attribute or entropy of each symbolic attribute. These properties cannot be directly used in propositional meta-learning, where the dataset description is a fixed-length vector of properties. In order to use the DCT properties of the individual attributes, we have to aggregate them using average, minimum or maximum function. Kalousis and Theoharis [8] have proposed an alternative approach to dataset characterization. They use histograms for fine grained aggregation of the DCT properties of the individual attributes. Histograms, used as an aggregation me1

Note that there is an important constraint on the complexity dataset characterization method. The dataset description should be generated faster than evaluating the performance of the learning algorithms on the dataset. Otherwise, the task of metalevel leering would be meaningless. However, analysis of computational complexity of different dataset description approaches is beyond the scope of this paper, it be found in [7].

448


thod, preserve more information about the DCT properties of the individual attributes compared to the simple aggregation functions of average, minimum and maximum used in the DCT approach. For detailed description of how aggregations based on histograms are calculated see [8]. In this paper, we used the same set of histograms as the one used in [2]. This set includes histograms for four DCT properties of individual attributes and twelve DCT properties of the whole dataset. We refer to the histogram approach to dataset description as HISTO. Finally, in the landmarking approach to dataset description [10], the performances of a set of simple and fast learning algorithms, named landmarkers, are estimated and used as dataset properties. In the original study on using landmarkers for meta-learning, a set of seven landmarkers was proposed. This set includes simple classifiers, such as different versions of a decision node classifier (i.e., a decision tree with a single decision node), naive Bayes, linear discriminant and 1-nearest neighbor. However, three of the landmarkers are already included in the list of classifiers from Table 1 for which we predict the performance. Therefore, in the present study, we use the set of the remaining four landmarkers and we will refer to this approach to dataset description as LAND. 2.3

The Performance of a Learning Algorithm

When building a dataset for meta-learning, we also need an estimate of the performance of the learning algorithms on a given classification task. Most often, the performance of a learning algorithm a on a given classification task d is measured by the predictive error ERR(a, d), i.e., the percentage of incorrectly classified examples. To estimate the predictive error on test examples, unseen during the training of the classifier, a standard ten-fold cross validation method has been used. 2.4

The Performance of Ranking

The performance of ranking is measured by comparing the ranking predicted by the ranking method with the true ranking of the learning algorithms on a given dataset. We used a standard measure of similarity of two rankings, Spearman’s rank correlation coefficient [11]: n ( i=1 Di 2 ) , (1) rs = 1 − 6 n3 − n where Di is the difference between actual and predicted rank of the i’th algorithm and n is the number of learning algorithms. Again, to estimate the performance of ranker on test datasets, unseen during the training of the ranker, a standard ten-fold cross validation method has been used.


3

449


This section first briefly describes predictive clustering trees. It then discusses how they could be used to predict the errors of different learning algorithm on a given dataset simultaneously. It finally proposes to use the ranks calculated from the errors as the target variables, rather than the errors themselves. 3.1

Predictive Clustering Trees

Decision trees are most often used in the context of classification or single-target regression; i.e., they represent a model in which the value of a single variable is predicted. However, as a decision tree naturally identifies partitions of the data (course-grained at the top of the tree, fine-grained at the bottom), one can also consider a tree as a hierarchy of clusters. A good cluster hierarchy is one in which individuals that are in the same cluster are also similar with respect to a number of observable properties. This leads to a simple method for building trees that allow the prediction of multiple target attributes at once. If we can define a distance measure on tuples of target variable values, we can build decision trees for multi-target prediction. The standard TDIDT algorithm can be used: as a heuristic for selecting tests to include in the tree, we use the minimization of intra-cluster variance (and maximization of inter-cluster variance) in the created clustering. A detailed description of the algorithm can be found in [4]. An implementation is publicly available in the first-order learner Tilde that is included in the ACE tool [5]; however for this paper we have used Clus, a downgrade of Tilde that works only on propositional data. 3.2

Ranking via Predicting Errors

The instance-based approaches to ranking predict rankings of algorithms on a dataset by predicting the errors of the algorithms on the dataset, then creating a ranking from these [11]. An instance here consists of a description of a dataset, plus the performance of eight different algorithms on that dataset. Based on these eight target values, an example can be positioned in an eight-dimensional space. In its standard mode of operation, Clus builds its trees Nso that the2 intra¯ ) where cluster variance is minimized, where variance is defined as j=1 d(xj , x ¯ is the mean vector of the cluster, xj is an element of the cluster, N is the x number of elements in the cluster, and d represents the euclidean distance. So, what Clus does is trying to create clusters in such a way that a given algorithm will perform similarly on all datasets in that cluster. Note that this is different from what we want: creating clusters in which several algorithms have the same relative performance. To illustrate this, suppose we have four algorithms which on two datasets score the following errors: {(0.1, 0.2, 0.3, 0.4), (0.5, 0.6, 0.7, 0.8)}

450


Clearly the relative performance of the four algorithms is exactly same on the three datasets, so they belong to the same cluster. However, the variance in this cluster is relatively large. Compare this to {(0.1, 0.2, 0.3, 0.4), (0.4, 0.3, 0.2, 0.1)} which has a smaller variance than the previous cluster but is clearly worse: the relative performances are opposite. 3.3

Ranking Trees

A solution for this problem is to first rank the algorithms and to predict these ranks instead of the errors themselves. In this way, we obtain ranking trees. A ranking tree has leaves in which a ranking of the performance of different algorithms is predicted. This transformation removes fluctuations in the variance that are caused by differences in absolute rather than relative performance. Moreover, given the formula for the Spearman’s rank correlation coefficient (1), it is clear that a linear relationship between variance and expected Spearman correlation exists. ¯ )2 Indeed, note that in the case when the ranks are predicted, the variance d(xi , x n 2 equals i=1 Di from the formula (1). This is true under an assumption that the exact ranking number of each algorithm is predicted. This assumption is not fulfilled. Instead of predicting exact ranks, the clustering tree predicts only approximations of rank numbers, e.g.: (6.0, 6.4, 3.65, 6.1, 5.65, 3.5, 5.65, 3.7). Of course, by comparing these approximations we can easily obtain the following exact ranking: (6, 8, 2, 7, 4.5, 1, 4.5, 3) of the eight algorithm. However, the aforementioned equivalence of variance and Spearman’s correlation coefficient does not hold anymore. Thus, minimizing intra-cluster variance should be seen as an approximation to maximizing Spearman’s correlation coefficient. Note, however, that this approximation is far better than minimizing intra-cluster variance based on the error rates themselves.

4

Experiments

Our experiments investigate the performance of ranking with predictive clustering trees induced using the three different dataset characterization approaches presented in Section 2. Following the discussion from Section 3.3, we transformed the target error values into ranks. The remainder of this section first describes the experimental setup. It then presents the experimental results, including an example ranking tree and performance figures on the correlation between actual and predicted rankings.


4.1

451

Experimental Setup

Clus was run several times with the same system settings, but on different datasets that vary along two dimensions: – Language bias: DCT, HISTO, LAND, DEF – Targets: errors, ranks The first three language bias settings correspond to the three dataset characterization approaches described in Section 2. DCT uses set of properties of the whole dataset and aggregations of the properties of the individual attributes. HISTO uses more sophisticated aggregation method of histograms for aggregating the properties of the individual attributes. LAND uses estimated performances of four landmarkers for dataset description. Finally, DEF uses no information at all to induce a “default” model tree that consists of a single leaf (i.e., a model that just predicts the average of the performances encountered in the training set). For the first three cases, tests in the constructed tree are always of the form A < c, where A is a numerical attribute from one of the DCT, HISTO or LAND datasets (note that all meta-level attributes are numeric) and c some value for it (any value from A’s domain was allowed). The target values were either the errors themselves, which allows us to compare some results directly with [2], or the ranks, which (according to our explanation in Section 3.3) we hope to yield better results w.r.t. the Spearman’s rank correlation coefficient. Our evaluation is based on a ten-fold cross validation, in order to maximize comparability with the results in [2]. Unfortunately we could not use exactly the same cross validation folds. The Clus system has a number of parameters with which it can be tuned. One parameter that influences the results quite strongly is the so-called “ftest” parameter. Clus uses a stopping criterion that is based on a statistical F-test (the standard way to test whether the average intra-cluster variance after a split is significantly smaller than the original variance); the “significance level” at which this test is performed is the “ftest” parameter. Values close to 0 cause Clus to quickly stop adding nodes to the tree, yielding small trees; the value 1 yields trees of maximal size. Preliminary experiments with Clus and Tilde on this and similar datasets indicated that ftest=1 yielded best results. Therefore we adopted this setting for all the experiments described here. Except for this ftest parameter, the default values were used for all parameters. 4.2


Table 4 shows the mean absolute deviations of the predicted error rates from the true ones for different learning algorithms. The left-hand side (Clus - ranks) gives results of clustering trees: these are compared to the results in the righthand side taken from [2]. We can see that on average, predictive clustering trees score approximately equally good as the Kernel or Cubist methods on DCT

452


and HISTO meta-level datasets. Clustering trees perform worse on the LAND dataset. This is due to the fact that we decided to use a set of landmarkers that are disjoint with the set of target classifiers. In [2] seven landmarkers have been used, three of them being the same as the target classifiers. However, having meta-level attributes (landmarkers) that are the same to the meta-level class to be predicted (target classifiers) makes the task of predicting their performance trivial. Thus, the results on the LAND dataset are hard to compare. Note, however, that a single predictive clustering tree predicting the performance of all the learning algorithms at once has a very important advantage over the set of eight regression trees for predicting the performance of individual algorithms. A clustering tree provides a single model that can be easily interpreted. While the above MAD values are useful to compare our approach with previous approaches, our ultimate criterion is the Spearman’s rank correlation coefficient between the predicted ranking and the actual ranking of the methods. Spearman correlations are shown in Table 5.

Table 4. Mean absolute deviations (MADs) for a single predictive clustering tree (predicting all the error rates at once) induced with Clus compared to the MADs of a set of regression trees (one for each learning algorithm) induced with Kernel and Cubist methods. The Kernel and Cubist results are taken from [2]. Note that LAND and LAND* meta-level datasets are different Classifier c50boost c50rules c50tree lindiscr ltree mlcib1 mlcnb ripper

Clus - errors DCT HISTO LAND DEF 0.105 0.114 0.139 0.136 0.100 0.110 0.136 0.135 0.101 0.109 0.137 0.139 0.119 0.124 0.126 0.139 0.106 0.107 0.123 0.134 0.120 0.124 0.144 0.155 0.124 0.135 0.145 0.149 0.135 0.114 0.138 0.147

DCT 0.112 0.110 0.110 0.118 0.105 0.120 0.121 0.113

Kernel HISTO LAND* DCT 0.123 0.050 0.103 0.121 0.051 0.121 0.123 0.054 0.114 0.129 0.063 0.118 0.113 0.041 0.114 0.138 0.081 0.150 0.143 0.064 0.126 0.128 0.056 0.128

Cubist HISTO LAND* 0.128 0.033 0.126 0.036 0.130 0.044 0.140 0.054 0.121 0.032 0.149 0.067 0.149 0.044 0.131 0.041

Table 5. Spearman’s rank correlation coefficients (SRCCs) for the predictive clustering trees (predicting error rates and rankings) approach compared to SRCCs of other ranking approaches. Results for Cubist, Kernel and Zooming are taken from [2] Clus regression trees ranks errors Kernel Cubist Zooming DEF 0.372 0.349 0.330 0.330 0.330 0.399 0.380 0.435 0.083 0.341 DCT 0.174 0.371 HISTO 0.429 0.426 0.405 ∗ ∗ 0.266 0.197 0.090 0.190 LAND


453

A first observation is that for each meta-level dataset, ranking trees built from ranks score better than ranking trees built directly from error rates. This corresponds with our intuition, explained in Section 3.3. Furthermore, both clustering trees approaches have better scores than all the others, except for the kernel method with the DCT dataset, which has also the highest overall value. These experimental results provide support for the two effects we identified earlier as possibly positively influencing the results. First, predictive clustering trees capture dependencies between different algorithms better than separate predictive models for each algorithm can. Second, when using intra-cluster variance minimization as a heuristic, it is better to first convert values into ranks. We conclude this discussion with an example tree. Table 6 shows a predictive clustering tree induced on the DCT dataset with ranks as target values. Each leaf node in the tree predicts a ranking of the eight algorithms from Table 1. For example, first leaf node in the tree (marked with (*)) predicts that c50boost (c50b) will perform better than c50rules (c50r) that will perform better than ltree (lt) and so on. The tree indicates that the number of attributes with outliers is most influential for the ranking of the algorithms. It also indicates that the two properties of number of symbolic and numeric attributes in the dataset seem to have good predictive power. Further interpretation and analysis of the tree is possible but it is beyond the scope of this paper.

5

Summary and Further Work

We have used predictive clustering trees to rank (predict the relative performance of) classification algorithms according to the performance on a given dataset using dataset properties. Three different dataset descriptions were used. Two different tasks were considered: predicting actual performances and predicting relative performances (ranking). On the first task of predicting the performance of classifiers, a single clustering tree predicting performances of all classifiers at once performs as well as a set of regression trees, each of them predicting performances of an individual classifier. However, the important advantage of the clustering trees approach is that it provides a single interpretable model. On the second task of predicting ranking, the experimental results show that using ranks as target variables in clustering trees works better than using actual performances for all dataset descriptions. Ranking with a single clustering tree performs better than ranking with a set of regression trees for two out of three dataset description approaches. Finally, ranking with clustering trees outperforms also instance-based approach of Zooming. An immediate direction for further work is to extend our ranking approach to work with relational dataset descriptions, similar to the one presented in [12]. Following the relational approach, properties of individual attributes can be included in the dataset description without being aggregated using mean, maximal and minimal values or histograms. This can be easily done, due to the fact that

454


Table 6. An example ranking tree (see Table 1 for the legend of the algorithms’ acronyms. Note that symbol < in the leaves denotes “performs better than” NumAttrsWithOutliers > 3 +-yes: AVGAttr_gFunction > -1.236 | +-yes: ClassEntropy > 0.977 | | +-yes: ClassEntropy > 0.999 | | | +-yes: c50b

Machine Learning: ECML 2002: 13th European Conference on Machine Learning, Helsinki, Finland, August 19-23, 2002. Proceedings

Machine Learning: ECML-94

Machine Learning: ECML-98: 10th European Conference on Machine Learning, Chemnitz, Germany, April 21-23, 1998, Proceedings