Recent advances in simulated evolution and learning

Recent Advances In Simulated Evolution and Learning Advances In Natural Computation - Vol. 2

ipo vvang

Recent Advances in Simulated Evolution and Learning

™'

ADVANCES IN NATURAL COMPUTATION Series Editor:

Xin Yao (University of Birmingham, UK)

Assoc. Editors: Hans-Paul Schwefel (University of Dortmund, Germany) Byoung-Tak Zhang (Seoul National University, South Korea) Martyn Amos (University of Liverpool, UK)

Vol. 1:

Applications of Multi-Objective Evolutionary Algorithms edited by Carlos A. Coello Coello (CINVESTAV-IPN, Mexico) and Gary B. Lamont (Air Force Institute of Technology, USA)

Advances

In

Natural

Computation

-

Vo

Recent Advances in • Simulated Evolution and Learning

editors

Kay Chen Tan National University of Singapore, Singapore

Meng Hiot Lim Nanyang Technological University, Singapore

Xin Yao University of Birmingham, UK

Lipo Wang Nanyang Technological University, Singapore

\ ^ 3 World Scientific NEW JERSEY • LONDON • SINGAPORE • BEIJING • SHANGHAI • HONGKONG * TAIPE

Published by World Scientific Publishing Co. Pte. Ltd. 5 Toh Tuck Link, Singapore 596224 USA office: Suite 202,1060 Main Street, River Edge, NJ 07661 UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE

British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library.

RECENT ADVANCES IN SIMULATED EVOLUTION AND LEARNING Advances in Natural Computation — Vol. 2 Copyright © 2004 by World Scientific Publishing Co. Pte. Ltd. All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system now known or to be invented, without written permission from the Publisher.

For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher.

ISBN 981-238-952-0

Printed in Singapore by World Scientific Printers (S) Pte Ltd

PREFACE

Inspired by the Darwinian framework of evolution through natural selection and adaptation, the field of evolutionary computation has been growing very rapidly, and is involved today in many diverse application areas. Evolutionary computation encompasses a wide range of adaptive and computational algorithms, methods and techniques that are inspired by natural evolution: e.g., genetic algorithms, evolutionary programming, evolution strategies, genetic programming and related artificial life strategies. Such simulated evolution and learning techniques offer advantages of simplicity, ease of interfacing with existing techniques, and extensibility in finding good solutions efficiently to complex practical problems. This volume contains substantially extended and revised papers selected from the 4th Asia-Pacific Conference on Simulated Evolution and Learning (SEAL'2002), 18-22 November 2002, Singapore. SEAL'2002 received a total of 230 submissions with 5 special sessions featuring various applications of evolutionary computation. After extensive reviews by the technical committee, 139 papers were accepted for oral presentation and 25 for poster presentation. Among the accepted papers, 43 papers were invited to be extended and revised for inclusion in this volume. The double review processes have ensured a volume of the highest quality. We hope the readers will enjoy it. The papers included in this volume cover the latest advances in the theories, algorithms, and applications of simulated evolution and learning techniques. It also highlights future research directions in the field. The volume is organized into two broad categories, e.g., evolutionary computation theory and evolutionary computation applications. The first category, comprising Chapters 1 through 18, provides insights to different evolutionary computation techniques. The second category, Chapters 19 through 43, presents many practical applications of evolutionary computation techniques, such as scheduling, control and power systems, robotics, signal processing, data mining, and bioinformatics.

VI

Preface

This volume will be of significant interest and value to all postgraduates, research scientists and practitioners dealing with evolutionary computation or complex real-world problems. We hope that this volume will motivate researchers and practitioners to extend the presented results of evolutionary computation and to broaden their implementations in practice. The editors thank authors of all the chapters for their excellent contributions to this volume. Without their valuable work, this volume would not have been possible.

EDITORS Kay Chen Tan, Meng Hiot Lim, Xin Yao and Lipo Wang

TABLE OF CONTENTS

Preface

v

PART 1: Evolutionary Theory Chapter 1: Co-Evolutionary Learning in Strategic Environments (Akira Namatame; Naoto Sato; Kazuyuki Murakami)

1

Chapter 2: Using Evolution to Learn User Preferences (Supiya Ujjin; Peter J. Bentley)

20

Chapter 3: A Parallel Genetic Algorithm for Clustering (Juha Kivijarvi; Joonas Lehtinen; Olli S. Nevalainen)

41

Chapter 4: Using SDVID Genetic Programming for Fault-Tolerant Trading Strategies (Nils Svangard; Peter Nordin; Stefan Lloyd)

61

Chapter 5: An Efficient Coevolutionary Algorithm Based on Merging and Splitting of Species 78 (Myung Won Kim; Soungjin Park; Joung Woo Ryu) Chapter 6: Schema Analysis of Genetic Algorithms on Multiplicative Landscape 93 (Hiroshi Furutani) Chapter 7: Evolutionary Learning Strategies for Artificial Life Characters 112 (Marcio Lobo Netto; Henrique Schiitzer Del Nero; Claudio Ranieri)

vii

viii

Table of

Contents

Chapter 8: Adaptive Strategy for GA Inspired by Biological Evolution (Hidefumi Sawai; Susumu Adachi)

132

Chapter 9: The Influence of Stochastic Quality Functions on Evolutionary Search (Bernhard Sendhoff; Hans-Georg Beyer; Markus Olhofer)

152

Chapter 10: Theoretical Analysis of the GA Performance with a Multiplicative Royal Road Function (Hideaki Suzuki; Hidefumi Sawai)

173

Chapter 11: A Real-Coded Cellular Genetic Algorithm Inspired by Predator-Prey Interactions (Xiaodong Li; Stuart Sutherland)

191

Chapter 12: Observed Dynamics of Large Scale Parallel Evolutionary Algorithms with Implications for Protein Engineering 208 (Martin Oates; David Corne; Douglas Kell) Chapter 13: Using Edge Histogram Models to Solve Flow Shop Scheduling Problems with Probabilistic Model-Building Genetic Algorithms (Shigeyoshi Tsutsui; Mitsunori Miki)

230

Chapter 14: Collective Movements of Mobile Robots with Behavior Models of a Fish 250 (Tatsuro Shinchi; Tetsuro Kitazoe; Masayoshi Tabuse; Hisao Ide; Takahiro Horita) Chapter 15: Automatic Modularization with Speciated Neural Network Ensemble 268 (Vineet R. Khare; Xin Yao)

Table of Contents

Chapter 16: Search Engine Development using Evolutionary Computation Methodologies (Reginald L. Walker)

ix

284

Chapter 17: Evaluating Evolutionary Multi-Objective Optimization Algorithms using Running Performance Metrics 307 (Kalyanmoy Deb; Sachin Jain) Chapter 18: Visualization Technique for Analyzing Non-Dominant Pareto Optimality 327 (Kiam Heong Ang; Gregory Chong; Yun Li)

PART 2: Evolutionary Applications Chapter 19: Image Classification using Particle Swarm Optimization (Mahamed G. Omran; Andries P. Engelbrecht; Ayed Salman) 347 Chapter 20: A Coevolutionary Genetic Search for a Layout Problem (Thomas Dunker; Engelbert Westkamper; Giinter Radons) 366 Chapter 21: Sensitivity Analysis in Multi-Objective Evolutionary Design (Johan Andersson) 386 Chapter 22: Integrated Production and Transportation Scheduling in Supply Chain Optimisation 406 (Gang Wu; Chee Kheong Siew) Chapter 23: Evolution of Fuzzy Rule Based Controllers for Dynamic Environments 426 (Jeff Riley; Vic Ciesielski)

X

Table of

Contents

Chapter 24: Applications of Evolution Algorithms to the Synthesis of Single/Dual-Rail Mixed PTL/Static Logic for Low-Power Applications 446 (Geun Rae Cho; Tom Chen) Chapter 25: Evolutionary Multi-Objective Robotics: Evolving a Physically Simulated Quadruped using the PDE Algorithm (Jason Teo; Hussein A. Abbass)

466

Chapter 26: Applying Bayesian Networks in Practical Customer Satisfaction Studies (Waldemar Jaroriski; Josee Bloemer; Koen Vanhoof; Geert Wets)

486

Chapter 27: An Adaptive Length Chromosome Hyper-Heuristic Genetic Algorithm for a Trainer Scheduling Problem 506 (Limin Han; Graham Kendall; Peter Cowling) Chapter 28: Design Optimization of Permanent Magnet Synchronous Machine using Genetic Algorithms 526 (R.K. Gupta; Itsuya Muta; G. Gouthaman; B. Bhattacharjee) Chapter 29: A Genetic Algorithm for Joint Optimization of Spare Capacity and Delay in Self-Healing Network (Sam Kwong; H.W. Chong)

542

Chapter 30: Optimization of DS-CDMA Code Sequences for Wireless Systems 562 (Sam Kwong; Alex C.H. Ho) Chapter 31: An Efficient Evolutionary Algorithm for Multicast Routing with Multiple QoS Constraints 581 (Abolfazl T. Haghighat; Karim Faez; Mehdi Dehghan) Chapter 32: Constrained Optimization of Multilayered Anti-Reflection Coatings using Genetic Algorithms 603 (Kai-Yew Lum; Pierre-Marie Jacquart; Mourad Sefrioui)

Table of Contents

xi

Chapter 33: Sequential Construction of Features Based on Genetically Transformed Data 623 (Jacek Jelonek, Roman Slowiriski, Robert Susmaga) Chapter 34: Refrigerant Leak Prediction in Supermarkets using Evolved Neural Networks 643 (Dan W. Taylor; David W. Corne) Chapter 35: Worst-Case Instances and Lower Bounds via Genetic Algorithms (Matthew P. Johnson; Andrew P. Kosoresow)

662

Chapter 36: Prediction of Protein Secondary Structure by Multi-Modal Neural Network 682 (Hanxi Zhu; Ikuo Yoshihara; Kunihito Yamamori; Moritoshi Yasunaga) Chapter 37: Joint Attention in the Mimetic Context — What is a "Mimetic Same"? 698 (Takayuki Shiose; Kenichi Kagawa; An Min; Toshiharu Taura; Hiroshi Kawakami; Osamu Katai) Chapter 38: Autonomous Symbol Acquisition Through Agent Communication (A. Wada; K. Takadama; K. Shimohara; O. Katai)

711

Chapter 39: Search of Steady-State Genetic Algorithms for VisionBased Mobile Robots 729 (Naoyuki Kubota; Masayuki Kanemaki) Chapter 40: Time Series Forecast with Elman Neural Networks and Genetic Algorithms 747 (LiXin Xu; Zhao Yang Dong; Arthur Tay)

Xll

Table of

Contents

Chapter 41: Co-Adaptation to Facilitate Naturalistic Human Involvement in Shared Control System (Yukio Horiguchi; Tetsuo Sawaragi)

769

Chapter 42: Distributed Evolutionary Strategies for Searching Oligo Sets of Yeast Genome 789 (Arthur Tay; Kay Chen Tan; Ji Cai; Huck Hui Ng) Chapter 43: Duration-Dependent Multi-Schedule Evolutionary Curriculum Timetabling (Chee Keong Chan; Hoay Beng Gooi; Meng Hiot Lim)

803

PART 1 EVOLUTIONARY THEORY

CHAPTER 1 CO-EVOLUTIONARY LEARNING IN STRATEGIC ENVIRONMENTS

Akira Namatame, Naoto Sato and Kazuyuki Murakami Dept. of Computer Science National Defense Academy, Yokosuka, JAPAN E-mail: [email protected] An interesting problem is under what circumstances will a collection of interacting agents realize efficient collective actions. This question will depend crucially on how self-interested agents interact and how they learn from each other. We model strategic interactions as dilemma games, coordination games or hawk-dove games. It is well known that the replicator dynamics based on natural selection converge to an inefficient equilibrium. In this chapter, we focus on the effect of coevolutionary learning. Each agent is modeled to learn interaction rules defined as the function of own strategy and the strategy of the neighbor. We show that a collection of interacting agents converges into equilibrium in which the conditions of efficiency and equity are satisfied. We investigate interaction rules acquired by all agents and show that they share several rules with the common features to sustain equitable social efficiency. This chapter also presents a comparative study of two evolving populations, one in a spatial environment, and the other in a small-world environment. The effect of the environment on the emergence of social efficiency is studied. The small-world environment is shown to encourage the emergence of social efficiency further than the spatial structure.

1. Introduction In many applications it is of interest to know which strategies can survive in the long run. While the concept and techniques of game theory have

1

2

A. Namatame,

N. Sato and K.

Murakami

been used extensively in many diverse contexts, they have been unsuccessful in explaining how agents realize if a game has many equilibria8. Introspective or educative theories that attempt to explain equilibrium selection problem directly at the individual decision-making level impose very strong informational assumptions. The game theory is also not able to address issues on how agents know which equilibrium should be realized when games have multiple equally plausible equilibria3. The game theory is also not able to provide answer in explaining how agents should behave in order to overcome an inefficient equilibrium situation7. One of the variations involves the finitely iterated games. The standard interpretation of game theory is that the game is played exactly once between fully rational individuals who know all details of the game, including each other's preferences over outcomes. Evolutionary game theory, instead, assumes that the game is repeated many times by individuals who are randomly drawn from large populations1518. An evolutionary selection process operates over time on the population distribution of behaviors. It is also of interest to know which strategies can survive in the long run. According to the fundamental theorem principle of natural selection, the fitter behavior is selected. The evolutionary dynamic model with the assumption of uniform matching can be analyzed using replicator dynamics4. The criterion of evolutionary equilibrium highlights the role of mutations. The replicator dynamics, highlight the role of selection. Evolutionary game theory assumes that the game is repeated by individuals who are randomly drawn from large populations6. However, the growing literatures on evolutionary models have not considered learning at individual levels9. They treat agents as automata, merely responding to changing environments without deliberating about individuals' decisions. Within the scope of our model, we treat models in which agents make deliberate decisions by applying rational reasoning about what to do and also how to decide13'14. Two features of this approach distinguish it from the introspective approach. First, agents are not assumed to be so rational or knowledgeable as to correctly guess or anticipate the other agent's strategies. Second, an explicit dynamic process is specified describing how agents adapt their strategies as they repeat the games.

Co-Evolutionary

Learning in Strategic

Environments

3

An interesting problem is under what circumstances agents with individual learning may converge to some particular equilibrium1'2. We endow our agents with some simple way of learning and describe the evolutionary dynamics that magnifies tendencies toward better situation. By incorporating a consideration of how agents interact into models we not only make them more realistic but we also enrich the types of aggregate behavior that can emerge10'1112. It is an important question to answer the following question: how the society groups its way towards an efficient equilibrium in an imperfect world when self-interested agents learn from each other. The term evolutionary dynamics often refers to systems that exhibit a time evolution in which the character of the dynamics may change due to internal mechanisms. In this chapter, we focus on evolutionary dynamics that may change in time according to certain local rules. Evolutionary models can be characterized both by the level at which the mechanisms are working and the dimensionality of the system. We use the evolutionary models based on microscopic individuals who interact locally16. The search for evolutionary foundations of game-theoretic solution concepts leads from the notion of an evolutionarily stable strategy to alternative notions of evolutionary stability to dynamic models of evolutionary processes. The commonly used technique of modeling the evolutionary process as a system of a deterministic difference or differential equations may tell us little about equilibrium concepts other than that strict Nash equilibrium are good. We can attempt to probe deeper into these issues by modeling the choices made by agents with their learning models. We focus on collaborative learning in strategic environments. Noncooperative games are classified into dilemma games coordination games, and hawk-dove games. It is well known that natural selection does leads to inefficient equilibria in these games. In this chapter each agent learns interaction rules by repeating games. We provide a general class of adaptation models and relate their asymptotic behavior to equilibrium concepts. We assume agents behave myopically, and they evolve their interaction rule over generation. They learn from the most successful strategy of their neighbor. Hence their success depends in large part on how well they do in their interactions with their neighbors.

4

A. Namatame,

N. Sato and K.

Murakami

If the neighbor is doing well, the rule of the neighbor can be imitated, and in this way successful rule can spread throughout a population, from neighbor to neighbor. We consider two fundamental models of interaction, local interaction with the lattice model and small-world model17. We show that all agents mutually learn acquire the common-rule, which lead to social efficiency. We also investigate acquisition by rules, and show that those rules of agents are categorized into a few rules with some commonality. 2. Interaction with Lattice Model and Small-World Networks It is important to consider with whom an agent interacts and how each agent decides his action depending on others' actions. In order to describe the interactions among agents, we may have two fundamental models, random matching and local matching9. The approach of random (or uniform) matching is modeled as follows: In each time period, every agent is assumed to match (interact) with one agent drawn at random from a population. An important assumption of the random matching is that they receive knowledge of the current strategy distribution. Each agent makes his rational decision strategy based on a sample of information about what other agents have done in the previous time period. Agents are able to calculate best replies and learn the strategy distribution of play in society. Agents may adapt based on the aggregate information representing the current status of the whole system (global adaptation). In this case, each agent chooses an optimal decision based on aggregate information about how all other agents behaved in the past. An agent calculates her reward and plays her best response strategy. An important assumption of global adaptation is that they receive knowledge of the aggregate. In many situations, however, agents are not knowledgeable so as to correctly guess or anticipate other agents' actions, or they are less sophisticated and that they do not know how to calculate best replies8. We assume that a spatial environment is a more realistic representation since interactions in real life rarely happen on such a macro scale; Spatial interaction is generally achieved through the

Co-Evolutionary

Learning in Strategic

Environments

5

use of a 2D grid as shown in Fig. 1(a), with each agent inhabiting a cell on the grid. Interaction between agents is restricted to neighboring cells. This may allow for individuals, which may have been eliminated if assessed against all players, to survive in a niche. The recognition of the importance of spatial interactions has led to many exploring and extending aspects of it. Nowak and May focused upon evolutionary niching and the pattern of emergence of cooperation in the spatial environment13. With local adaptation each agent is modeled to adapt to his neighbors. The hypothesis of local adaptation also reflects limited ability of agents' parts to receive, decide, and act based upon information they receive in the course of interaction. Agents observe the current performance of their neighbors, and learn from the most successful agent. Agents are less sophisticated in that they do not know how to calculate best replies and are using other agent's successful strategies as guides for their own choices. Each agent interacts with the agents on all eight adjacent squares and imitates the strategy of any better performing one. In each generation, each agent attains a success score measured by its average performance with its eight neighbors. Then if an agent has one or more neighbors who are more successful, the agent converts to the rule of the most successful neighbor. Complex networks describe a wide range of systems in nature and technology. They can be modeled as a network of nodes where the interactions between nodes are represented as edges. Recent advances in understanding these networks revealed that many of the systems show a small-world structure. Watts and Storogatz introduced a small-world network which transforms from a nearest neighbor coupled system to a random coupled network by rewiring the links between the nodes17. Two parameters are used to describe the transition. The mean path length L, which specifies the global property of the network, is given as the mean of the shortest path between all pairs of vertices. In contrast, the clustering coefficient C characterizes the local property of the system and can be calculated as the fraction of the connections between the neighbors of a node divided by the number of edges of a globally coupled neighborhood, averaged over all vertices. Consider one lattice model in which each node is coupled with its nearest neighbors as shown in Fig. 1(b). It has a large mean path length

6

A. Namatame, N. Sato and K. Murakami

and a high clustering coefficient. If one rewires the links between the node with a small probability the local structure of the network remains almost conserved keeping the clustering coefficient contrast. In contrast, due to the introduction of short cuts by the rewiring procedure the mean path length becomes strongly reduced. Networks with these properties are small-world networks. Further increase of the rewiring probability results in a random coupled network with a short mean path length and a low clustering coefficient.

(a) Local interaction with a lattice model

(b) Interaction with a small-world network (Illustration of one-lattice model)

Fig. 1. The topology of interaction

3. Learning Models Game theory is typically based upon the assumption of a rational choice8. In our view, the reason for the dominance of the rational-choice approach is not that scholars think it to be realistic. Nor is game theory used solely because it offers good advice to a decision maker, because its unrealistic assumptions undermine much of its value as a basis for advice. The real advantage of the rational-choice assumption is that it often allows deduction. The main alternative to the assumption of rational choice is some form of adaptive behavior. The adaptation may be at the individual level through learning, or it may be at the population level through differential survival and reproduction of the more successful individuals. Either way, the consequences of adaptive processes are often very hard to deduce when there are many interacting agents following rules that have

Co-Evolutionary

Learning in Strategic

Environments

7

nonlinear effects. Among many adaptive mechanisms that have been discussed in the literature on learning are classified as follows: (1) Best response learning In most game theoretic models, agents have perfect knowledge of the consequences of their decision. An important assumption of bestresponse learning is that they receive knowledge of the current strategy distribution. Agents can calculate their best strategy based on information about what other agents have done in the past. Then agents gradually learn the strategy distribution in the society. Agents adopt actions that optimize their expected payoff given what they expect others to do. In this learning model, agents choose the best replies to the empirical frequencies distribution of the previous actions of the others. (2) Reinforcement learning Agents tend to adopt actions that yielded a higher payoff in the past, and to avoid actions that yielded a low payoff. Payoff describes choice behavior, but it is one's own past payoffs that matter, not the payoffs of the others. The basic premise is that the probability of taking an action in the present increases with the payoff that resulted from taking that action in the past. (3) Evolutionary learning Agents with higher payoff are at a productive advantage compared to agents who use low-payoff strategies, hence the latter decrease in frequency in the population over time (natural selection). In the standard model of this situation agents are viewed as being genetically coded with a strategy and selection pressure favors agents that are fitter, i.e., whose strategy yields a higher payoff against the population. The idea of using a genetic algorithm (GA) to create strategies has been developed further by Lindgren9. He showed that strategies could be made more robust by seeding the initial population with expert, hand-coded strategies. (4) Social learning Agents learn from each other with social learning. For instance, agents may copy the behavior of others, especially behavior that is popular to yield high payoffs (imitation). In contrast to natural selection, the payoffs describe how agents make choices, and agents' payoff must be

8

A. Namatame,

N. Sato and K.

Murakami

observable by others for the model to make sense. The crossover is a kind of social learning. 4. Evolutionary Dynamics with Individual Learning We make a distinction between evolutionary systems and adaptive systems. The equations of motion in an evolutionary system reflect the basic mechanisms of biological evolution, i.e., inheritance, mutation, and selection. In an adaptive system, other mechanisms are allowed as well, e.g., modifications of strategies based on individual forecasts on the future state of the system. But, increasing the possibilities for individualistic rational behavior does not necessarily improve the outcome for the species to which the individual belongs in the long run. The introduction of spatial dimensions, so that individuals only interact with those in their neighborhood, may affect the dynamics of the system in various ways. The possibility of space-temporal structures may allow for global stability where the mean-field model (random matching) would be unstable. The presence of these various forms of spacetemporal phenomena may, therefore, also alter the evolutionary path compared with the mean-field model and we may see other strategies evolve. Different aspects of the evolutionary behavior have been investigated by many researchers: (i) by varying the payoff matrix of the game, (ii) by introducing spatial dimensions, and (iii) by introducing co-evolution. An important aspect of evolution is the learning strategy adapted by individuals3. Evolution in the hawk-dove game, for instance, drives the population to an equilibrium polymorphism state. But this symmetrical mixed equilibrium of hawk-dove is so inefficient that it is far from optimal. The term evolutionary dynamics often refers to systems that exhibit a time evolution in which the character of the dynamics may change due to internal mechanisms. In this chapter, we focus on evolutionary dynamics that may change in time according to certain local rules of individuals. Evolutionary models can be characterized both by the level at which the mechanisms are working and the dimensionality of the system. Therefore

Co-Evolutionary

Learning in Strategic

Environments

9

we describe the evolutionary dynamics specifying microscopic behavior with individuals learning. The search for evolutionary foundations of game-theoretic solution concepts leads from the notion of an evolutionarily stable strategy to alternative notions of evolutionary stability to dynamic models of evolutionary processes. The commonly used technique of modeling the evolutionary process as a system of a deterministic difference or differential equations may tell us little about equilibrium concepts other than that strict Nash equilibrium is good. We can attempt to probe deeper into these issues by modeling the choices made by the agents with their own internal models. We also focus on dynamical systems described by equations of motion that may change in time according to certain rules, which can be interpreted as crossover operations. Each agent learns to acquire the rule of interaction in the long-run. Non-cooperation games can be categorized into, dilemma games coordination games, HawkDove games and minority games. It is known that natural selection does not lead to social efficiency in these games. We show that all agents mutually learn to cooperate which result in social efficiency. 5. Learning Coupling Rules In most game theoretic models, agents calculate their best strategy based on information about what other agents have done in the past. Then agents may gradually learn the equilibrium strategy. A number of evolutionary models based on the iterated general non-cooperation games have been proposed. Many dynamical systems and evolutionary models have been constructed with the PD1. Yao applied a genetic algorithm (GA) to the iterated Prisoner's Dilemma and used a bit-string representation of finite memory strategies21. We use the different approach. In the models that we discuss here, the equations of motion for the different individuals are usually coupled, which means that we have co-evolutionary systems. The success or failure for a certain type of individual depends on which other individuals are present. In this case, there is not a fixed fitness landscape in which the co-evolutionary dynamics climbs toward increasing to

A. Namatame, N. Sato and K. Murakami

10

fitness. This ever-changing character of the world determining the evolutionary path allows for evolutionary phenomena. Co-evolutionary dynamics differ, in this sense, from the common use of the genetic algorithm, in which a fixed goal is used in the fitness function and where there is no coupling between individuals. In the genetic algorithm, the focus is on the final result what is the best or a good solution. In models of co-evolutionary systems, one is usually interested in the transient phenomenon of evolution. Each strategy in the repeated game is represented as a binary string so that the genetic operators can be applied. In order to accomplish this we treat each strategy as deterministic bit strings. We use a memory of one or two, which means that the outcomes of the previous one or two moves are used to make the current choice. We assume that 0 = Si and 1 = S2 then as Fig. 2(a) shows, there are four possible outcomes between two agents for each move SiSi(0,0), SiS2(0,l), S2S 1(1,0), S2S2(1,1). We can fully describe a deterministic strategy by recording what the strategy will do in each of the 4 different situations that can arise in the iterated game. Since no memory exists at the start, an extra 2 for 4 bits are needed to specify a hypothetical history. Each rule can be defined by a 6 bit string as shown in Fig. 2(b). \ Architecture of an Agent previous strategy bit

next strategy Own

Opp

4

0

0

5

0

1

6

1

0

7

1

1

(a) Coupling Rule

# # # #

- First Owns Hand - Memory of histories * Own Strategy

(b) Rule Representation

Fig. 2. An interaction rule of memory one

At each generation, agents repeatedly play the game for T iterations. Agent i, ie [1...N] uses a binary string / to choose his strategy at iteration t ,te.[l...T]. Each position of a binary string in Fig. 2(b) as follows: The first position, pi encodes the action that agent takes at

Co-Evolutionary

Learning in Strategic

Environments

11

iteration t = 1. A position pj J e[2,3] encodes the memories that agent i takes at iteration t - 1 and his opponent. A position pj, j e[4...7] , encodes the action that agent i takes at iteration t > 1, corresponding to the position pj, je[2,3]. An agent /' compares the position pj, j e[2,3], decides the next action. Each agent mimics the rule of the most successful neighbor. We arrange agents for an area of 20 x 20 (N = 400 agents) with the lattice model as shown in Fig. 1(a) with no gap, and four corners and end of an area connect it with an opposite side. At each time period t, each agent plays with his 8 neighbors. At the next time period, each agent mimic the interaction rule of the most successful neighbor who obtain the highest payoff. 6. Simulation Results Non-cooperation games can be categorized into dilemma games coordination games, hawk-dove games. It is known that equilibrium situations led by natural selection is far from social efficiency. A genetic algorithm is used to evolve strategies. A generation involves each player playing with 8 neighbors with the spatial model or some proportion of partners to interact are chosen from all other members in the population with the model of small-world networks. The iterated game is played fifty times between each agent. The fitness of an agent is the average payoff it achieved over repeating games. Mutation of random alleles may occur with a probability of 0.01 for all cases. 6.1. Dilemma Game Many works on evolution of cooperation have been focused on dilemma games, which is formulated as follows: Each agent faces the problem of selecting one of two decisions, cooperate (Si) or defect (S2). The payoff for each decision depends on the decisions of the other agent. Table 1 shows the payoffs for all the possible combinations of decisions. The most startling effect of the iterated Prisoner's Dilemma simulation, as observed by Axelrod1, is the fact that a group of purely egotistical

A. Namatame, N. Sato and K. Murakami

12

individuals, working towards nothing but improving themselves can lead to a population which is actually highly cooperative. Each pair of agents interacts 50 times at one generation. Fig. 3(a) shows the ratio of agents who chose the cooperative strategy Si with the lattice model. After few generations the ratio of the cooperative becomes to be 0.85. Initially the ratio of defective strategy increases, however, it is quickly wiped out and more cooperative opponents obtain higher payoffs, and the population exhibits reciprocal cooperation. Fig.3(b) shows the same experiment in a small-world network model. There are a couple of important differences in this graph and the graph obtained using the spatial environment (Fig.3(a)). Fig.3(b) shows that it is actually easier for cooperation to evolve in a small-world network environment. As a result the cooperative strategy could be realized with the dilemma game after a few generation. The cooperation is clearly more stable in the small-network environment. At beginning, each agent has a different interaction rule specified by the 4 bits information. All rules learnt by 400 agents, which were aggregated into only one type as shown in Table 3.

Table 1. The payoff matrix of a dilemma game ""\. The Others T h e O ^ - S ^ Strategy ^^\ Si

Si

S2

3

5

3

0 0

S2

1 1

5

Table 2. Learnt Rules by 400 agents in small-world environment Initial strategy

4

0

0

Array Location 5 6 1

1

7 1

Number of Agents

400

Co-Evolutionary Learning in Strategic Environments

13

Table 3. Learnt Interaction rule bit

previous strategy

strategy at t

Own

Opp

4

0

0

0

5

0

1

1

6

1

0

1

7

1

1

1

Si (Cooperate)

Si (Cooperate) 0.85

(a) Local interaction with a lattice model

(b) Interaction with a small-world network

Fig. 3 The ratio of cooperation in iterated dilemma games

o Fig. 4 The state transitions specified by the rule in Table 3

The acquired rule specified as "Oil 1" in Table 3 can be interpreted as follows: If agents behave as "cooperate", then both agents behave "cooperate", however one of them "defect", then both agents behave "defect". The state transition of this learnt rule is illustrated in Fig. 4 as the state transition diagram. There are two absorption points "00" and "11". Since each agent also acquires the rule to behave "cooperate" at the first play of each generation as shown in Table 2, they remain at the absorption points "00".

14

A. Namatame,

N. Sato and K.

Murakami

6.2. Coordination Games The coordination game with the payoff matrix in Table 4 has two equilibria with the pairs of the pure strategies (S ] , S,), (S 2 , S2), and one equilibrium of the mixed strategy. The most preferable equilibrium, Pareto-dominance is ( 5,, 5,), which dominates the other equilibrium. There is another equilibrium concept, the risk-dominance, and (S 2 , S2) risk-dominates (S,, S,). How do agents choose their strategy when the equilibria of Pareto-dominance and the risk-dominance are different? With such indigenous selection of strategy, the question is whether the society of agents may select the socially efficient Pareto-optimal strategy. Fig. 5(a) shows the ratio of agents to choose the Pareto-optimal strategy Si with the lattice model. After few generations the ratio of the Pareto-optimal strategy becomes to be 0.85. Fig.5(b) shows the same experiment using a small-world network framework. There are a couple of important differences in this graph and the graph obtained using the spatial environment (Fig.5(a)). Fig.5(b) shows that it is easier for the Pareto-optimal strategy to evolve in a small-world network environment. As a result Pareto optimal strategy could be spread out after a few generation. At beginning, each agent has a different interaction rule specified by the 4 bits information. In Table 5, we show the rules learnt by 400 agents, which are aggregated into only one type. After 10 generations, all rules of agents were converged into one rule as shown in Table 5. The acquired rule specified as "0111" in Table 5 can be interpreted as follows: If both agents choose the Pareto-optimal strategy, then they choose the same one, however if one of them chooses the risk-dominant strategy, then both agents choose the risk-dominant strategy. The state transition of the learning rule is illustrated in Fig.6 as the state transition diagram. There are two absorption points "00" and "11". Since each agent also acquires the rule to choose "Pareto-optimal strategy" at the first play of each generation as shown in Fig.6, they remain at the absorption points "00".

Co-Evolutionary Learning in Strategic Environments

15

Table 4. Payoff matrix of a coordination game ^ v ^ ^ The Other's Si

The&^I^B' Strategy ^ \

S2

1

Si

1

0 -9

-9

S2

0 0

0

Table 5. Learnt Rules by 400 agents in Interaction with a small-world network Initial strategy 0

4 0

Array Location 5 6 1 1

7 1

Number oi Agents 400

Table 6. Interaction rule bit

previous strategy

strategy at t

Own

Opp

4

0

0

0

5

0

1

1

6

1

0

1

7

1

1

1

0.85

(a) Local interaction with a lattice model

(b) Interaction with a small-world network

Fig. 5. The ratio of Pareto-optimal strategy (Si) in iterated coordination games

16

A. Namatame, N. Sato and K. Murakami

® .©

o Fig. 6. The state transitions specified by rules in Table 6

6.3. Hawk-Dove Game The hawk-dove game is formulated with the payoff matrix in Table 7. In this game, we suppose there are two possible behavioral types; one escalates the conflict until injury or sticks to display and retreats if the opponent escalates. These two types of behavior are described as "hawk" and "dove". There is the unique symmetric Nash equilibrium in mixed strategies, both agents use the strategy Si ('hawk') with probability p=V/C and the strategy S2 ('dove') with the probability 1-p =1-(V/C) [2]. Therefore if the cost of injury C is very large, the hawk frequency (V/C) will be small. At equilibrium of the mixed strategy, the expected fitness is given at the level of (V/2){1-(V/C)j. If each agent chooses the strategy S2 ('dove'), (however, the situation that both behave as doves are not equilibrium) he receives V/2. This implies that the mixed-strategy results in inefficient equilibrium. And evolutionary game can realize Pareto optimal equilibrium but it is possible that an inferior equilibrium is chosen. Fig.7(a) shows the ratio of agents to choose the strategy of Dove (S2) with the lattice model. We set to F=10, C=12 in Table 7. After few generations the ratio of the strategy of Dove becomes 0.95. Fig.7(b) shows the same experiment using a small-world network framework. In Table 8, we show the rules learnt by 400 agents, which are aggregated into only one type. After 10 generation, all rules of agents were converged into one rule as shown in Table 8. The acquired rule specified as "0001" in Table 8 can be interpreted as follows: If both agents choose the Dove strategy, then they choose the same one, however one of them chooses the Hawk strategy, then both agents

Co-Evolutionary Learning in Strategic Environments

17

choose the Hawk strategy. The state transition of the learnt rule is illustrated in Fig. 8 as the state transition diagram. There are two absorption points "00" and "11". Since each agent also acquires the rule to choose " the Dove strategy" at the first play of each generation as shown in Table 8, they remain at the absorption points "11". Table 7. The payoff matrix of the Hawk-dove game \

^

The Others

Strategy

Si (Hawk)

^ ^

S2

(Dove)

(y-cyi

Si

0 V

(Hawk)

Rats

V/2

V

S2

(Dove)

V/2

0

1.0

S2(DoveJ

0.95

/

r-'"\

/vA \ v.. (a) Local interaction with a lattice model (b) Interaction with a small-world network Fig. 7. The ratio of Dove (S2) in iterated dilemma games Table 8. Learnt Rules by 400 agents with a small-world network Initial strategy

1

Array Location

4 0

6 0

5 0

7 1

Table 9. Interaction rule bit

previous strategy

strategy at t

Own

Opp

4

0

0

0

5

0

1

0

6

1

0

0

7

1

1

1

Number of Agents

400

A. Namatame, N. Sato and K. Murakami

18

© ©

o

Fig. 8. The state transitions specified by rules in Table 8

7. Conclusion We focused on co-evolutionary dynamical systems described by equations of motion that may change in time according to rules. In the models that we discuss here, the equations of motion for the different individuals are usually coupled, which means that we have coevolutionary systems. The success or failure for a certain type of individual depends on which other individuals are present. In this case, there is not a fixed fitness landscape in which the evolutionary dynamics climbs toward increasing elevation, but a position that at one time is a peak may turn into a valley. This ever-changing character of the world determining the evolutionary path allows for complex dynamic phenomena. Co-evolutionary dynamics differ, in this sense, from the common use of the genetic algorithm, in which a fixed goal is used in the fitness function and where there is no interaction between individuals. In the genetic algorithm, the focus is on the final result what is the best or a good solution. In models of co-evolutionary systems, we consider the case of open-ended evolution. We discussed the role of individual learning in realizing social efficiency. The hypotheses we employed here reflect the limited ability of interaction with the individual learning capability. The learning strategy employed here is a kind of meta-learning. One of the variations involves the finitely iterated game that has Nash equilibria of inferior strategies. It is illustrated that when the interaction architecture of the small-world network is added, evolution usually avoids this inferior state.

Co-Evolutionary Learning in Strategic Environments

19

Our comparison of individuals playing the social games evolved in a spatial and small-world network environment has yielded some interesting results. It has been demonstrated in this chapter that interaction on a small-world network framework encourages and promotes efficiency to a greater extent, and in a more stable way, than when interaction is performed on a spatial model. This suggests that efficiency will be easier to attain when exchanges are restricted to those in an open society rather than a closed society. References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20.

R. Axelrod, The Complexity of Cooperation (Princeton Univ. Press, 1997). W. B. Arthur, American Economic Review, Vol.84, 406 (1994). D. Challet and C. Zhang, Physica, A246 (1997). D. Fudenberg and D. Levine, The Theory of Learning in Games (The MIT Pres, 1998). P. Hammerstein and R. Selten, in Handbook of Game Theory with Economic Applications, Vol.2, Eds. Auman, R., Hart, S, (Elsevier Science, 1994), p. 931. J. Hansaryi and R. Selten, A Game Theory of Equibrium Selection in Games (MIT Press, 1988). J. Hofbauer and K. Sigmund, Ecolutionary Games and Population Dynamics (Cambridge Univ. Press, 1998). M. Kandori and G. Mailath, Econometrica, Vol.61 (1993) p29. Y. Kaniovski, A. Kryazhimskii and H. Young, Games and Economics Behavior, Vol 31, (2000) p50. K. Lindgren, The Economy as an Evolving Complex System II, (1997) p337. Y. Murakami, H. Sato and A. Namatame, in International Conference on Computational Intelligence and Multimedia Applications, (2001) p241. Y. Murakami, H. Sato, and A. Namatame, in The 5lh Australia-Japan Joint Workshop on Intelligent & Evolutionary Systems, (2001) pi9. M. A. Nowak et al., The Arithmetics of Mutual Help (Scientific American, June, (1995). M. A. Nowak and R. M. May, Evolutionary Games and spatial Chaos, (Nature, 359, 1992) p826. M. Sipper, Evolution of Parallel Cellular Machines (Springer, 1996). J. M. Smith, Evolution and the Theory of Games (Cambridge University Press, 1982). K. Uno, and A. Namatame, in GECCO'99 Workshop on Artificial life, (1999). D. J. Watts, Small Worlds (Princeton University Press, 1999). J. Weibull, Evolutionary Game Theory (The MIT press, 1996). X. Yao, and P. Darwen, Informatica, Vol. 18(1994) p435.

CHAPTER 2 USING EVOLUTION TO LEARN USER PREFERENCES

Supiya Ujjin and Peter J. Bentley Department of Computer Science University College London, Gower Street, London WC1E 6BT S. Ujjin@cs. ucl. ac. uk, P. Bentley@cs. ucl. ac. uk

Recommender systems are new types of internet-based software tools, designed to help users find their way through today's complex on-line shops and entertainment websites. This chapter describes a new recommender system, which employs a genetic algorithm to learn personal preferences of users and provide tailored suggestions.

1. Introduction The rapid expansion of the Internet has brought about a new market for trading. Electronic commerce or e-commerce has enabled businesses to open up their products and services to a massive client base that was once available only to the largest multinational companies. As the competition between businesses becomes increasingly fierce, consumers are faced with a myriad of choices. Although this might seem to be nothing but beneficial to the consumer, the sheer wealth of information relating to the various choices can be overwhelming. One would normally rely on the opinions and advice of friends or family members but unfortunately even they have limited knowledge. Recommender systems provide one way of circumventing this problem. As the name suggests, their task is to recommend or suggest items or products to the customer based on his/her preferences. These systems are often used by E-commerce websites as marketing tools to

20

Using Evolution to Learn User

Preferences

21

increase revenue by presenting products that the customer is likely to buy. An internet site using a recommender system can exploit knowledge of customers' likes and dislikes to build an understanding of their individual needs and thereby increase customer loyalty1'2. This chapter focuses on the use of evolutionary search to fine-tune a profile-matching algorithm within a recommender system, tailoring it to the preferences of individual users. This enables the recommender system to make more accurate predictions of users' likes and dislikes, and hence better recommendations to users. The chapter is organised as follows: section 2 outlines related work, and section 3 describes the recommender system and genetic algorithm. Section 4 provides experimental results and analysis. Finally section 5 concludes. 2. Background From the literature, it seems that the definition of the term "recommender system" varies depending on the author. Some researchers use the concepts: "recommender system", "collaborative filtering" and "social filtering" interchangeably3'4. Conversely, others regard "recommender system" as a generic descriptor that represents various recommendation/prediction techniques including collaborative, social, and content based filtering, Bayesian networks and association rules5'6. In this chapter, we adopt the latter definition when referring to recommender systems. MovieLens (http://www.movielens.umn.edu), a well-known research movie recommendation website, makes use of collaborative filtering technology to make its suggestions. This technology captures user preferences to build a profile by asking the user to rate movies. It searches for similar profiles (i.e., users that share the same or similar taste) and uses them to generate new suggestions. One shortcoming that most websites using collaborative filtering suffer from is that they do not have any facility to provide explanations of how recommendations are derived. This is addressed in the paper7 which proposes explanation

22

Supiya Ujjin and Peter J.

Bentley

facilities for recommender systems in order to increase users' faith in the suggestions. By contrast, LIBRA (http://www.cs.utexas.edu/users/ libra') combines a content-based approach with machine learning to make book recommendations. The content-based approach differs from collaborative filtering in that it analyses the contents of the items being recommended. Furthermore, each user is treated individually - there is no sense of "community" which forms the basis of collaborative filtering. Dooyoo (http://www.dooyoo.co.uk) operates in a slightly different way. It too is a useful resource that provides recommendations to those seeking advice, but it focuses mainly on gathering qualitative opinions from its users, and then making them available to others. Visitors will often submit reviews on items or services ranging from health spas to mobile phones. These items are categorised in a similar fashion to the layout on a structured search engine, such as Yahoo! Researchers at the University of the West Of England have also been working on a movie Recommender System9. Their idea is to use the immune system to tackle the problem of preference matching and recommendation. User preferences are treated as a pool of antibodies and the active user is the antigen. The difference in their approach and the other existing methods is that they are not interested in finding the one best match but a diverse set of antibodies that are a close match. 3. System Overview The system described in this chapter is based around a collaborative filtering approach, building up profiles of users and then using an algorithm to find profiles similar to the current user. (In this chapter, we refer to the current user as the active user, A). Selected data from those profiles are then used to build recommendations. Because profiles contain many attributes, many of which have sparse or incomplete data7, the task of finding appropriate similarities is often difficult. To overcome these problems, current systems (such as MovieLens) use stochastic and heuristic-based models to speed up and improve the

Using Evolution to Learn User Preferences

23

quality of profile matching. This work takes such ideas one step further, by applying an evolutionary algorithm to the problem of profile matching. 3.1. MovieLens Dataset The dataset collected through the MovieLens website (http://www.movielens.umn.edu) has been made available for research purposes and used in this research. The database contains details of 943 users, each with many parameters or features: demographic information such as age, gender and occupation is collected when a new user registers on the system. The evolutionary recommender system uses 22 features from this data set: movie rating, age, gender, occupation and 18 movie genre frequencies: action, adventure, animation, children, comedy, crime, documentary, drama, fantasy, film-noir, horror, musical, mystery, romance, sci-fi, thriller, war, western. Table 1 and Fig. 1 show the relationships and dependencies between the various elements in the database. ITEM MOVIE J D •MOVIE TITLE RF1FASF DATE VIDEO RELEASE IMDb URL . UNKNOWN ACTION ADVENTURE ANIMATION CHILDREN COMEDY CRIME DOCUMENTARY DRAMA FANTASY FILM_NOIR HORROR MUSICAL MYSTERY ROMANCE SCI-H THRILLER WAR - WESTERN rf

USER Number 4USER ID Number^. "AGE GENDER OCCUPATION ID Number Text ZIP.CODE

OCCUPATION ' OCCUPATIONJD OCCUPATION

/

^.DATA USER ID ^ 1—"Number MOVIE ID Number RATING Number TIMESTAMP Number

^GENRE_FREQUENCY USERJD Number Number' UNKNOWN Number ACTION Number ADVENTURE Number ANIMATION Number CHILDREN Number COMEDY Number CRIME DOCUMENTARY Number Number DRAMA Number FANTASY Number FILM_NOIR Number HORROR Number MUSICAL Number MYSTERY Number ROMANCE Number SCI-FI Number THRILLER Number WAR Number WESTERN

GENRE GENRE E) ..GENRE A '

4

_„.-y

Number Tex!

V

Number Text Number Date/Ti me Text Boolean Boolean Boolean Boolean Boolean Boolean Boolean Boolean Boolean Boolean Boolean Boolean Boolean Boolean Boolean Boolean Boolean Boolean Boolean

V

Fig. 1. Relationships and dependencies between the various elements in the database

24

Supiya Ujjin and Peter J. Bentley Table 1. Table Descriptions

DATA

100,000 ratings by 943 users on 1682 items. Each user has rated at least 20 movies. Demographic information about the users. Information about the items (movies). The last 19 fields represent the genres, a boolean value, 0 or 1 indicates whether the movie belongs to the specific genre; a movie can be in several genres. A list of the occupations. A list of the genres. The frequencies of the genres for all the items the user has rated. This has been added to the database as it is thought to represent how much the user prefers each genre.

USER ITEM

OCCUPATION GENRE GENRE FREQUENCY

3.2. Profile Generator Before recommendations can be made, the movie data must first be processed into separate profiles, one for each person, defining that person's movie preferences. 3

4

..22

Rating

1

2 Age

Gender

Occupation

18 Genre frequencies

5

23

0

45

000000100010000000

Fig. 2. profiled,i) - profile for user/ with rating on movie item ;', if;' has a rating of 5

A profile for user/, denoted proJlleQ) is represented as an array of 22 values for the 22 features considered. The profile has two parts: a variable part (the rating value, which changes according to the movie item being considered at the time), and a fixed part (the other 21 values, which are only retrieved once at the beginning of the program). Because user j may have rated many different movies, we define profiled,i) t o mean the profile for user/ on movie item /, seefig.2. Once profiles are built, the process of recommendation can begin. Given an active user A, a set of profiles similar to profile(A) must be found. 3.3. Neighbourhood Selection

Using Evolution

to Learn User

Preferences

25

The success of a collaborative filtering system is highly dependent upon the effectiveness of the algorithm in finding the set or neighbourhood of profiles that are most similar to that of the active user. It is vital that, for a particular neighbourhood method, only the best or closest profiles are chosen and used to generate new recommendations for the user. There is little tolerance for inaccurate or irrelevant predictions. The neighbourhood selection algorithm consists of three main tasks: (i) profile selection (ii) profile matching (iii) best profile collection. 3.3.1 Profile Selection In an ideal world, the entire database of profiles would be used to select the best possible profiles. However this is not always a feasible option, especially when the dataset is very large or if resources are not available. As a result, most systems opt for random sampling and this process is the responsibility of the profile selection part of the algorithm. This work investigates two methods of profile selection: (i) Fixed: the first n users from the database are always used in every experiment (ii) Random: n users are picked randomly from the database, where n = 10 or 50 in our experiments. 3.3.2 Profile Matching After profile selection, the profile matching process then computes the distance or similarity between the selected profiles and the active user's profile using a distance function. This research focuses on this profile matching task, i.e., the evolutionary algorithm is used to fine-tune profile matching for each active user. From the analysis of Breese et al.3, it seems that most current recommender systems use standard algorithms that consider only "voting information" as the feature on which the comparison between two profiles is made. However in real life, the way in which two people are said to be similar is not based solely on whether they have

Supiya Ujjin and Peter J. Bentley

26

complimentary opinions on a specific subject, e.g., movie ratings, but also on other factors, such as their background and personal details. If we apply this to the profile matcher, issues such as demographic and lifestyle information which include user's age, gender and preferences of movie genres must also be taken into account. Every user places a different importance or priority on each feature. These priorities can be quantified or enumerated. Here we refer to these as feature weights. For example, if a male user prefers to be given recommendations based on the opinions of other men, then his feature weight for gender would be higher than other features. In order to implement a truly personalised recommender system, these weights need to be captured and fine-tuned to reflect each user's preference. Our approach shows how such weights can be evolved by a genetic algorithm. A potential solution to the problem of evolving feature weights, w(A), for the active user, A is represented as a set of weights as shown below in Fig. 3 where wyis the weight associated with feature/whose genotype is a string of binary values. W\

w2

w3

W>22

Fig. 3. Phenotype of an individual in the population

Each individual contains 22 genes, which are evolved by an elitist genetic algorithm (described in section 3.4). The comparison between two profiles can now be conducted using a modified Euclidean distance function, which takes into account multiple features. Euclidean(AJ) is the similarity between active user ^4 and usery: euclidean(A,j) = — S\

\S\/f*diffi,f(AJ)2

where: A is the active user j is a user provided by the profile selection process, where y ^ A z is the number of common movies that users A andy have rated. Wf, is the active user's weight for feature/

Using Evolution to Learn User Preferences

27

/' is a common movie item, where profile(A,i) and profile(j,i) exists. diffi/Aj) is the difference in profile value for feature / between users A and j on movie item i. Note that before this calculation is made, the profile values are normalised to ensure they lie between 0 and 1. When the weight for any feature is zero, that feature is ignored. This way we enable feature selection to be adaptive to each user's preferences. The difference in profile values for occupation is either 0, if the two users have the same occupation, or 1 otherwise.

DB

euclidean(A,j)

=

similarity(/A,y)

weights(A)

* \ Genetic * j Algorithm

Fig. 4. Calculating the similarity between A andy

3.3.3 Best Profile Collection Once the Euclidean distances, euclidean(AJ), have been found between profile(A) and profiled) f° r a ^ values of/ picked by the profile selection process, the "best profile collection" algorithm is called, see Fig. 4. This ranks every profiled) according to its similarity to profile(A). The system then simply selects the users whose Euclidean distance is above a certain threshold value (considered most similar to the active user) as the

28

Swpiya Ujjin and Peter J.

Bentley

neighbourhood of A. This value is a system constant that can be changed. 3.4. Making a Recommendation To make a recommendation, given an active user A and a neighbourhood set of similar profiles to A, it is necessary to find movie items seen (and liked) by the users in the neighbourhood set that the active user has not seen. These are then presented to the active user through a user interface. Because the neighbourhood set contains those users who are most similar to A (using in our case the specific preferences of A through evolved weighting values), movies that these users like have a reasonable probability of being liked by A. 3.5. Genetic Algorithm As described earlier, a genetic algorithm is used to evolve feature weights for the active user, and hence help tailor the matching function to the user's specific personality and tastes. An elitist genetic algorithm was chosen for this task, where a quarter of the best individuals in the population are kept for the next generation. When creating a new generation, individuals are selected randomly out of the top 40% of the whole population to be parents. Two offspring are produced from every pair of parents, using single-point crossover with probability 1.0. Mutation is applied to each locus in genotype with probability 0.01. A simple unsigned binary genetic encoding is used in the implementation, using 8 bits for each of the 22 genes. The GA begins with random genotypes. A genotype is mapped to a phenotype (a set of feature weights) by converting the alleles of the binary genes to decimal. The feature weights can then be calculated from these real values. First, the importance of the 18 genre frequencies are reduced by a given factor, the weight reduction size. This is done because the 18 genres can be considered different categories of a single larger feature, Genre. Reducing the effect of these weights is therefore intended to give the

Using Evolution to Learn User

Preferences

29

other unrelated features (movie rating, age, gender, occupation) a more equal chance of being used. Second, the total value of phenotype is then calculated by summing the real values for all 22 features. Finally, the weighting value for each feature can be found by dividing the real value by the total value. The sum of all the weights will then add up to unity. 3.5.1 Fitness Function Calculating the fitness for this application is not trivial. Every set of weights in the GA population must be employed by the profile matching processes within the recommender system. So the recommender system must be re-run on the MovieLens dataset for each new set of weights, in order to calculate its fitness. But running a recommender system only produces recommendations (or predictions), not fitnesses. A poor set of weights might result in a poor neighbourhood set of profiles for the active user, and hence poor recommendations. A good set of weights should result in a good neighbourhood set, and good recommendations. So a method of calculating the quality of the recommendations is required, in order that a fitness score can be assigned to the corresponding weights. One solution would be to employ the active user as a fitness function. This would involve obtaining feedback from the user by asking him to judge the quality of recommendations8. His input could be used to help derive fitness scores for the current set of feature weights. This fitness score would give a highly accurate view of the user's preferences. However, it is unlikely that every user will be willing to participate in every recommendation - the time needed would be too great. Instead, it was decided to reformulate the problem as a supervised learning task. As described previously, given the active user A and a set of neighbouring profiles, recommendations for A can be made. In addition to these recommendations, it is possible to predict what A might think of them. For example, if a certain movie is suggested because similar users saw it, but those users only thought the movie was "average", then it is likely that the active user might also think the movie was "average". Hence, for the MovieLens dataset, it was possible for the

30

Supiya Ujjin and Peter J.

Bentley

system to both recommend new movies and to predict how the active user would rate each movie, should he go and see it. The predicted vote computation used in this chapter has been taken from the paper3 and modified such that the Euclidean distance function (section 3.2.2) now replaces the weight in the original equation. The predicted vote, predict_vote(A,i), for A on item /', can be defined as: n predict _ vote(A, i) = meanA, + t X euclidean(A, j)(\ole(j, 7=1

i) - mean .) J

where: mean, is the mean vote for user/ k is a normalising factor such that the sum of the euclidean distances is equal to 1. vote(j,i) is the actual vote that user/ has given on item / n is the size of the neighbourhood.

Profile Selection and Matching

euclidean(AJ) for all users where A tj

Best Profile Selection Neighbourhood set

predict vote(AJ) for all items /',.., in training set

fitness™

fitness.

....fitness^

Average(fitness : „fitness„,..,fitnessJ

Fitness Score

Fig. 5. Finding the fitness score of an individual (the active user's feature weights)

Using Evolution

to Learn User

Preferences

31

All the movie items that the active user has seen are randomly partitioned into two datasets: a training set (1/3) and a test set (2/3). To calculate a fitness measure for an evolved set of weights, the recommender system finds a set of neighbourhood profiles for the active user, as described in section 3.3. The ratings of the users in the neighbourhood set are then employed to compute the predicted rating for the active user on each movie item in the training set. Because the active user has already rated the movie items, it is possible to compare the actual rating with the predicted rating. So, the average of the differences between the actual and predicted votes of all items in the training set are used as fitness score to guide future generations of weight evolution, see Fig. 5. 4. Experiments Four sets of experiments were designed to observe the difference in performance between the evolutionary recommender system and a standard, non-adaptive recommender system based on the Pearson algorithm3. In each set of experiments, the predicted votes of all the movie items in the test set (the items that the active user has rated but were not used in weights evolution) were computed using the final feature weights for that run. These votes were then compared against those produced from the simple Pearson algorithm. The Pearson algorithm used in the experiments is based on the k Nearest Neighbour algorithm. A correlation coefficient, shown below, is used as the matching function for selecting the k users that are most similar to the active user to give predictions. This replaces the Euclidean function described earlier; all other details remain the same. z y correlation(A,j)

=

,

(vote(A,i)-

meanA)(vote(j,i)

-

meant)

l=1

y

(vote(A,i)-meanA)

(vote(j,i)-

mean;)

\ '=1

The four experiments also evaluated two system variables to assess their effect on system performance: the profile selection task (the way in

Supiya Ujjin and Peter J.

32

Bentley

which profiles were selected from the database), and the size of the neighbourhood. Table 2 lists parameter values used in all four experiments. Table 2. Parameter values used in the experiments Parameter Name population size

Parameter Value 75

termination threshold

0.06

maximum generations (per run) weight reduction size number of runs

300

k users (Pearson algorithm)

5 or 25

4 30

Description The number of individuals in the population at each generation. When the fitness score of the best individual (set of feature weights) is below the threshold, a good solution is found and this set of weights is used as the final result for the current run. If the number of generations reaches this value and the solution has not been found, the best individual for that generation is used as the final result. The scaling factor for the 18 genre frequencies. The number of times the system was run for each active user. Vi of the number of users in each experiment

The four sets of experiments were as follows: Experiment 1: Each of the first 10 users was picked as the active user in turn, and the first 10 users (fixed) were used to provide recommendations. Experiment 2: Each of the first 50 users was picked as the active user in turn, and the first 50 users (fixed) were used to provide recommendations. Experiment 3: Each of the first 10 users was picked as the active user in turn, and 10 users were picked randomly and used to provide recommendations (the same 10 used per run). Experiment 4: Each of the first 50 users was picked as the active user in turn, and 50 users were picked randomly and used to provide recommendations (the same 50 used per run).

Using Evolution to Learn User Preferences

33

4.1. Results Figs. 6 to 9 show the results for experiments 1 to 4, respectively. Each graph shows the percentage of the number of ratings that the system predicted correctly out of the total number of available ratings by the current active user. Whilst the predictions computed with the Pearson algorithm always remain the same given the same parameter values, those obtained from the GA vary according to the feature weights of that run. Out of the 30 runs for each active user in each experiment, the run with the best feature weights (that gave the highest percentage of right predictions) was chosen and plotted against the result from the Pearson algorithm.1 Fig. 6 shows that in the first experiment, the GA recommender performed equally well (or better) compared to the Pearson algorithm on 8 active users out of 10. Fig. 7 shows that in the second experiment, out of the 50 users the accuracy for the GA recommender fell below that of the Pearson algorithm for 14 active users. On the rest of the active users, the accuracy for the GA recommender was found to be better - in some cases the difference was as great as 31%. The random sampling for experiment 3 showed great improvement on the prediction accuracy for the GA recommender, see Fig. 8. All 10 active users performed better than the Pearson algorithm. The results for the last experiment show that the accuracy for the GA recommender was significantly better for all but 4 active users, see Fig. 9.

1

The best rather than average was plotted since this is closest to the real world scenario where this system could be run off-line and the current best set of feature weights would be set as the initial preference of the active user. Following this, the evolved weights could be stored on the user's local machine. A local copy of the system would then be responsible for fine-tuning the weights to suit that user's preferences further. This way the processing load on the server would be reduced and parallelism can be achieved.

34

Supiya Ujjin and Peter J. Bentley

J j 40

Pearson • GA Recommender

liiiiilllil

20 7

1 2

3

4

5

6

7

Active User

Fig. 6. Results for experiment 1

d Pearson

1 J I'llillfi t

III i l 111

llll! II lilt 1

3

5

7

9

• GA Recommender

11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 Active User

Fig. 7. Results for experiment 2

^ Pearson I GA Recommender

lliJiliJ 1 2

3

4

5

6

7

Active User

Fig. 8. Results for experiment 3

1 Pearson

20 -

\l

I

, iillli

II!!

I

0 1

3

5

7

9

lil1

ll

11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 Active User

Fig. 9. Results for experiment 4

i

B GA Recommenderi

Using Evolution to Learn User

Preferences

35

4.2. Analysis of Results Fig. 6 indicates that the prediction accuracy for the active user 3 and 8 on the GA recommender was worse than that obtained from using the Pearson algorithm. But when the number of users was increased to 50 in experiment 2, the accuracy for the two mentioned active users rose and outperformed the other algorithm. This was expected - as the number of users goes up, the probability of finding a better matched profile should be higher and hence, the accuracy of the predictions should increase as well. The patterns in both experiments 3 and 4 for the active users 1 to 10 look very similar. Both show an improved accuracy compared to the Pearson algorithm but in experiment 4 there seems to be a greater improvement. Again, this is likely to be because of the increase in the number of users. The results suggest that random sampling is a good choice for the profile selection task of retrieving profiles from the database. Random sampling was expected to be better than fixing which users to select because it allowed the search to consider a greater variety of profiles (potentially 10*30 runs = 300 users in experiment 3 and 50 * 30 = 1500 users in experiment 4) and hence find a better set of well matched profiles. As mentioned earlier, only the run(s) with the best feature weights for each active user were considered for this analysis. We now look into these runs in more detail to see how the feature weights obtained and users selected for the neighbourhood in these runs played a part in determining user preference. Looking at experiment 1, when more than 1 run for an active user achieved the same best performance (highest number of votes being predicted correctly) results indicate that the same set of users had been selected for the neighbourhood to give recommendations. Moreover, for other runs that did not perform as well as the best run(s), different users that gave the best performance had been selected. For example, for active user 2 in experiment 1, all the runs that got the same percentage as the best, chose user 4 to be in the neighbourhood. The other active users did not select any users to give recommendations, instead the mean vote was used. Data gathered during experiment 2 corroborates this view. In

36

Supiya Ujjin and Peter J.

Bentley

addition, as the number of users was increased, the users that were originally selected for the neighbourhood in experiment 1 were still being chosen in experiment 2 as a subset of a larger neighbourhood. For example, as mentioned above, in experiment 1 user 2 picked user 4 to be in the neighbourhood, in experiment 2 this user picked users 4,13,18,22,42,43,49. This, however, only applies to the active users that performed better than the Pearson algorithm in experiment 1. The accuracy for active user 8 was worse in experiment 1, in which users 4, 5, 7 and 10 were selected. In experiment 2 when users 4 and 10 were not included in the neighbourhood, the accuracy improved tremendously as seen in Fig. 7. The trend described could not be observed when random sampling was used in experiments 3 and 4, as it was more difficult for the system to select the same users to examine at each run. Looking at the final feature weights obtained for each active user, many interesting observations have been found. Here we focus on the first two experiments as they have 10 common active users. Firstly, in experiment 2 when more than 1 run came up with the best performance, the feature weights seem to show very similar trends. For example, Fig. 10 shows the weight emphasis on the first 2 features: rating and age. It is also clear that this user does not show any interest in the 3 rd feature which is gender. So as long as the people that are giving him recommendations have similar opinions and are in the same age group as him, he does not care whether they are male or female. 0.25

. 0

i = f • ". i 5

10

s

«i ! : \ t 15

l

i • •* 20

Fig. 10. Feature weights for active user 2 (weights 5 to 22 are lower because of the scaling factor)

Using Evolution to Learn User

Preferences

37

The feature weights obtained for active user 8 were also interesting. They show that for this user, age and gender (features 2 and 3) are more significant. By looking further at the movie genres (features 5-22), we found that people who have similar opinions as this user on action (feature 5), adventure (feature 6), horror (feature 15), romantic (feature 18) and war (feature 21) movies are likely to be picked for the neighbourhood set. As these genres are stereotypically related to gender and age, for example, men prefer action movies and war movies, the weights showed consistent description of the user's preference. Another example is active user 7 whose weights show strong feelings for documentary, mystery, sci-fi and thriller genres and emphasis on age. This user is a 57-year old male who may explain reduced significance of children and romance genres. Also, we discover that children and animation genre features usually have similar weights - this could be because these two genres are usually related i.e. most animation films are children's films like Disney cartoons. From the observations above, we can see that age is often as or more important as rating. This shows that the theory behind the original collaborative filtering does not always hold. This is hardly surprising as everyday experience suggests that most people listen to the recommendations made by their friends who are most likely to be in the same age group as them. In experiment 1, all active users seem to have similar feature weights as the ones obtained in experiment 2 apart from users 3, 8, 9 and 10. We divide our analysis of this into 2 parts. Firstly, users 3 and 8 performed worse than the Pearson algorithm in experiment 1. This was because the weights obtained did not describe the users realistically. As the number of users was increased in experiment 2, the weights could be captured better and hence produce better performance. Secondly, the weights for active users 9 and 10 did not display any useful information in experiment 2. This resulted in reduced performance for them compared to the original algorithm. But in experiment 1, the weights for these 2 users show a consistent trend resulting in increased accuracy compared to the Pearson algorithm in this experiment.

38

Supiya Ujjin and Peter J. Bentley

This approach has been shown to work well, but there are problems. As fitness scores are computed by getting the differences between the actual and predicted votes, this is only achievable if the user has already actively voted on movies, otherwise the intersection between recommended items and those already voted for by the active user would return very few titles or even none. In this case, this approach will fail, as a fitness score cannot be determined. Table 3. Recommended movies for active user 1 Film

Action

Braveheart

Yes

Apollo 13

Yes

Blade Runner

No

Aladdin

No

Independence Day (ID4)

Yes

Die Hard

Yes

Top Gun

Yes

Empire Strikes Back, The

Yes

Return of the Jedi

Yes

GoodFellas

No

Blues Brothers, The

Yes

Sting, The

No

Dead Poets Society

No

Star Trek: First Contact

Yes

Raising Arizona

No

Men in Black

Yes

0.6 0.5 0.4

J 0.3

• Best Run

0.2 0.1 0

1. Feature weights for active user 1 in 4-feature experiment

Using Evolution to Learn User Preferences

39

In an earlier experiment with only 4 features: Rating, Age, Gender and Occupation, it was noticed that many solutions were found for items which are sometimes associated with gender (inferred by gender). For example, when the active user's feature weights showed that the user preferred to be recommended by people of the same gender (3rd feature), solutions were often found for items that belonged to the Action genre. Fig. 11 and Table 3 illustrate this. 10 out of 16 items (with a predicted vote of 4 or above) that were being recommended to the active user Using this set of weights are action movies. Because of this, it would be interesting to see if results can be improved if we make use of association rules. 5. Conclusions This work has shown how evolutionary search can be employed to finetune a profile-matching algorithm within a recommender system, tailoring it to the preferences of individual users. This was achieved by reformulating the problem of making recommendations into a supervised learning task, enabling fitness scores to be computed by comparing predicted votes with actual votes. Experiments demonstrated that, compared to a non-adaptive approach, the evolutionary recommender system was able to successfully fine-tune the profile matching algorithm. This enabled the recommender system to make more accurate predictions, and hence better recommendations to users. References 1. J. B. Schafer, J. A. Konstan and J. Riedl. E-Commerce Recommendation Applications. Journal of Data Mining and Knowledge Discovery (2001). 2. J. B. Schafer, J. Konstan and J. Riedl. Recommender Systems in E-Commerce. Proceedings of the ACM 1999 Conference on Electronic Commerce (1999). 3. J. S. Breese, D. Heckerman and C. Kadie. Empirical analysis of predictive algorithms for collaborative filtering. Proceedings of the 14th Conference on Uncertainty in Artificial Intelligence, pp. 43-52 (1998). 4. K. Goldberg, T. Roeder, D. Gupta and C. Perkins. Eigentaste: A Constant Time Collaborative Filtering Algorithm. UCB ERL Technical Report MOO/41 (2000).

40

Supiya Ujjin and Peter J. Bentley 5. L. Terveen and W. Hill. Beyond Recommender Systems: Helping People Help Each Other. In HCI In The New Millenium, J. Carroll ed. Addison-Wesley (2001). 6. J. A. Delgado. Agent-Based Information Filtering and Recommender Systems on the Internet. PhD thesis, Nagoya Institute of Technology (2000). 7. J. L. Herlocker, J. A. Konstan and J. Riedl. Explaining Collaborative Filtering Recommendations. Proceedings of the ACM 2000 Conference on Computer Supported Cooperative Work (2000). 8. P. J. Bentley and D. W. Come. Creative Evolutionary Systems. (Morgan Kaufman Pub, 2001). 9. S. Cayzer and U. Aickelin. A Recommender System based on the Immune Network. Proceedings of 2002 World Congress on Computational Intelligence, pp 807-813 (2002)

CHAPTER 3 A PARALLEL GENETIC ALGORITHM FOR CLUSTERING

Juha Kivijarvi, Joonas Lehtinen and Olli S. Nevalainen Turku Centre for Computer Science (TUCS) Department of Information Technology University of Turku, 20014 Turku, Finland E-mail: [email protected] Parallelization of genetic algorithms (GAs) has received considerable attention in recent years. Reasons for this are the availability of suitable computational resources and the need for solving harder problems in reasonable time. We describe a new parallel self-adaptive GA for solving the data clustering problem. The algorithm utilizes island parallelization using a genebank model, in which GA processes communicate with each other through the genebank process. This model allows one to implement different migration topologies in an easy manner. Experiments show that significant speedup is reached by parallelization. The effect of migration parameters is also studied and the development of population diversity is examined by several measures, some of which are new. 1. Introduction The objective in clustering is to divide a given set of data objects into a number of groups called clusters in such a way that similar objects belong to the same cluster whereas dissimilar objects are in different ones 1,2 . The problem appears as many variations in numerous fields of science such as data compression, pattern recognition, image analysis, medical data analysis, data mining, social sciences, bioinformatics, etc. The problem instances are commonly large in several respects: the dimensionality of data objects may be high, their number may be thousands or millions, and the number of clusters may be several hundreds. Thus the amount of computation needed for finding satisfactory solutions is often high, even if the hope of finding a true global optimum is abandoned. In the present study we consider the case of Euclidean clustering. In 41

42

J. Kivijarvi,

J. Lehtinen

and O. S.

Nevalainen

particular, we assume that the data objects can be considered as points in a Euclidean space and calculation of artificial cluster centers is meaningful. Furthermore, the number of clusters is expected to be known. This situation is met for example in vector quantization 3 . There exists a great number of algorithms for clustering 4,5 . These can be classified as partitional and hierarchical. Partitional algorithms aim to divide the given data into a number of clusters whereas hierarchical methods generate a hierarchy of clusterings of different sizes. Partitional algorithms are commonly iterative, i.e. they start with an initial solution and iteratively improve it. Hierarchical methods can be divided into divisive and agglornerative methods. They apply split and merge operations, respectively, until a clustering with the desired number of clusters has been reached 6 . General heuristic search techniques 7 have gained popularity in solving hard combinatorial optimization problems and clustering is not an exception. High quality results have been reported for e.g. simulated annealing, tabu search8 and especially genetic algorithms (GAs) 9,10 . In the present study we concentrate on GAs since they are very effective while still conceptually simple and have been shown to achieve excellent results in clustering problems 11 . GAs perform stochastic optimization by applying stochastic evolution inspired operators to a set of candidate solutions. These operations include mutation, crossover and selection. There are several properties which have increased the popularity of GAs as a general framework for solving hard optimization problems. The quality of solutions found by GAs is in many cases excellent. The method is also easy to understand and an exact mathematical formulation is not needed; it suffices to determine a suitable representation for the individuals and a pertinent crossover operator. All the above benefits, however, are not earned for free: GAs often suffer from very long running times so that a common complaint on their usefulness deals with the practicality of the approach. This drawback is underlined in many practical applications of GAs which include complicated objective functions or time constraints for problem solving. In addition, there are many design alternatives to choose from, and the final efficiency often strongly depends on details of the design and parameter values. To overcome the latter difficulty, adaptive GAs have been developed 12 ' 13 . A self-adaptive genetic algorithm for clustering (SAGA) is described in Ref. 14. In this algorithm, each individual contains several parameter values in addition to the actual solution. SAGA was demonstrated to be very robust and to achieve excellent results. The main drawback of

A Parallel Genetic Algorithm for

Clustering

43

the method is the long running time. Fortunately, GAs are known to be easily parallelizable. Thus, using several interconnected processors one would expect to be able to reduce the actual running time considerably. Our primary goal is to speed up SAGA, but it is also interesting to see whether parallelization leads to algorithmic benefits as occasionally suggested. For a discussion of different models of parallelizing GAs and a literary survey, see Ref. 15. In Ref. 16 one can find in-depth mathematical analysis on different aspects of parallel GAs. 2. Clustering Problem The clustering problem is defined as follows. Given a set of N data objects Xi, partition the data set into M clusters in such a way that similar objects are grouped together and dissimilar objects belong to different groups. Each object Xi has K features x\ '. The features are assumed to be numerical and of the same scale. Mapping P defines a clustering by giving for each data object Xi the index p, of the cluster it is assigned to. Furthermore, each cluster j has a cluster representative Cj. We measure the dissimilarity (distance) between objects xt and Xj by the Euclidean distance dyXi-) Xj) —

Our representation of a solution to the clustering problem includes both mapping and cluster representatives, i.e. a solution is of the form w = (P, C) where P — ( p i , . . . ,PN) and C = ( c i , . . . ,CM)- The objective is to find a solution with minimal mean square error (MSE), which is calculated as e(u) = j^jrd(xi,cPi)2.

(2)

t=i

Given a mapping P , the optimal cluster representatives are the cluster centroids Cj = ^Pi=iX'l<j<M,l