Intelligent Agent Software Engineering

Intelligent Agent Software Engineering Valentina Plekhanova University of Sunderland, UK IDEA GROUP PUBLISHING Hershey...

Author: Valentina Plekhanova

86 downloads 2185 Views 2MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

Intelligent Agent Software Engineering Valentina Plekhanova University of Sunderland, UK

IDEA GROUP PUBLISHING Hershey • London • Melbourne • Singapore • Beijing

Acquisition Editor: Senior Managing Editor: Managing Editor: Development Editor: Copy Editor: Typesetter: Cover Design: Printed at:

Mehdi Khosrowpour Jan Travers Amanda Appicello Michele Rossi Lori Eby Jennifer Wetzel Integrated Book Technology Integrated Book Technology

Published in the United States of America by Idea Group Publishing (an imprint of Idea Group Inc.) 701 E. Chocolate Avenue, Suite 200 Hershey PA 17033 Tel: 717-533-8845 Fax: 717-533-8661 E-mail: [email protected] Web site: http://www.idea-group.com and in the United Kingdom by Idea Group Publishing (an imprint of Idea Group Inc.) 3 Henrietta Street Covent Garden London WC2E 8LU Tel: 44 20 7240 0856 Fax: 44 20 7379 3313 Web site: http://www.eurospan.co.uk Copyright © 2003 by Idea Group Inc. All rights reserved. No part of this book may be reproduced in any form or by any means, electronic or mechanical, including photocopying, without written permission from the publisher. Library of Congress Cataloging-in-Publication Data

Plekhanova, Valentina. Intelligent agent software engineering / Valentina Plekhanova. p. cm. ISBN 1-59140-046-5 (hardcovers) — ISBN 1-59140-084-8 (ebook) 1. Intelligent agents (Computer software) 2. Computer software—Development. I. Title. QA76.76.I58 P78 2003 006.3—dc21 2002014186 British Cataloguing in Publication Data A Cataloguing in Publication record for this book is available from the British Library.

NEW from Idea Group Publishing • Digital Bridges: Developing Countries in the Knowledge Economy, John Senyo Afele/ ISBN:1-59140-039-2; eISBN 1-59140-067-8, © 2003 • Integrative Document & Content Management: Strategies for Exploiting Enterprise Knowledge, Len Asprey and Michael Middleton/ ISBN: 1-59140-055-4; eISBN 1-59140-068-6, © 2003 • Critical Reflections on Information Systems: A Systemic Approach, Jeimy Cano/ ISBN: 1-59140-040-6; eISBN 1-59140-069-4, © 2003 • Web-Enabled Systems Integration: Practices and Challenges, Ajantha Dahanayake and Waltraud Gerhardt ISBN: 1-59140-041-4; eISBN 1-59140-070-8, © 2003 • Public Information Technology: Policy and Management Issues, G. David Garson/ ISBN: 1-59140-060-0; eISBN 1-59140-071-6, © 2003 • Knowledge and Information Technology Management: Human and Social Perspectives, Angappa Gunasekaran, Omar Khalil and Syed Mahbubur Rahman/ ISBN: 1-59140-032-5; eISBN 1-59140-072-4, © 2003 • Building Knowledge Economies: Opportunities and Challenges, Liaquat Hossain and Virginia Gibson/ ISBN: 1-59140-059-7; eISBN 1-59140-073-2, © 2003 • Knowledge and Business Process Management, Vlatka Hlupic/ISBN: 1-59140-036-8; eISBN 1-59140-074-0, © 2003 • IT-Based Management: Challenges and Solutions, Luiz Antonio Joia/ISBN: 1-59140-033-3; eISBN 1-59140075-9, © 2003 • Geographic Information Systems and Health Applications, Omar Khan/ ISBN: 1-59140-042-2; eISBN 159140-076-7, © 2003 • The Economic and Social Impacts of E-Commerce, Sam Lubbe/ ISBN: 1-59140-043-0; eISBN 1-59140-077-5, © 2003 • Computational Intelligence in Control, Masoud Mohammadian, Ruhul Amin Sarker and Xin Yao/ISBN: 159140-037-6; eISBN 1-59140-079-1, © 2003 • Decision-Making Support Systems: Achievements and Challenges for the New Decade, M.C. Manuel Mora, Guisseppi Forgionne and Jatinder N.D. Gupta/ISBN: 1-59140-045-7; eISBN 1-59140-080-5, © 2003 • Architectural Issues of Web-Enabled Electronic Business, Nansi Shi and V.K. Murthy/ ISBN: 1-59140-049-X; eISBN 1-59140-081-3, © 2003 • Adaptive Evolutionary Information Systems, Nandish V. Patel/ISBN: 1-59140-034-1; eISBN 1-59140-082-1, © 2003 • Managing Data Mining Technologies in Organizations: Techniques and Applications, Parag Pendharkar/ ISBN: 1-59140-057-0; eISBN 1-59140-083-X, © 2003 • Intelligent Agent Software Engineering, Valentina Plekhanova/ ISBN: 1-59140-046-5; eISBN 1-59140-084-8, © 2003 • Advances in Software Maintenance Management: Technologies and Solutions, Macario Polo, Mario Piattini and Francisco Ruiz/ ISBN: 1-59140-047-3; eISBN 1-59140-085-6, © 2003 • Multidimensional Databases: Problems and Solutions, Maurizio Rafanelli/ISBN: 1-59140-053-8; eISBN 159140-086-4, © 2003 • Information Technology Enabled Global Customer Service, Tapio Reponen/ISBN: 1-59140-048-1; eISBN 159140-087-2, © 2003 • Creating Business Value with Information Technology: Challenges and Solutions, Namchul Shin/ISBN: 159140-038-4; eISBN 1-59140-088-0, © 2003 • Advances in Mobile Commerce Technologies, Ee-Peng Lim and Keng Siau/ ISBN: 1-59140-052-X; eISBN 159140-089-9, © 2003 • Mobile Commerce: Technology, Theory and Applications, Brian Mennecke and Troy Strader/ ISBN: 1-59140044-9; eISBN 1-59140-090-2, © 2003 • Managing Multimedia-Enabled Technologies in Organizations, S.R. Subramanya/ISBN: 1-59140-054-6; eISBN 1-59140-091-0, © 2003 • Web-Powered Databases, David Taniar and Johanna Wenny Rahayu/ISBN: 1-59140-035-X; eISBN 1-59140-0929, © 2003 • E-Commerce and Cultural Values, Theerasak Thanasankit/ISBN: 1-59140-056-2; eISBN 1-59140-093-7, © 2003 • Information Modeling for Internet Applications, Patrick van Bommel/ISBN: 1-59140-050-3; eISBN 1-59140094-5, © 2003 • Data Mining: Opportunities and Challenges, John Wang/ISBN: 1-59140-051-1; eISBN 1-59140-095-3, © 2003 • Annals of Cases on Information Technology – vol 5, Mehdi Khosrowpour/ ISBN: 1-59140-061-9; eISBN 159140-096-1, © 2003 • Advanced Topics in Database Research – vol 2, Keng Siau/ISBN: 1-59140-063-5; eISBN 1-59140-098-8, © 2003 • Advanced Topics in End User Computing – vol 2, Mo Adam Mahmood/ISBN: 1-59140-065-1; eISBN 1-59140100-3, © 2003 • Advanced Topics in Global Information Management – vol 2, Felix Tan/ ISBN: 1-59140-064-3; eISBN 159140-101-1, © 2003 • Advanced Topics in Information Resources Management – vol 2, Mehdi Khosrowpour/ ISBN: 1-59140-062-7; eISBN 1-59140-099-6, © 2003

Excellent additions to your institution’s library! Recommend these titles to your Librarian! To receive a copy of the Idea Group Publishing catalog, please contact (toll free) 1/800-345-4332, fax 1/717-533-8661,or visit the IGP Online Bookstore at: [http://www.idea-group.com]! Note: All IGP books are also available as ebooks on netlibrary.com as well as other ebook sources. Contact Ms. Carrie Stull at [[email protected]] to receive a complete list of sources where you can obtain ebook information or IGP titles.

Intelligent Agent Software Engineering Table of Contents

Preface ................................................................................................... vi Valentina Plekhanova, University of Sunderland, UK Chapter I Machine Learning for Agents and Multi-Agent Systems ................... 1 Daniel Kudenko, University of York, UK Dimitar Kazakov, University of York, UK Eduardo Alonso, City University, UK Chapter II Architectures for Cognitive and A-Life Agents ................................. 27 Darryl N. Davis, University of Hull, UK Chapter III SoFAR: An Agent Framework for Distributed Information Management ........................................................................................ 49 Luc Moreau, University of Southamptom, UK Norliza Zaini, University of Southampton, UK Don Cruickshank, University of Southampton, UK David De Roure, University of Southampton, UK Chapter IV A Formal Method for the Development of Agent-Based Systems ... 68 P. Kefalas, CITY College, Greece M. Holcombe, University of Sheffield, UK G. Eleftherakis, CITY College, Greece M. Gheorghe, University of Sheffield, UK

Chapter V Engineering Emotionally Intelligent Agents ....................................... 99 Penny Baillie, University of Southern Queensland, Australia Mark Toleman, University of Southern Queensland, Australia Dickson Lukose, Mindbox Inc., USA Chapter VI Evolutionary Computation as a Paradigm for Engineering Emergent Behaviour in Multi-Agent Systems .................................................. 118 Robert E. Smith, The University of the West of England, UK Claudio Bonacina, The University of the West of England, UK Chapter VII The Logic Behind Negotiation: From Pre-Argument Reasoning to Argument-Based Negotiation ........................................................... 137 Luís Brito, Universidade do Minho, Portugal Paulo Novais, Universidade do Minho, Portugal José Neves, Universidade do Minho, Portugal Chapter VIII Designing Agent-Based Process Systems—Extending the OPEN Process Framework ........................................................................... 160 J. Debenham, University of Technology, Sydney, Australia B. Henderson-Sellers, University of Technology, Sydney, Australia Chapter IX Toward an Organization-Oriented Design Methodology for Agent Societies ............................................................................................. 191 Virginia Dignum, Utrecht University, The Netherlands Hans Weigand, Tilburg University, The Netherlands Chapter X Cooperation Between Agents to Evolve Complete Programs ......... 213 Ricardo Aler, Universidad Carlos III Madrid, Spain David Camacho, Universidad Carlos III Madrid, Spain Alfredo Moscardini, University of Sunderland, UK About the Authors .............................................................................. 229 Index ................................................................................................... 238

vi

Preface Intelligent software agents are a unique generation of information society tools that independently perform various tasks on behalf of human user(s) or other software agents. The new possibility of the information society requires the development of new, more intelligent methods, tools, and theories for the modeling and engineering of agent-based systems and technologies. This directly involves a need for consideration, understanding, and analysis of human factors, e.g., people’s knowledge and skill, learning, and performance capabilities, and compatibilities in certain software development environments. This is because software developers utilize their experience and represent their mental models via development and engineering of intelligent agent(s). Therefore, the study of interrelated factors such as people’s and agents’ capabilities, constitutes an important and critical area in intelligent agent software engineering. This should eventually lead to more robust, intelligent, interactive, learning, and adaptive agents. From the theoretical and practical viewpoints, the application of intelligent software agents is a topic of major interest. There has been a growing interest not only in new methodologies for development of intelligent software agents but also in the way in which these methodologies can be supported by theories and practice. Engineering intelligence in a software system is a young area, and there are many issues that need new and further research. Many definitions and types of agents have been introduced in research literature in order to define a common understanding of this phenomenon in modern virtual environments. In fact, these definitions reflect a variety of viewpoints, problems, and applications of the agents, and therefore, they outline the major directions in this important research and practical field. It is recognized that general characteristics of intelligent agents are an agent’s character, reasoning, and learning capabilities, autonomy, reactivity, proactivity, goal orientation, mobility, communication, and cooperation. The intelligent agents can be self-organized and can be integrated into a multiagent system that performs the tasks and solves the problems.

vii

It is important to take into account the specific character of the cognitive tasks. In many cases: • They are dependent on the agents’ knowledge and learning capabilities. • They are managed by the agent (self-managing work), rather than being “administratively” managed. • They cannot be subdivided into smaller subtasks, i.e., cognitive tasks cannot be defined as nested or fine-grained tasks. • They can be performed in parallel. In addition, agents can be available and capable of overseeing cognitively driven tasks, but factors such as knowledge and technical, may be incompatible. Agent resource capability and compatibility have become the focus of agentbased software engineering. These factors have considerable impact on the formation of a multiagent system (i.e., a team of intelligent agents) for cognitive tasks, i.e., capability and compatibility factors define an integration problem in engineering the task of intelligence in software systems. Because agent availability issues do not contribute much to the success of performance, there is a risk of selecting agents to do work because they are available, not because they have a particular combination of key capability and compatibility factors. In addition, (single) agents can be available and capable of overseeing the individual cognitive tasks, but factors such as the knowledge of agents and their implementation may not be compatible with the goals and constraints of a multiagent system’s application. Hence, managerial aspects such as intelligent capability and compatibility evaluation have to be an important focus of intelligent agent software engineering and improvement of agent-based processes. However, for cognitive tasks involving agents’ capabilities and compatibilities as critical variables, existing approaches to the definition of resource capability and compatibility do not sufficiently examine how effectively agent resources fit the needs for their implementation. The same cognitive tasks may be performed by different (single) agents or by a multiagent system using different capabilities, which correspond to different performance constraints (e.g., costs, risk, duration, productivity). Moreover, for completeness, data regarding the intelligent capabilities of agents and performance capabilities should be provided in connection with the learning methods utilized by agents to perform their cognitive tasks. This book consists of work where intelligent agent software engineering is considered as the application of the integration of formal methods and heuristic approaches to ensure support for the evaluation, comparison, analysis, and evolution of agent behavior.

viii

MISSION OF THE BOOK This book does not attempt to teach any software/information technology engineering techniques. It does not claim to be comprehensive. There are many books on the market that present the classical approaches to software engineering and process modeling. Instead, it is designed to be used to support solutions to problems in developing and engineering the agent-based software/information technology and in engineering the intelligence in software systems. Available books on intelligent systems and software/information technology engineering focus too much on the technical problems in software engineering. They provide limited coverage of the critical factors in engineering the intelligence in software systems, such as agent intelligent capability and compatibility, motivation, and agent’s capability integration problems. In addition, little attention has been placed on the problems of engineering the intelligence in a multiagent system. Furthermore, we consider intelligent agent software engineering on a multiagent system’s level, not only on the (single) agent level, and define the task of intelligent agent software engineering not only as a software engineering task, but also as a task of engineering the intelligence in a multiagent system. This book is an attempt to fill these gaps.

AIMS OF THE BOOK •

•

• •

•

The primary aims of this book are as follows: To introduce the readers to the concept of intelligent agent software engineering so that they will be able to develop their own agents and implement concepts in practice, i.e., in unique conditions, constraints, and environments (The book covers theoretical and practical aspects.) To present current research in the area of engineering the intelligence in software systems—in this book, we consider engineering the intelligence in software systems as the application of mathematical techniques and thorough engineering methods to the development of intelligent agents To offer methodologies that provide guidelines and procedures for the development of intelligent agents by applying various theories and techniques, e.g., machine learning methods, cognitive science, and the object-oriented paradigm To describe current problems in engineering the intelligence in software systems, e.g., evaluation and application of contemporary machine learning technology, application of object-oriented methodology to the development of intelligent agents, etc. To define future work and directions in the area of intelligent agent software engineering

ix

THE AUDIENCE FOR THE BOOK We envisage a number of distinct audiences for this book that make it useful and important. This book is for readers who have interests in developing intelligent software agents. It is suitable for practitioners, who want to use state-of-the-art techniques in the management and engineering of intelligence in software systems and the production of high-quality software. And, it is directed to the academics, advanced undergraduate or postgraduate students interested in new perspectives and directions in engineering the software development processes and intelligent agents and systems. Academics may also find this book useful as a course book for specific courses on advanced software (IT) engineering and intelligent systems. This book is useful as an advanced text for final-year undergraduate students of computer science, intelligent systems, software engineering and information technology, or as part of postgraduate masters and doctoral programs.

OUTLINE OF THE BOOK The book is organized into ten chapters as follows. Chapter 1 (Daniel Kudenko, Dimitar Kazakov, and Eduardo Alonso) discusses the differences between pure Machine Learning and the one performed by (single) Learning Agents. The authors describe critical aspects in machine learning technology and discuss the relationship between learning and communication, learning to collaborate and compete, the learning of roles, evolution and natural selection, and distributed inductive learning, These discussions are followed by a number of general recommendations for learning agent designers and some practical applications of machine learning to agents and multiagent systems. Two case studies being developed at the University of York are also presented. Chapter 2 (Darryl N. Davis) introduces research into the nature of drives and motivations in computational agents from a perspective drawing on artificial life and cognitive science. The background to this research is summarized in terms of the possibility of developing artificial minds. A particular cognitive architecture is described in terms of control states. Steps toward producing an implementation of this architecture are described by means of experimentation into the nature of specific control states. The design of a simple A-life architecture for a predator– prey scenario is described using a transition state diagram. This architecture is then used as a platform with which to develop an agent with drives. This second architecture is used to develop an agent with explicit motivations. The discussion of these (and other) experiments shows how these simple architectures can help to provide some answers to difficult research questions in cognitive science.

x

Chapter 3 (L. Moreau, N. Zaini, D. Cruickshank, and D. De Roure) presents a versatile multiagent framework (named SoFAR, the Southampton Framework for Agent Research) designed for Distributed Information Management tasks. SoFAR embraces the notion of proactivity as the opportunistic reuse of the services provided by other agents and provides the means to enable agents to locate suitable service providers. The contribution of SoFAR is to combine ideas from the distributed computing community with the performative-based communications used in other agent systems: communications in SoFAR are based on the start point/end point paradigm, a powerful abstraction that can be mapped onto multiple communication layers. SoFAR also adopts an XML-based declarative approach for specifying ontologies and agents, providing a clear separation with their implementation. Chapter 4 (P. Kefalas, M. Holcombe, G. Eleftherakis, and M. Gheorghe) introduces a detailed and comprehensive account of the ways in which some modern software engineering research can be applied to the construction of effective and reliable agent-based software systems. More specifically, the authors show how simple agents motivated from biology can be modeled as X-machines. Such modeling facilitates verification and testing of an agent model, because appropriate strategies for model checking and testing are already developed around the Xmachine method. In addition, modular construction of agent models is feasible, because X-machines are provided with communicating features that allow simple models to interact. Chapter 5 (P. Baillie, M. Toleman, and D. Lukose) examines the engineering of a mechanism that synthesizes and processes an artificial agent’s internal emotional states: the Affective Space. Through use of the Affective Space, an agent can predict the effect that certain behaviors will have on its emotional state and, in turn, decide how to behave. Furthermore, an agent can use the emotions produced from its behavior to update its beliefs about particular entities and events. This chapter explores the psychological theory used to structure the Affective Space, the way in which the strength of emotional states can be diminished over time, how emotions influence an agent’s perception, and the way in which an agent can migrate from one emotional state to another. Chapter 6 (R.E. Smith and C. Bonacina) describes new implications of Evolutionary Computation theories and practices. Agents that interact according to these theories are no longer locked inside the laboratory conditions imposed by Evolutionary Computation researchers and users. Thus, it is important that authors consider the embodiment of Evolutionary Computation in “real” agents, that is, agents that involve the real restrictions of time and space within a multiagent system. This issue is addressed in two ways. First, authors have developed a platform independent agent system to assure that we work within the generic, but

xi

realistic, constraints of agents. Second, authors have developed an agent-based system of Evolutionary Computation objects. The prime thrust of this research with these tools is to facilitate understanding of Evolutionary Computation within agents and understanding of more general agent interactions in the light of Evolutionary Computation theories. This chapter presents the platform independent agent system, along with the generic software framework for Evolutionary Computation in a multiagent system. Chapter 7 (L. Brito, P. Novais, and J. Neves) discusses the use of agents in Electronic Commerce (EC) environments. A four-step method is introduced for developing EC-directed agents, which are able to take into account nonlinearites such as gratitude and agreement. Negotiations that take into account a multistep exchange of arguments provide extra information, at each step, for the intervening agents, enabling them to react accordingly. This argument-based negotiation among agents has much to gain from the use of Extended Logic Programming mechanisms. Incomplete information is common in EC scenarios; therefore, arguments must also take into account the presence of statements with an unknown valuation. Chapter 8 (J. Debenham and B. Henderson-Sellers) expands on objectoriented (OO) methodologies to adequately support agent-oriented technology. It is recognized that OO methodologies are highly prescriptive and overly specified and are hard to extend when a new variant or a new paradigm appears. Authors consider OPEN (Object-oriented Process, Environment, and Notation) as a more flexible approach to building methodologies or processes. Using OPEN, process components are selected from a repository, and the actual methodology (or process) is constructed using identified construction and tailoring guidelines. Proposed extensions (new Tasks and Techniques for OPEN) are illustrated in two case studies of business processes. Goal orientation is readily accommodated in the existing and new tasks and techniques of the OPEN process, leading to a specification of the individual agents in the system. Chapter 9 (V. Dignum and H. Weigand) proposes a framework that describes all the stages of development of a multiagent system, takes an organizational perspective on systems design, and specifies all the development steps for the design and development of an agent-based system for a particular domain. Specific agent-oriented methodologies can be used for the development and modeling of each of the development steps. Authors believe that such a generic framework, based on the organizational view, will contribute to the acceptance of multiagent technology by organizations. Authors define a social framework for agent communities based on organizational coordination models that “implements” the generic interaction, cooperation, and communication mechanisms that occur in the problem domain. The proposed methodology allows a generic coordination

xii

model to be tailored to a given application and to determine its specific agent roles and interactions. Chapter 10 (R. Aler, D. Camacho, and A. Moscardini) concludes this book with a multiagent system approach, with its purpose to build computer programs. Each agent in the multiagent system will be in charge of evolving a part of the program, which in this case, can be the main body of the program or one of its different subroutines. There are two kinds of agents: the manager agent and the genetic programming (GP) agents. The former is in charge of starting the system and returning the results to the user. The GP agents include skills for evolving computer programs, based on the genetic programming paradigm. There are two sorts of GP agents: some can evolve the main body of the program, and the others evolve its subroutines. Both kinds of agents cooperate by telling each other their best results found so far, so that the search for a good computer program is made more efficient. In this chapter, this multiagent approach is presented and tested empirically. Valentina Plekhanova, PhD University of Sunderland, UK

xiii

Acknowledgments

Many people made this book possible. Foremost on the list of acknowledgements, of course, are the chapter authors, who were not only contributors to the new book but also performed extra work as reviewers of other’s chapters. Particular gratitude is due to the publishing team at Idea Group Publishing. In particular to Jan Travers and Michele Rossi, who continuously prodded via e-mail to keep the project on schedule, and to Mehdi Khosrow-Pour, whose enthusiasm motivated me to initially accept his invitation to take on this project. A further special note of thanks goes also to the staff at Idea Group Publishing, whose contributions throughout the whole process from inception of the initial idea to final publication have been invaluable. Their efforts are greatly appreciated and will never be forgotten.

Valentina Plekhanova

Machine Learning for Agents and Multi-Agent Systems 1

Chapter I

Machine Learning for Agents and Multi-Agent Systems Daniel Kudenko University of York, UK Dimitar Kazakov University of York, UK Eduardo Alonso City University, UK

ABSTRACT In order to be truly autonomous, agents need the ability to learn from and adapt to the environment and other agents. This chapter introduces key concepts of machine learning and how they apply to agent and multiagent systems. Rather than present a comprehensive survey, we discuss a number of issues that we believe are important in the design of learning agents and multi-agent systems. Specifically, we focus on the challenges involved in adapting (originally disembodied) machine learning techniques to situated agents, the relationship between learning and communication, learning to collaborate and compete, learning of roles, evolution and natural selection, and distributed learning. In the second part of the chapter, we focus on some practicalities and present two case studies.

Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

2 Kudenko, Kazakov & Alonso

INTRODUCTION Intelligence implies a certain degree of autonomy, which in turn, requires the ability to make independent decisions. Truly intelligent agents have to be provided with the appropriate tools to make such decisions. In most dynamic domains, a designer cannot possibly foresee all situations that an agent might encounter, and therefore, the agent needs the ability to learn from and adapt to new environments. This is especially valid for multi-agent systems, where complexity increases with the number of agents acting in the environment. For these reasons, machine learning is an important technology to be considered by designers of intelligent agents and multi-agent systems. The goal of this chapter is not to present a comprehensive review of the research on learning agents (see Sen & Weiss, 1999, for that purpose) but rather to discuss important issues and give the reader some practical advice in designing learning agents. The organization of the chapter is as follows. In the following section, the differences between pure machine learning and that performed by (single) learning agents are discussed. We start with the introduction of basic machine learning concepts, followed by examples of machine learning techniques that have been applied to learning agents, such as Q-learning, explanation-based learning, and inductive logic programming. In the third section, we discuss several issues surrounding multi-agent learning, namely, the relationship between learning and communication; learning to collaborate and compete; the learning of roles, evolution, and natural selection; and distributed inductive learning. Following this discussion, we focus on some practicalities and present two case studies. We finish the chapter with conclusions and further work.

FROM MACHINE LEARNING TO LEARNING AGENTS In this section, we discuss the nature of machine learning (ML), its integration into agents, and the parallels between machine learning systems and learning agents. We start with a basic introduction to machine learning. While most of the fundamental ML concepts introduced below are commonly associated with supervised learning (SL) (i.e., the generalization from annotated examples provided by a teacher), they are equally relevant for reinforcement learning (RL), where an agent learns through the feedback (i.e., reinforcement) from the environment in each entered state. To date, most attention in agent learning has been reserved for RL techniques such as QCopyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


learning (see below), due to its suitability to situated agents. Nevertheless, we see SL and RL as strongly related approaches. In the case of RL, the environment could be seen as the teacher, and the generalization process would be over states (which correspond to the examples). In fact, SL methods can be directly applied in a RL setting (see, e.g., the use of neural networks in TD learning (Tesauro, 1992). Note that we mostly exclude a third class of ML techniques, unsupervised learning (or learning by discovery), from our discussion, because there are only few research results in this area to date.

Introduction to Machine Learning A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E (Mitchell, 1997). Machine learning can be seen as the task of taking a set of observations represented in a given object (or data) language and representing (the information in) that set in another language called the concept (or hypothesis) language. A side effect of this step can be the ability to deal with unseen observations. As an example, one can consider an object language that consists of pairs of coordinates (x,y). A number of observations are provided, and each is labeled as a positive or negative example of the target concept, i.e., the concept to be learned. Let the hypothesis language define a concept as an ellipse,1 such that a point (x,y) would represent a positive example of the concept if it is inside that ellipse or a negative one otherwise. In the example in Figure 1, there are infinitely many such ellipses, each of which would satisfy the usual requirements for completeness and consistency, i.e., it would cover all positive examples and none of the negative. The set of all complete and consistent hypotheses for the definition of the target concept is referred to as the version space for that concept.

Selecting the Agent’s Learning Bias The hypothesis language specifies the language bias, which limits the set of all hypotheses that will be considered by the learner. In the example in Figure 1, it is obvious that there are certain arrangements of positive and negative examples on the plane that can never be separated by an ellipse in the required way. A priori, all concepts in the version space are equally good candidates for the description of the training data provided. Only when additional test data



Figure 1: Object and concept language.

Most specific concept

-

+

-

+ + +

-

Most general concept

Version space

is available can one find that certain concepts in the version space classify unseen members of the target concept better than others. A preference bias is a principle that allows the learner to choose between two hypotheses if they both classify the training data equally. If hypotheses from the entire version space are to be ranked according to the preference bias, the order in which they are explored is of no importance. Alternatively, a search bias can be introduced to specify that order, and the search can stop after the first acceptable hypothesis with respect to a given criterion is found. The appropriate choice of language, preference, and search biases is crucial for the learning outcome. The language bias has to be sufficiently general to include the target concept, yet restricted enough to make the search feasible. A well-chosen search bias can significantly speed the learning. At the same time, specifying the way in which the hypothesis space is searched can result in a preference bias being enforced, as in the cases when search starts with the most general, or the most concise, hypotheses. The choice of preference bias can be guided by some intuition about the type of hypothesis likely to be successful for the given type of data, or by some general principle, such as Occam’s razor, favoring the hypothesis using the minimum number of entities, i.e., the simplest or shortest one, and the Minimal Description Length (MDL) principle, which recommends “the shortest method for re-encoding the training data, where we count both the size of the hypothesis and any additional cost of encoding the data given this hypothesis” (Mitchell, 1997).



Selecting the Agent’s Learning Technique Machine learning techniques can be classified according to a number of criteria. Inductive learning has the power to produce theories that are more general than the initial information provided, whereas deductive learners only specialize, i.e., restrict the coverage of existing theories. White box learners generate hypotheses that can be interpreted — and modified — by the user; the opposite is true for black box learners, which can be used if understanding of or changes in the hypothesis learned are not necessary. Incremental learning updates the current hypothesis when provided with additional examples, whereas batch learning has to process the whole data set again. Both can be used in an agent, but time constraints and quality of learning have to be taken into account, and the benefits of each method must be compared. Eager learning generates a hypothesis from the training data, which is then thrown away, and the hypothesis used unchanged for all test examples; for each test example supplied, lazy learning generates a separate hypothesis. One can see that eager learning saves memory, but, in comparison with lazy learning, requires more extensive computation to be done before test examples are supplied. As mentioned above, the data used for learning can belong to one of three classes. Annotated data assigns to each observation a label, which can contain the name of the concept or some other information associated with the observation. In this case, known as supervised learning, the learning task is to find a hypothesis that, given the observation, will produce the label. Unsupervised learning deals with nonannotated data. Here, no explicit target concepts are given, and the aim of learning is to change the data representation in a way that highlights regularities in the data. Finally, in reinforcement learning, only some of the training examples are annotated (with the associated immediate rewards), and the learner has to propose the annotation of the remaining ones. From another point of view, one could also argue that reinforcement learning is a specific case of unsupervised learning, because the optimal action with respect to the agent’s goal (e.g., maximum discounted cumulative reward) and time horizon (often infinite) is not explicitly given.

Parallels between Machine Learning and Learning Agents Machine learning has gone through different stages with respect to the way in which training data is selected. In the classic setting, the ML algorithm does not choose the training data, which is provided by the user. Active learning



(Thompson et al., 1999) aims at minimizing the need of hand-annotated data. It starts with a training data set of which only a small part is annotated, and then it gradually requests manual annotation for those remaining training examples likely to be the most helpful for the learning process. Close loop machine learning (CLML) (Bryant & Muggleton, 2000) eliminates the need of an external annotator, and instead, autonomously plans and performs experiments based on its current hypothesis in order to obtain additional training data when necessary. If we were to compare machine learning and learning agents, CLML would be considered a borderline case. CLML is, indeed, a ML technique, the implementation of which could also be seen as an autonomous, rational, and deliberative agent, with the single aim of learning. In the general case of learning agents though, learning is just one of many goals that an agent has to achieve within its limited resources, and additional issues, such as time constraints, have to be taken into account. The remaining part of this section will introduce a few learning algorithms of specific relevance to learning agents.

Q-Learning Q-learning, a reinforcement learning technique, is the most widely used learning method for agent systems. The goal of Q-learning is to compute a stateto-action mapping (a so-called policy) that leads to the maximum utility for the agent. The policy generation is based on a numerical reward feedback from the environment after each action execution. In other words, the agent learns by exploring the environment, i.e., experimenting with different actions and observing the resulting reward. The result of Q-learning, i.e., the policy, may be seen as a table that assigns each state–action pair (s,a) a numerical value, which is an estimate of the (possibly long-term) reward to be received when executing a in s. After receiving a reward, an agent updates the numerical value of the state–action pair based on the reward and on the estimated best reward to be gained in the new state. Thus, with time, the agent is able to improve its estimates of the rewards to be received for all state–action pairs. Due to space restrictions, we do not present details of the Q-learning algorithm but rather refer the reader to Mitchell (1997) or Kaebling et al. (1996). The advantages of Q-learning are its efficiency, guaranteed convergence toward the optimum, and natural applicability to agent systems because of the coupling of learning and exploration. But even though most research on learning agents has been concentrated on Q-learning, it has a few drawbacks: (1) defining a suitable numerical reward function can be a nontrivial task for some Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


application domains; (2) convergence to the optimal (i.e., utility-maximizing) policy requires that all state–action pairs be visited infinitely often, which obviously leads to problems with large state–action spaces in real-world domains (note that the choice of exploration strategy influences the convergence speed); and (3) the learning result (a table assigning each state–action pair a numerical value reflecting the expected reward) is not transparent in the sense that no explanation is given for action preferences. Nevertheless, given a state, actions can be ranked, which may yield some limited insight into the performance of the agent.

Explanation-Based Learning Computing a solution (e.g., a plan) directly from basic principles is a hard problem in terms of complexity. Explanation-based learning (EBL) adds potentially useful macro-operators to the knowledge base and thus speeds the search process for an optimal solution. EBL has been widely used in artificial intelligence to speed the performance of planners [e.g, in Prodigy (Carbonell et al., 1990)]. Generally speaking, the agents are concerned with improving the efficiency of the problem solver rather than acquiring new knowledge. Obviously, problem solvers, when presented with the same problem repeatedly, should not solve it in the same way and amount of time. On the contrary, it seems sensible to use general knowledge to analyze, or explain, each problem-solving instance in order to optimize future performance. This learning is not merely a way of making a program run faster but also of producing a more accurate hypothesis. In short, EBL extracts general rules from single examples by generating an explanation for the system’s success or failure and generalizing it. This provides a deductive (rather than statistical) method to turn first-principles knowledge into useful, efficient, special-purpose expertise. The learned rules enable the planner to make the right choice when a similar situation arises during subsequent problem solving. We will see a practical example in a conflict simulation domain in Section 4.

Inductive Logic Programming Inductive logic programming (ILP) is a white-box learning method based on the use of induction, logic programming, and background knowledge (Muggleton & De Raedt, 1994). In more detail, ILP uses a subset of first-order predicate logic known as Horn clauses as its object and concept language. The object language is at times further restricted to ground facts, i.e., to a subset of Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


Table 1: ILP Object Language Example Good bargain cars

ILP representation

Model

Mileage

Price

yes/no

gbc(#model,+mileage,+price).

BMW Z3

50000

£5000

yes

gbc(z3,50000,5000).

Audi V8

30000

£4000

yes

gbc(v8,30000,4000).

Fiat Uno

90000

£3000

no

gbc(uno,90000,3000).

propositional logic, in which case, the examples of the target predicate have a simple relational representation (see Table 1). The concept language in this example could be restricted to the basic relations provided with the ILP learner, e.g.: equal(X,X). greater(X,Y) :- X > Y. The language of Horn clauses is more powerful than propositional logic, as the latter, for instance, has to express the concept of equality between the values of two arguments as a list of pairs of values: (arg1=1 & arg2=1) or (arg1=2 & arg2=2) ... which is inefficient in the case of finite domains and impossible otherwise. In most ILP systems, one can learn one target concept at a time, but there are some notable exceptions (Progol5). Each of the attributes of the target concept has to be defined by its type, defining the range of values, and mode, describing whether in the concept definition that attribute will be an input variable, an output variable, or a constant (see last column of Table 1). Apart from the standard, built-in concepts, the concept language in ILP can be extended with user-defined concepts or background knowledge. The use of certain background predicates may be a necessary condition for learning the right hypothesis. On the other hand, redundant or irrelevant background knowledge slows the learning. To continue with our example, one may consider helpful a concept that associates with a car the product of its mileage and price



and compares it with a fixed threshold T. In logic programming speak, this can be represented as: prod(Miles, Price, T) :- Miles * Price < T. Now, one of the possible definitions of a good bargain BMW Z3 can be expressed as a car of that model for which the product mentioned is below 250000001: gbc(z3, Miles, Price) :- prod(Miles, Price, 250000001).

The preference bias in ILP is typically a trade-off between the generality and the complexity of the hypothesis learned; some ILP learners allow the users to specify their own bias.

MACHINE LEARNING FOR MULTI-AGENT SYSTEMS Learning becomes a much more complex task when moving from a single agent to a multi-agent setting. The environment becomes more dynamic and less predictable due to many (potentially adversarial) agents acting in it, each equipped with its own goals and beliefs. New issues arise such as coordination in teams of cooperating agents and competition with adversarial agents, all of which can (and should) be improved by learning. In this section, we present several important issues concerning the application of ML in a multi-agent setting. First, we contrast the straightforward approach of directly transferring single-agent learners into a multi-agent domain with the more complex approach of designing learning agents with a social awareness. We continue with a discussion of two major learning goals, namely, learning to coordinate and learning to compete. We then present issues surrounding communication, team roles, and evolution for learning, all of which are highly relevant in a multi-agent domain. Finally, we take a brief look at distributed learning and its application to data mining. The area of multi-agent learning is still young, and therefore, many of the questions we present have no definitive answer yet. Nevertheless, they are important considerations for the designers of learning agents.



Multiple Single-Agent Learning Versus Social MultiAgent Learning An obvious question is why not use the same single-agent learning techniques discussed in the previous section directly (i.e., without further modification) in a multi-agent setting, an approach we call Multiple SingleAgent Learning. In that case, a learning agent would perceive other agents only as a part of the environment and have no explicit awareness of their existence, let alone their goals and beliefs. Nevertheless, changes in the environment due to the actions of other agents are still being perceived, and thus, a model of other agents can be acquired indirectly during the learning process as part of the environment model. In contrast, agents can have a high awareness of other agents and incorporate this knowledge in the learning process, potentially using communication, coordination, and agent modeling techniques to support the learning task. While this Social Multi-Agent Learning approach is certainly more complex, does it necessarily lead to improved learning performance? First, it is interesting to note that for nonlearning agents, social awareness is not necessary to achieve near-optimal behavior, as has been shown in experiments in a simple foraging domain (Steels, 1990). For learning agents, initial experiments in a simple two-player cooperative game setting (Claus & Boutillier, 1998; Mundhe & Sen, 2000) show the surprising results that social awareness is not always beneficial in learning and may even hurt performance under certain conditions. The research draws on a classification of different levels of social awareness (Vidal & Durfee, 1997): level-0 agents have no knowledge and awareness of other agents, while level-k agents model other agents as having level at most (k-1). For example, a level-1 agent uses a model of the other agents, but this model assumes that other agents are of level 0, i.e., their actions are not influenced by any model of the level-1 agent (i.e., they are not choosing their actions based on direct observations of the agent’s past actions). Mundhe and Sen looked at the behavior of Q-learning agents of social awareness levels 0, 1, and 2 and the impact of these levels on convergence in terms of speed and effectiveness of the result. Their experiments show that two level-1 agents display the slowest and least effective learning, worse than two level-0 agents. While these results are a first indication that social awareness may decrease learning performance, the experiments have been carried out in a rather simple setting, and there is the need for further research into these issues in order to be able to give more general guidelines to the agent designer.



Learning to Coordinate and to Compete Characteristically, agents must learn to collaborate and to compete in multi-agent domains. Not only do the players have to learn low-level skills, but they must also learn to work together and to adapt to the behaviors of different opponents. Team games are a rich domain for the study of collaborative and adversarial multi-agent learning. Teams of players must work together to achieve their common goal, while at the same time defending against the opposing team. Learning is essential for this task, because the dynamics of the system change due to changes in the opponents’ behaviors. Stone and Veloso (1998) illustrate how agents learn to cooperate and compete in a robotic soccer scenario: The passer in (robotic) soccer would need to pass the ball in such a way that the shooter could have a good chance of scoring a goal. The parameters to be learned by the passer and the shooter are the point at which to aim the pass and the point at which to position itself, respectively. At the same time, as teammates are cooperating and passing the ball among themselves, they must also consider how best to defeat their opponents. As time goes on, the opponents need to co-evolve in order to adjust to each other’s changing strategies.

Reinforcement Learning for Multi-Agent Systems In most approaches to multi-agent learning, reinforcement learning (specifically Q-learning) has been the method of choice for the same reasons as in a single-agent setting (see also the previous section). While reinforcement learning can usually be applied to MAS straightforwardly in a multiple singleagent learning setting (e.g., Crites & Barto, 1998), problems arise when agents working in a team are not able to observe each others’ actions all of the time. Cooperating agents often receive rewards globally as a team for the combined actions rather than locally for individual actions. When updating the Q table based on such a reward and in the absence of information about the other agents’ actions, an agent has to use an heuristic to determine the contribution of each agent to this reward. Lauer and Riedmiller (2000) present two such heuristics: • Pessimistic assumption: Given my action, the other agents always perform actions that yield the minimum payoff for the team. • Optimistic assumption: Given my action, the other agents always perform actions that yield the maximum payoff for the team.



If each agent uses the pessimistic assumption, then this leads to overly cautious behavior and thus to slow convergence toward the optimal policy (without guarantee of convergence). Lauer and Riedmiller show that an optimistic assumption for each agent leads to guaranteed convergence. Nevertheless, no results have been presented on convergence speed. Note that Lauer and Riedmiller’s approach does not require any communication among the agents. We look at this and the resulting improvements in learning performance in the following subsection.

Learning and Communication When several learning agents work in a team, it may be beneficial for them to cooperate not just on the task achievement but also on the learning process. Clearly, communication is an important tool for such cooperation. Tan (1993) considered the following types of communication between Q-learning agents in the hunter–prey domain: • Sharing sensation: Each agent has only a limited sensory range that can be extended by each agent communicating its sensory information to the others. • Sharing policies: All agents share the same policy in the form of a blackboard. Each agent is permitted to update the policy based on its experience. • Merging policies: After a certain number of moves, the policies of two or more agents are merged, e.g., by averaging the values in the Q tables. • Sharing episodes: After a successful completion of an episode, the corresponding action–reward sequence is communicated to all other agents, who use it as a training episode. As expected, empirical results show that communication during Q-learning speeds up the convergence. Nevertheless, in some cases, sharing sensory information can detract an agent’s learning from the correct hypothesis and hurt the learning process. In related work by Provost and Hennessy (1996), a team of inductive rule learners generate a joint hypothesis. The individual learners communicate the individually computed classification rules, and the other agents critique them based on their local training data. Only those rules that receive a positive feedback are used in the final hypothesis. The above examples show the use of low-level communication that is mainly an exchange of data. To date, higher-level communication, such as



indications of intentions or negotiation of learning targets has not been looked at in depth in the context of multi-agent learning. We are currently developing role learning techniques that incorporate an explicit communication and negotiation for the coordination of learning in a team.

Learning of Roles When working in a team, specialization of team members on specific tasks or task categories may be beneficial. One way to achieve this kind of team heterogeneity in a MAS is to equip agents with different behaviors or sensor and effector capabilities and thus predefine the roles that they are going to play in the team. While this method may lead to good results (Parker, 1994), it has a number of drawbacks. First, it is not always obvious how to specify an optimal (or even useful) distribution of behaviors. Second, it may be quite expensive (in terms of hardware or in terms of development time) to design a system of heterogeneous agents. An alternative is to use a team of learning agents that are homogeneous to begin with but with time and experience will diversify and specialize. There are two main questions that a learning MAS designer faces: (1) How can agent specialization be achieved by learning, and (2) does the application benefit from such team heterogeneity? While as yet there are no definitive answers to these questions, research results are available that shed some light on them. Prasad et al. (1996) present a method to learn the optimal distribution of predefined roles (i.e., behaviors) in a team. While this is not role learning per se, it is a first noteworthy approach to answer the above questions. Balch (1999) studied the conditions under which a team of agents based on reinforcement learning will converge toward heterogeneity. In his research, he distinguishes between two main types of reward functions: local, where each agent receives rewards individually for personally achieving a task; and global, in which all team members receive a reward when one of the team members achieves a task. Empirical results show that globally, reinforced agents converge toward a heterogeneous team, while local reinforcement leads to homogeneous agents. Furthermore, learning heterogeneity is not always desirable: in multirobot foraging, a locally reinforced and therefore homogeneous team outperforms a globally reinforced and therefore heterogeneous team. On the other hand, in the robotic soccer domain, Balch’s results are the opposite, i.e., global reinforcement (and thus heterogeneity) yields the better performance.



In related work, Crites and Barto (1998) show that a team of elevators that are locally reinforced do not display heterogeneous behavior, but they nevertheless perform highly effectively. Tumer and Wolpert (2000) discuss the connection of local rewards and the total world utility. Their wonderful life utility (WLU) permits agents to remove the noise and uncertainty related to the activity of other agents and focus on how the agent contributes to the world utility. This WLU has been shown to result in higher world utilities when used in teams of RL agents (as compared to global rewards). While these results do not provide a general answer to the when and how of learning roles, they show that team heterogeneity is an important factor in multi-agent performance, and that different learning methods can yield different levels of heterogeneity. It is an interesting open research problem to gain a clearer and more detailed understanding of the relationship between agent diversity and performance and learning in general.

Natural Selection, Language, and Learning The task of any learning agent is to modify the agent’s knowledge under the guidance of some fixed metaknowledge contained in the language, preference, and search biases. Introducing several metalevels does not change the fact that the top one will still have to be fixed and provided from a source external to the learner. That source can be the agent’s creator, but the bias can also be the product of evolution. In the latter case, the bias will be set at random in a population of agents, in which the principles of natural selection are applied, so that the “fitter” the agent, i.e., the better it achieves its goals, the higher the chance that some aspects of its bias will be used in the new agents introduced in the environment. The exact way in which these aspects are propagated into the agent’s “offspring” can be modeled with the standard notions of genetic crossover and mutation, so that the bias of every new agent is selected as a combination of the suitably represented biases of its two parents, to which small random changes are introduced. Whether knowledge and skills learned by an individual can influence the inherited features it will pass on to the offspring, has been the subject of many theories. According to Darwin, individual experience cannot change one’s genes; Lamarck believed the opposite possible. While Darwin’s theory has been universally accepted to reflect the true mechanisms of inheritance in nature, there are no reasons why one should not experiment with Lamarckian evolution among artifacts. The way in which Darwinian evolution is related to learning is captured in the so-called Baldwin effect (Baldwin, 1896). The effect



predicts that the evolution of new behavior goes through two phases. Initially, learning is favored by natural selection, as it helps to patch the inherited behavior, while the latter is still incomplete or suboptimal. In stable environments, as inherited behavior gets closer to the optimal, it gradually displaces learning to avoid the cost associated with the latter. One can represent the overall trend as a tendency to strengthen the language bias, so that search space is pruned to eliminate bad hypotheses and zoom in on the likely winners (Turney, 1996). From a MAS designer’s point of view, Lamarckian evolution is faster but brings the risks of inheriting too-specific concepts based on the parents’ personal experiences that have no analogue in the life of the offspring. One could expect from Darwinian evolution to encode as inherited only general concepts, as they would have to have remained for many generations relevant to a substantial part of the population. There is, however, a third way open to populations of agents able to communicate. Language uses concepts that are specific enough to be useful in the description of a variety of aspects of the agent’s environment (including other agents), yet general enough to correspond to shared experience. In this way, the concepts of a shared language serve as a language bias, which is inherited by the individuals through upbringing rather than genes. To preserve the additional advantage that the use of language brings about in the case of a changing environment, one should allow the language to evolve along with all other inherited features.

Distributed Inductive Learning When learning over large or physically distributed training data sets, using just one agent to compute a hypothesis is no longer feasible. In such cases, several learning agents, each receiving a subset of the training data, are needed to compute a global hypothesis for the complete training data set. Recently, many application areas for this kind of distributed inductive learning have emerged. First and foremost, distributed learning is a highly effective way of solving data mining problems caused by increasing data set sizes of physically distributed data (e.g., on the Internet) (Provost & Kolluri, 1999). In robotic soccer, player agents have only limited local observations available in order to induce global properties of the game, e.g., the strategy of the opposing team (we are currently working on distributed inductive learning algorithms that achieve this task). Current distributed inductive learning methods are employing one of two general approaches:



•

•

Hypothesis combination: Each agent individually computes a local hypothesis based on the local training data that are then combined into a global hypothesis, by a separate agent (e.g., Fayyad et al., 1993) or collaboratively by the local agents (e.g., Provost & Hennessy, 1996). Hypothesis update: One agent starts by inducing a local hypothesis from the local training data and then communicating it to another agent, which updates its hypothesis based on its own local data. An instance of this incremental batch learning approach can be found, e.g., in Domingos (1996).

There is still plenty of room for further research in distributed inductive learning, specifically in the area of explicit coordination during the learning process (which is related to research discussed in the subsection on communication).

INTEGRATION OF ML INTO MULTI-AGENT SYSTEMS General Issues As discussed earlier, the application of ML to agents involves many issues. Whereas the most important issue for a stand-alone machine learning algorithm is the quality of the theory learned, in a learning agent, the time needed for learning becomes a primary factor. The fact that computational resources of an agent have to simultaneously maintain several tasks, such as perception, planning, and control, imposes time constraints on learning. These constraints are relatively flexible, because individual tasks can be rescheduled, as long as the hard constraints imposed by the environment are met (find food or die, run or get eaten, etc.). Time constraints can be a significant factor in the choice of the learning strategy. In comparison to lazy learning, eager learning typically results in more compact theories, which are faster to use, but take more time to learn. A combination of both methods is also possible, so that observations are minimally preprocessed on the fly, and the computationally expensive part of the learning is postponed until sufficient time is available. For instance, certain hypothesis languages, such as first-order decision lists (Kazakov & Manandhar, 2001) permit all new observations that are not handled correctly by the current hypothesis to simply be added to a list of exceptions, as a lazy learner would do. At a later stage, an eager learning algorithm can be used to replace that list with a more compact hypothesis, if at all possible.



Apart from the time complexity and accuracy of the results, another consideration in choosing a particular learning algorithm is whether its worstcase complexity is known or not. For learning algorithms that explore the entire hypothesis, space finding an optimal solution is guaranteed after running a prescribed number of steps. Other algorithms, sometimes known as any-time algorithms, cannot recognize an optimal solution when they see one but can only compare the relative merits of two candidates. As a result, a longer run always has the potential to improve the best solution found so far. Using the former type of algorithms means that one can use worst-case complexity analysis to find an upper bound on the execution time and allocate time for learning accordingly. This is not possible with any-time algorithms. Nevertheless, they have the advantage of being able to provide at any time a draft of the hypothesis being learned. A possible policy in this case is to stop learning to meet deadlines or when cost outweighs expected improvements of accuracy.

MAL for Conflict Simulations Our first case study involves a conflict simulation (CS) as a particular example of game playing. Conflict simulations provide a suitable and challenging application domain with which to test and evaluate logic-based learning techniques such as EBL and ILP, mainly because a large amount of useful background knowledge is readily available in the form of the simulation model. RL has been widely applied to simple games such as Robocup (e.g., Tambe et al., 1999) where domain knowledge is not necessary for the agents to learn and perform well. A command and control hierarchy was chosen as a natural organizational structure for the CS domain. The main target concept is “being successful” (i.e., eliminate the enemy units). However, different agents will learn such a concept differently, according to their roles and corresponding knowledge bases. Commanders will use abstract knowledge to form and learn strategic and operational plans, whereas particular units will use domain-specific knowledge to form and learn tactical plans. Each agent in the hierarchy learns on its own how to improve its skills. Units learn how to succeed in moving and attacking the enemy. Commanders, whose skills and action repertoire are limited to issuing orders, will learn how to have their sections effectively coordinated. Coordination is a matter of command. Consequently, units do not learn how to cooperate—commanders do. Some might argue that this is a type of isolated learning (Stone & Veloso, 1998), because agents seem to learn separately and individually. However, what a unit learns will affect the planning and learning processes of other agents. Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


Figure 2: A CS agent architecture.

If a unit learns how to achieve its goal effectively, its commander will assign it the same task in a similar operation. If the unit learns that it cannot achieve the goal, it will be replaced, or the original commander’s plan will change accordingly. As a consequence, new assignments (roles) or combinations of assignments will be learned. The agent architecture and how it learns is depicted in Figure 2. A CS agent has three different knowledge bases: one containing perfect information about the rules of the simulation, one containing induced knowledge about the opponent’s behavior, and one in which a history of past successes and failures is stored to serve as training examples for learning. After an action has been executed and the result observed, it can be potentially used as a training example for the machine learning algorithms. The learning result is added to the respective knowledge base: results of explanation-based learning are added to the perfect knowledge base (given that the generalization process was based on perfect knowledge), and the results of ILP learning are added to the imperfect knowledge base, because there is no guarantee of correctness for inductive reasoning. We sketch how EBL works with a simple example. Let us assume that a commander has sent its units to eliminate an enemy unit and that these units have chosen their locations in such a way that the enemy is surrounded. Let us further



assume that they are successful and achieve their goal. In order to explain this success, the commander then uses the following rules that were not used previously (although they have always been in the knowledge database): Fact: Each unit has a zone of control, i.e., the six hexagons adjacent to its current location. Rule 1: A unit entering an enemy zone of control must stop its movement phase. Rule 2: A unit withdrawing into an enemy zone of control is eliminated. With these rules, it is now easy for the commander to explain the success of its section. They succeeded because they surrounded the enemy unit, cutting all its possible paths of withdrawal. This rule is then generalized: EBL Rule: If a unit is surrounded, then it is eliminated. This new general rule is then added to the knowledge database and used directly to elaborate more precise plans more quickly. Of course, nothing new was learned. Theoretically, the agents could deduce such a rule from the original database. However, it would have been a more expensive process to do so each time it was needed. It is one thing to deduce from general rules the order and assignment of the subtasks that should, in theory, lead an army to victory; it is another to learn from examples (and some prior knowledge) whether the conditions for such subtasks to be executed successfully actually hold. In contrast to EBL methods, ILP computes a hypothesis not just based on simulation rules known beforehand but also on external and initially unknown circumstances, such as opponent’s strategies. Generally, relying exclusively on EBL-generated rules can turn out to be impractical in real-world domains in which agents work with incomplete knowledge, and thus, ILP is an important addition to the system’s effectiveness. We illustrate the use of ILP in our system with an example. An agent might deduce the following rule from the current knowledge base (taking into account, for example, the units’ combat strength). Rule: If a single tank unit attacks an infantry unit on a bridge, it is successful. The unit uses this rule to compute and execute an appropriate plan (move to the adjacent location, etc.). The plan, however, fails repeatedly (e.g., the infantry unit withdraws and blows the bridge up). These new examples Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


contradict the above rule. ILP will be used to find a new hypothesis. In this particular case, the learned hypothesis is the negation of the previous rule. Hypothesis: If a single tank unit attacks an infantry unit on a bridge, it is not successful. The unit will then rule out a frontal attack and try to find an alternative plan (e.g., moving some friendly units to the other side of the bridge first). The implementation of the CS consists of two parts. The user interface (user input, map/agent visualization, plan/move visualization, communication visualization, and agent hierarchy visualization) is being implemented in JAVA. The simulation server and the agents are being implemented in Progol. The simulation is synchronized and turn-based. In other words, all agents of one side send their actions to be executed simultaneously to the simulation server, which then computes a world-update accordingly and sends the changes back to all agents. Afterwards, the agents of the opposing side take their actions, and so on. Communication could be implemented in MAS-Progol, a multi-agent version of Progol that is being developed in the AI research group at the University of York. It enables an agent to send Prolog goals to other agents and receive variable bindings or truth values in return. More details on the multi-agent system architecture and implementation can be found in Alonso and Kudenko (1999) and Kudenko and Alonso (2001).

The York Multi-Agent Environment The York MAE has been developed as a general-purpose platform on which to study the integration of various machine learning techniques in agents, with a focus on ILP and evolutionary techniques. The platform allows the user to define a simulated ecosystem with different types of terrain and a number of artificial species. Animal-like species are defined by the types of terrain they can walk over, the species they feed on, and the ones that prey on them; one can further define the senses that each species possesses, their range, and the medium used (e.g., light). There is also one predefined type of plant to start the food chain. The behavior of an individual is defined by three factors: • Default behavior: This is a relatively simple behavior that has to be coded directly at source level and is shared by all individuals of a species. The default behavior is defined as a range of drives related to elements of the



•

•

environment that are necessary for survival: thirst, hunger, sex drive, fear. Each time the agent has to take an action, all drives are evaluated, and the strongest triggers an action aiming at its reduction. Inherited features: The default behavior can be parametrized, and those parameters are subjected to natural selection. Each agent has an array of integers used to store the inherited features. Under certain conditions, two agents can mate and produce offspring, with genetic material obtained from the parents. The genetic operators of crossover and mutation are implemented as standard methods and can be used with minimum effort. Learning: Each agent can be optionally allocated a separately running process of the ILP learner Progol (Muggleton & Firth, 2001). The agent then can send observations to the learner and receive recommendations about its next action.

The above setting permits the use of evolution and individual learning separately or in conjunction, so that, for instance, Darwinian evolution of the ILP language bias is combined with ILP learning in individuals. The discrete coordinate system is suitable for experiments with Q-learning, which can be easily implemented in Prolog, and run on Progol, as the latter contains a full Prolog interpreter. The implementation of the system involved a number of issues related to learning. For instance, the designers had to decide how to implement individual agents. Having them as separate threads or processes would lead to the most realistic simulation, where agents would act and learn asynchronously. That would also put the strongest pressure on the learning component of an agent’s behavior. Because most machine learning techniques have not been developed with real-time use in mind, it was deemed more reasonable to opt for a different setting, where time pressure on learning could be controlled and increased gradually. That led to a design in which each agent is prompted in turn to select an action. The initial intuition is that carrying out all agents’ actions simultaneously would mean that no agent is favored. However, mutually exclusive actions would lead to conflicts. To resolve the conflicts, one would have to employ ranking among agents, and favor the ones with the higher rank. Instead, the implementation was based on a loop that rearranges all agents at random then prompts each to choose an action that is carried out immediately. In this setting, if the agent’s decision process involves learning, it can take all the time it needs, while ensuring that the environment and the other agents remain the same. Time pressure can be added gradually by imposing a limit on the time used to learn or, more generally, to make a decision. Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


As each agent’s learning is implemented as a separate process, one can use several processors or computers to implement true parallelism. This opens the door for several strategies combining learning with the use of default behavior. One could wait for the actions suggested by the two and go for the more conservative, or wait for the learning component for a limited period of time and then, if no answer is obtained, use the (easily computed) default behavior. If the behavior based on learning is considerably more successful than the default, but also slower, one may decide to use the actions proposed by the former, even if their choice has not been based on the most recent observations. These, again, can be compared to the actions suggested by the default behavior on the basis of the latest data, and a trade-off can be chosen. The York MAE has been developed as part of several undergraduate student projects. It has been used so far for simulations of kinship-driven altruism. The system is available on request for noncommercial purposes.

SUMMARY AND CONCLUSION The ability to learn is a central feature of intelligent agents. In this chapter, we presented many issues surrounding the application of machine learning to agents and multi-agent systems. Furthermore, we presented two examples of systems that are being developed at the University of York. ML for agents is still a relatively young area, and there are many issues that still need further research, some of which have already been mentioned. While many agent and multi-agent learning techniques and methods still need more evaluation to prove their practicality, we summarize a number of general recommendations for learning agent designers: • Worst-case and average time complexity of the learning algorithm and the theory it produces should be considered when a learning agent is being developed. Also, provisions may be made to ensure that the agent has a partial theory or a fallback plan to act upon should learning or recall have to be interrupted. • The choice of bias can prove crucial to the speed and accuracy of an agent’s learning. In the cases where this choice is not obvious or in dynamic environments, a good engineering approach can be to employ natural selection among agents for the search of the best bias. • When designing learning agents for multi-agent systems, try multiple single-agent learning first. This is much simpler and may already lead to good results. Should it fail, gradually increase the social awareness of the agents. Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


•

• •

While Q-learning is the most common method to date, it may cause difficulties in more complex real-world domains. It is worth looking at alternative methods, especially hybrid solutions (e.g., Dzeroski et al., 1998). Communication between learning agents can significantly improve the coordination and learning performance. Even simple forms of communication such as sharing experience have a significant effect. In some multi-agent applications, role heterogeneity is beneficial and in others, role homogeneity. When using RL approaches, one should take this into account in the choice of reward functions.

In addition to the open problems mentioned earlier in this chapter, further areas of future work on learning agents include: • Formal models of MAL: To date, most developers are not able to predict the behavior of learning agents and depend purely on observing emergent patterns. Formal models of MAL that can be used to predict (or at least constrain) the behavior of learning agents would be useful. For example, Gordon’s research (Gordon, 2000) describes one such approach. • More complex applications: Most MAL application domains are relatively simple. It would be interesting to see MAL research for more complex, real-world applications. Eventually, such applications would encourage researchers to look beyond pure reinforcement (and specifically Q) learning, as suggested in the previous section.

ENDNOTES 1

2

An ellipse is described by a five-tuple: the coordinates of its two foci, and another constant representing the sum of the distances of each of the ellipse’s points to the two foci. Only vertical communication (communication among agents in two adjacent levels of the hierarchy) is allowed. Moreover, the content of the messages is also restricted to orders and requests.

BIBLIOGRAPHY Alonso, E. & Kudenko, D. (1999). Machine learning techniques for adaptive logic-based multi-agent systems. Proceedings of UKMAS-99. Balch, T. (1999). Reward and diversity in multi-robot foraging. Proceedings



of the IJCAI-99 Workshop on Agents Learning about, from, and with other Agents. Baldwin, J.M. (1896). A new factor in evolution. The American Naturalist, 30. Bryant, C.H. & Muggleton, S. (2000). Closed loop machine learning. Technical Report YCS330, University of York, Department of Computer Science, Heslington, York, UK. Carbonell, J., Knoblock, C., & Minton, S. (1990). PRODIGY: An integrated architecture for planning and learning. In K. VanLehn (Ed.), Architectures for Intelligence. Hillsdale, NJ: Lawrence Erlbaum Associates. Claus, C. & Boutillier, C. (1998). The dynamics of reinforcement learning in cooperative multi-agent systems. Proceedings of the 15th National Conference on Artificial Intelligence (AAAI). Crites, R. & Barto, A. (1998). Elevator group control using multiple reinforcement learning agents. Machine Learning, 33. Domingos, P. (1996). Efficient specific-to-general rule induction. Proceedings of the Second International Conference on Knowledge Discovery and Data Mining. Dzeroski, S., De Raedt, L., & Blockeel, H. (1998). Relational Reinforcement Learning. Proceedings of the Eighth International Conference ILP98, Heidelberg: Springer-Verlag. Fayyad, U., Weir, N., & Djorgovski, S. (1993). SKICAT: A machine learning system for automated cataloging of large scale sky surveys. Proceedings of the Tenth International Conference on Machine Learning. Gordon, D. (2000). Asimovian Adaptive Agents. Journal of Artificial Intelligence Research, 13. Kaebling, L.P., Littman, M.L., & Moore, A.W. (1996). Reinforcement learning: a survey. Journal of Artificial Intelligence Research, 4. Kazakov, D. & Manandhar, S. (2001). Unsupervised learning of word segmentation rules with genetic algorithms and Inductive Logic Programming. Machine Learning, 43. Kudenko. D. & Alonso, E. (2001). Machine learning for logic-based multiagent systems, Proceedings of the First Goddard Workshop on Formal Methods for Multi-agent Systems, Springer LNAI. To appear. Lauer, M. & Riedmiller, M. (2000). An algorithm for distributed reinforcement learning in cooperative multi-agent systems. Proceedings of the 17th International Conference in Machine Learning. Mitchell, T. (1997). Machine Learning. New York: McGraw-Hill.



Muggleton, S. & De Raedt, L. (1994). Inductive logic programming: theory and methods. Journal of Logic Programming, 19. Muggleton, S. & Firth, J. (2001). CProgol4.4: a tutorial introduction. In S. Dzeroski & N. Lavrac (Eds.), Relational Data Mining, (pp. 160-188) Springer-Verlag. Mundhe, M. & Sen, S. (2000). Evaluating concurrent reinforcement learners. Proceedings of the Fourth International Conference on Multi-agent Systems, IEEE Press. Parker, L.E. (1994). Heterogeneous multi-robot cooperation. PhD thesis, MIT Department of Electrical Engineering and Computer Science. Prasad, M.V.N., Lander, S.E., & Lesser, V.R. (1996). Learning organizational roles for negotiated search. International Journal of HumanComputer Studies, 48. Provost, F. & Hennessy, D. (1996). Scaling up: distributed machine learning with cooperation. Proceedings of the 13th National Conference on Artificial Intelligence (AAAI-96). Provost, F. & Kolluri, V. (1999). A survey of methods for scaling up inductive algorithms. Data Mining and Knowledge Discovery, 3. Sen, S. & Weiss, G. (1999). Learning in multi-agent systems. In G. Weiss (Ed.), Multi-agent Systems: A Modern Approach to Distributed Artificial Intelligence, The MIT Press. Steels, L. (1990). Cooperation between distributed agents through selforganization. In Demazeau, Y. & Mueller, J.P., (Eds.), Decentralized AI — Proceedings of the First European Workshop on Modeling Autonomous Agents in a Multi-agent World (MAAMAW-89). Amsterdam, New York: Elsevier Science. Stone, P. & Veloso, M. (1998). Towards collaborative and adversarial learning: a case study in robotic soccer. International Journal of Human Computer Studies, 48. Tambe, M., Adibi, J., Alonaizon, Y., Erdem, A., Kaminka, G., Marsella, S., & Muslea, I. (1997). Building agent teams using an explicit teamwork model and learning. Artificial Intelligence, 110. Tan, M. (1993). Multi-agent reinforcement learning: independent versus cooperative agents. Proceedings of the Tenth International Conference on Machine Learning. Tesauro, G. (1992). Practical issues in temporal difference learning. Machine Learning, 8. Thompson, C., Califf, M.E., & Mooney, R. (1999). Active learning for natural



language parsing and information extraction. Proceedings of the 16th International Conference on Machine Learning. Tumer, K. & Wolpert, D. (2000). Collective Intelligence and Braess’ Paradox. Proceedings of the 17th National Conference on Artificial Intelligence (AAAI). Turney, P. (1996). How to shift bias: lessons from the Baldwin effect. Evolutionary Computation, 4. Vidal, J.M. & Durfee, E. (1997). Agents learning about agents: a framework and analysis. Working Notes of the AAAI-97 Workshop on Multi-agent Learning.


Architectures for Cognitive and A-Life Agents 27

Chapter II

Architectures for Cognitive and A-Life Agents Darryl N. Davis University of Hull, UK

ABSTRACT In this chapter, research into the nature of drives and motivations in computational agents is visited from a perspective drawing on artificial life and cognitive science. The background to this research is summarized in terms of the possibility of developing artificial minds. A particular cognitive architecture is described in terms of control states. Steps toward producing an implementation of this architecture are described by means of experimentation into the nature of specific control states. The design of a simple a-life architecture for a predator–prey scenario is described using a transition state diagram. This architecture is then used as a platform with which to develop an agent with drives. This second architecture is used to develop an agent with explicit motivations. The discussion of these (and other) experiments shows how these simple architectures can help to provide some answers to difficult research questions in cognitive science.


28 Davis

INTRODUCTION This chapter explores the use of artificial life metaphors in providing a context for cognition and affect and provides a discourse on the possibilities offered by artificial life for modeling aspects of cognition, specifically, drives and motivations. It is suggested that these phenomena will have an important part to play in the future of agent software engineering as a means for bridging the conceptual gap between agent autonomy and agent benevolence. In this research, the term agent is placed in a broader perspective than the current trend in agent-oriented software (Ciancarini & Wooldridge, 2001; Ferber, 1999), relating to the use of agent in, for example, cognitive science (Wilson & Keil, 1999) and philosophical works (Ryle, 1949). A relatively simple a-life scenario is used to demonstrate the relation between theoretical concepts in the research areas of cognition and affect and computational architectures. The paper highlights a design methodology that allows incremental agent sophistication to be achieved. These explorations and designs make use of the SIMAGENT toolkit—an agent-based toolkit that runs in the Poplog environment. This chapter describes a framework used to investigate cognitive agent architectures. The framework is a synthesis of a number of approaches and aims to provide a means to map abstract theory to well-defined implementation. Influences include the control state approach to cognition (Simon, 1967; Sloman, 1993), the niche and design space perspective on mind (Davis, 2001b), the broad but shallow perspective to agent development of the CMU OZ project (Bates et al., 1991), and the architectural parsimony of Hayes-Roth (1993). It makes use of logic and finite models (including cellular automata) to specify process and behavior in an agent.

RESEARCH OBJECTIVES • • •

This research pursues the following three major objectives: A synthesis of concepts for studying cognitive agents based on an analysis and investigation of how different perspectives on emotion, motivation, and autonomy map onto computational frameworks A framework for developing rules on how artificial life and emergent behaviors can be combined with the more abstract decision-making processes associated with cognitive agents Insights into the nature of heterogeneous mechanisms across different processing, representational, and knowledge levels, and their explanatory role in describing mental phenomena



There are controversies about the terminology used in this area of cognitive science research. Motivation, drive, goal, and emotion are used to refer to and mean a number of different things. There is no universal definition of these terms across (or even within) the fields of philosophy, psychology, cognitive science, and artificial intelligence. Some (Ortony et al., 1988; Frijda, 1986) consider emotions to be a cognition-centered set of phenomena, while others (Frankel & Ray, 2001; Chapman, 1996) consider these terms to be centered on lowlevel (neuro-/physiological) control processes that affect cognition. These arguments are not addressed in depth here — research on emotion is discussed elsewhere (Davis, 2001a). However, such terms will be defined in the following sections as required and used to refer to specific phenomena pertinent to this work. Other major controversies relate to the question of whether a-life entities can be said to be alive or mere process simulacra (Boden, 1996). These controversies are akin to the arguments about the nature of intelligence and how best to investigate, design, and implement it (Brooks, 1991; VanLehn, 1991; Wilson & Keil, 1999). A fundamental issue is the difference between Cartesian- and biological-based cognitive science (Wheeler, 1997). From a Cartesian point of view, the rules governing physical bodies and those affecting psychological phenomena are distinct and noninteracting — mind is separate from body. From the biological-based view of cognitive science, such a distinction cannot be made. The question then arises that if by using synthetic agents in synthetic environments, are we in fact studying synthetic minds which are as alien to those associated with biological entities as silicon is to carbon (Davis, 1998)?

BACKGROUND The research described here makes extensive use of computational design and experimentation. In moving from theoretical perspectives to alternative designs and then to possible computational models, some research questions are answered without requiring implementation. This is termed the design stance. What is typical of this approach is that in moving through these different levels, flaws in our understanding are revealed, and insights into the nature of our research questions are compounded by further questions. Beyond the designs, alternative implementations are possible. Often, it is the case that what might seem trivial at the theoretical level is complex at the implementation level, even in shallow experiments. This is in part analogous to the challenges presented by agents to orthodox software engineering. Agent technology is Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

30 Davis

advancing fast, with an associated increase in the software engineering perspectives appropriate to building agent systems (Ciancarini & Wooldridge, 2001). For some well-defined applications, for example, e-commerce (Luo et al., 2002), it is possible to combine knowledge engineering principles (Scheiber et al., 1993) with toolkits (Nwana et al., 1999) that extend object-oriented software engineering techniques to encompass aspects of agent programming. However, for more investigative experimentation with ill-defined concepts, as investigated here, there is a shortage of easily manipulable toolkits. The agent toolkits that exist for such work, for example, SIMAGENT (Sloman & Logan, 1998), require advanced programming capabilities and prolonged and frequent use to achieve results acceptable to the research community. However, they provide a high-level departure point for experimentation into agent architectures and parameterization. A number of problem domains and test-beds have been used for experimental purposes. The choice of domain and test-bed is determined by the research questions to be addressed. If the major concern is about design options and how to resolve low-level behaviors or the study of collective emergent behavior in simple agent architectures, use is made of a number of variations on TileWorld (Hanks et al., 1993). In studying motivation and its relation to collective activity, we can use TileWorld, an artificial robotcontrolled factory (Davis, 1997; Davis, 2001b) or RoboCup (Nunes, 2001). In studying the mechanisms associated with cognition and affect, there are no conclusive assumptionless analyses that invalidate this approach. On the contrary, by using simulated worlds, greater progress can be made in determining which of the research issues can be tackled this way, which cannot, and why that is the case. The research into computational architectures for motivation, emotion, and autonomy does not require the use of robots. The behaviors, processes, and information structures of interest in this research can be studied without compromise in synthetic environments. The assumption underpinning this research perspective is that these mechanisms in a synthetic mind should be inherently similar across domains, irrespective of the nature of the situated action. Here, a simple predator–prey scenario is used as an exemplar. This environment is a continuous-spaced two-dimensional world of objects and agents. Objects are not mobile, exist for a specified period, and can be used as an energy source by some agents but act as an obstacle to others. Agents move around in this world consuming energy. Agents can sense objects and other agents in their environment if within a specified range. This form of sensing



is a simulated perceptual mechanism provided by the agent toolkit. Agents can reclaim energy by consuming specific classes of objects, and if predatory, specific agent classes. Agents breed in certain conditions, the offspring being some combination of its two parents. Agents, in moving, deposit an energy trail of agent-specific RGB-valued pixels in a color image. Agents with appropriate capabilities can sample and sense this color image to provide clues to the whereabouts and nature of other agents. This use of machine vision in the agents allows us to deepen the perceptual capabilities of our agent architectures and permits explorations into sensory fusion when combined with the in-built sensing mechanism of the toolkit.

DESIGNS FOR MIND CogAff (Sloman, 2001) is a three-column architecture of perception, central processing, and action. It provides the means to investigate the modeling of mental phenomena. It consists of three layers: reactive mechanisms that are (sometimes ballistic) behavioral responses to environmental and internal states; deliberative reasoning that requires explicit manipulation of some form of cognitive structure (for example, motivational constructs); and metamanagement mechanisms (reflective processes) that monitor the agent’s internal state, processes, and ongoing stimulus–response arcs. This abstract architecture accepts perceptual information and outputs action in the environment. Alarms provide feedback from action and other intermediate stages across the three layers and columns. Personae are associated with the reflective level. As different personae become active, the control state of the entire agent changes. This affects the interpretation of incoming information and the actions of the agent in any given external situation. For example, if modeling the type of visual perception described by Gibson (1986), sensory data when perceived is given affordances and valences. An environmental event in the path of an agent will be perceived according to the nature of the current control states. Certain events may be control state neutral; others may exist in antithesis to further control states. Consider a second agent in danger and placed in front of such an agent. If the perceiving agent has adopted a selfish persona, this external event will be given a negative valence and afforded avoidance behaviors. If the perceiving agent has adopted a selfless persona, this external event may be given a positive valence and afforded rescue behaviors. Some events may be completely irrelevant to the agent and are ignored or afforded neutral obstacle-avoidance behaviors. Other events


32 Davis

may be outside of an agent’s immediate repertoire of behaviors but deemed important enough to activate further deliberative processing (for example, motivations). From the control state perspective, an agent’s immediate repertoire of instantiated drives and behaviors can be considered to be transient control states. Motivations are more persistent (but ultimately temporary) control states that may or may not segue well with the agent’s current set of personae. Conflicts between control states can therefore emerge. An intelligent and truly adaptable and autonomous agent should be able to resolve these conflicts through internal changes. This type of phenomena is associated with emotions in humans, and similar affective states can be found in synthetic agents (Sloman & Croucher, 1987). For example, an agent in need of energy but not sensing any may activate a motivator or goal to find an energy source. The behaviors required to realize this motivation may exist within the agent’s overall repertoire of behaviors but be at odds with the desire or intention to explore or avoid certain aspects of the agent’s environment. This state of affairs can be resolved using control state concepts, and the motivator management information used in resolving this conflict can be stored with the motivator. This cannot be said for most structures used for representing goals. It is possible that most agents need to resolve these sorts of conflicts, and the designers do not realize that in producing such agents, they risk compromising the concept of an autonomy that underpins the agent. At the first ATAL workshop, autonomy was defined as one of four foundations for a weak notion of agency (Wooldridge & Jennings, 1995). Autonomy was defined as operating “…without the direct intervention of humans or others, and have some kind of control over their actions and internal state.” Castelfranchi (1995) categorizes and discusses the various types of autonomy that a cognitive agent (or robot) can demonstrate. In particular, a distinction is drawn between belief and goal autonomy in the “Double Filter” Autonomous Architecture. If an agent is to be autonomous, then it must set and have control over its goals. External agents, whether biological or computational, should not have direct control over the setting of an agent’s goals but can only influence these through the provision of information that affects an agent’s belief set. The management of belief sets is described in terms of rationality (logic-based reasoning) and credibility of incoming information (sensing). An agent can only be autonomous within certain bounds. Agent autonomy can be constrained by design specifications that require the agents to pursue certain goals on behalf of their designers (agent benevolence). In sophisticated applications with agents dealing in uncertain informaCopyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


tion and multiple goals, it would seem reasonable to expect conflicts to arise from this compromise between autonomy and the pursuit of possibly incompatible goals. The more encompassing control information associated with motivations in this work may provide agent designers with a tool by which such conflicts can be resolved. Our designs for intelligent systems are such that the resulting computational system should be capable of monitoring itself and catching perturbant behaviors before they become disruptive. Furthermore, an intelligent system should also be capable of recognizing and harnessing beneficial emergent behaviors. There is an argument running in the a-life approach to cognition that this type of cognitive architecture with its various modules and components cannot be designed but needs to be evolved (Husbands et al., 1993). The central thesis to that argument is that there are so many modules with so many interconnections that it is beyond the capabilities of the human mind to design and implement. We counter this argument by adopting a distributed approach to mind. This approach has its computational roots in the early work on agents and distributed blackboard systems (Erman & Lesser, 1975). A society of agents with specific and differing capacities and capabilities can be designed and implemented. The interactions within this society of agents then models the interconnections thought impossible to model explicitly. The thesis central to this counterargument is that the collective and emergent processing of a suitably designed society of specifically engineered agents will give the required overall capabilities. However, it is important that such an agent community have the capability to recognize and manage its emergent behaviors. This is the line being pursued in the research described here and sketched in Figure 1 on the next page. We aim to demonstrate that sophisticated cognitive behavior can arise from the interaction of suitably designed agents in much the same way as demonstrated for socioeconomics in Sugarscape (Epstein & Axtell, 1996). The remainder of this chapter will show how to pursue a line of research into these phenomena using computational tools guided by the principles of incremental complexity and architectural parsimony. This experimentation proceeds from relatively simple, easily realized architectures to quite complex designs.

DRIVES AND A-LIFE AGENTS Drives are low-level, ecological, physiological, and typically preconscious. They provide the basis for an agent’s behavior in the world, are periodic but short-lived, and are defined in terms of resources essential for an agent. Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

34 Davis

Figure 1: A societal approach to cognition based on the CogAff architecture. Perceptual Agents

Reflective agents Context Agent Attention Agents

Perceptual Filter

Sensory Input Information Flow

Inner Perception Behaviour Inner Action Agents Deliberative contexts

Deliberative & Abstract agents Memory Agents

Attention Filters Reactive Agents

Agents in this area are TightlyCoupled Control Feedback

REFLEX

Actions

Rerepresentation

Such activities for information agents include the need to gather resources and propagate information to associates in their society. In biological agents, such drives include thirst, hunger, and reproduction. Thresholds for the onset and satiation of such drives are variable and dependent upon processes internal to an agent and external factors arising in the agent’s environment. Such drives can be modeled relatively easily in computational agents. These simple “a-life” agents then provide a theoretical and computational platform with which to explore more interesting phenomena. Consider a base-level agent (Agent0) with no explicit motivations but three implicit drives: the need to eat, escape predators, and reproduce. Behaviors associated with these drives can be modeled using state transition diagrams as in Figure 2, described using propositional calculus and implemented as an action selection architecture (Agre & Chapman, 1987) in SIMAGENT. These agents move around an environment populated with food items and other agents (we have experimented with hundreds of such agents in



Figure 2: State transition diagram for drives in the simplest prey agent (Agent0). Sense Predator

Avoid

Breed

Breeder Adjacent

Avoid

Sense Obstacle

Sense Agent

Herding

Sensory Data Sense Food

Food Adjacent

Energy Low

Moveto Agent

Moveto Food

Combine Behaviour

Eat Food

any one environment), some of which are predatory. The agent can sense items within a specified radius of itself and discriminate between food, nonpredatory, and predatory agents. This agent is given an initial energy level and must locate food items to maintain its energy. Movement causes energy to be consumed in proportion to its velocity. When the energy level dips below a specified threshold, the agent changes state (it becomes hungry) and must sense and find food. If this energy level reaches zero, the agent dies. The agent can be in a herding or nonherding state. When in a nonherding state, all nonpredatory agents are effectively mobile obstacles to be avoided. Irrespective of state, predatory agents are mobile threats and need to be avoided. If their energy level is sufficiently high and another agent is adjacent, agents can reproduce. The resultant agent drains a percentage of the energy of both parents and is initialized with a mix of its parent’s internal information. The Eat-Food reflex (bottom of Figure 2) can be activated regardless of any other potential action or internal state, given that food is adjacent to the agent. Each


36 Davis

Figure 3: Model for internal energy based drives. The four energy thresholds have been subject to experimentation and could be optimized using, for example, genetic operators. IF E < ET0 THEN

0 → SignalHunger (Dead)

IF E < ET1 THEN

1 → SignalHunger

IF E < ET2 THEN

(ET2-E/ET2–ET1)→ SignalHunger

IF E > ET2 THEN

0 → SignalHunger (Satiation)

1.0

Hunger Drive Signal

Reproduce Drive Signal 0.0 ET0

ET1

ETB

ET2

node in the state transition diagram can be formalized using propositional logic. These formalisms then map onto object-oriented methods and production rules within the SIMAGENT toolkit. Velocity and direction of an agent are governed by a two-part vector representing velocity in vertical and horizontal directions. A change in direction, toward or away from any other object or agent, is expressed as a change to this vector. The actuation of any behavior in an agent maps onto a vector change impulse. Where more than one behavior is actuated, the combined impulses are summed in the “Combine Behavior” node of Figure 2. This simple (non)decision mechanism results in an agent responding to all inputs. Hence, a hungry agent may move toward a sensed predator if sufficient food items are similarly located. This agent makes for a useful plinth (left-hand side of Figure 5) with which to experiment further. The next class of agent (Agent1) extends the model associated with the drives of the base plinth. In Agent0, drives are on or off. Drives in Agent1 make use of signal strength based on two factors: an internal measure (energy) and an external measure (distance from some object or agent). The signal strength based on internal energy (used in the food and reproduce drive) is defined as Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


Figure 4: AgentWorld of methods, rule-sets and rule-families.

Until Simulation End: I.E. no agents left or stated epoch reached

Simulation Start

Update AgentWorld If agent1 use Rulefamily1 RS1: Perception

Post Agent Actions to AgentWorld

Simulation End: Obtain statistics

For Each Agent Call run-agent method If agent2 use Rulefamily2 RS1: Perception

RS2: Drive Activation

RS2: Drive Activation

RS3: Drive Instantiation

RS3: Drive Instantiation

RS4: Drive Selection

RS6: Motivation Activation

RS5: Drive Actuation

RS7: Motivation Selection RS4: Drive Selection RS5: Drive Actuation

shown in Figure 3. The signal strength associated with the Hunger drive makes use of the current energy level (E) and three thresholds, as given in Figure 3. The thresholds can be optimized through experimentation and analysis and are also subject to change through agent reproduction. A similar model is used for the drives associated with the agent’s perception of its external environment. Again, these models readily map onto the object-oriented methods and production rule capabilities of the agent toolkit. Specific capabilities (for example, perception, drive activation, drive selection, and behavior combination) are described using propositional calculus and then bundled together as rules in rulesets (see Figure 4). For example, one ruleset (RS1) deals with agent perception, while ruleset RS2 determines which of the agent drives can be currently active. A further ruleset (RS3) instantiates active drives with specific values for signal strength and target locations. A fourth ruleset (RS4) selects compatible drives. And, the final ruleset (RS5) maps these drives onto agent actions posted to the world update methods of the agent toolkit. Rulesets are bundled together as rule systems (e.g. RuleFamily1 for Agent1) that define the information processing ontology for any particular class of agent. Alternative rulesets for any specific behavior allow Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

38 Davis

the agent to switch between alternative processing modes. Rulesets can be shared across rule systems, and more complex agents build on simpler agents by extending the rule system with further rulesets (e.g., RuleFamily2 in Figure 4). Agent1 is a reactive architecture, responding to internal and external pressures. It can make use of the same decision-making mechanism as Agent0 but with the impulses (i.e., vector changes) weighted according to their signal strength. An alternative decision-making mechanism is to select the impulse with the highest signal as a primary goal and then any other impulse that does not conflict with this behavior. For this test harness, all agent behaviors result in a movement in two dimensions. Hence, a conflict in behavior is simply a vector that distracts from the direction vector of the primary goal. Again, there are experimentally derived parameters that define a maximum deviation to the nominal direction vector. Agent1 displays drives and implicit goals (the nominal direction vector of the drive with strongest signal) but no explicit motivation. Any affective qualities are an emergent property of the agent and its environment. Maundering, the switching between two goals or behaviors, is one example. Consider a simple experiment where a predominantly static predatory agent is flanked by two food items of benefit to a further agent. While the energy level of the nonpredatory agent is high, it will move away from the predatory agent. As the energy level drops, the signal strength to move toward a food item increases. At some point, the signal strength associated with moving toward the food becomes sufficient that it is adopted as the primary goal. The agent therefore moves toward one or both of the energy sources (these are not antithetic behaviors) and, hence, the predatory agent. The behavior to move away from a predator is then given a higher signal strength, and the agent moves in its original direction away from the predator and, hence, the food. Typically, the behavior of the agent fluctuates until it runs out of energy or is consumed by the predator. If vector summation is used instead, the agent oscillates around its original position but again gradually moves toward the predator with the same result. This agent is, in effect, caught in a cycle of goal conflicts and maunders between acting on the flee-predator and seek-energy drives. The agent can possibly reach the energy items through planning. A reactive planner with precompiled plans may not necessarily produce a nondirect path to the required items and may therefore result in the same behavior as described above but at greater computational expense. The agent needs to be able to perform planning that incorporates models of its internal states and external environment. These are deliberative processes and are associated with explicit motivations. Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


MOTIVATION IN COMPUTATIONAL AGENTS Motivations are a disposition to assess situations in certain ways and involve perception of problematic events and states, representations, and paths to modified states of affairs. This research (Beaudoin & Sloman, 1993; Davis, 2001b) has identified a number of subtypes: • Goals can be of several subtypes. Quantitative goals are those talked about in control theory and tend to involve negative feedback. Qualitative goals are most used in agents and involve relations, predicates, states, behaviors, etc. Hybrid goals are some mix of these two. • Attitudes are predispositions to respond or act (internally or externally) to specific (perceptual or cognitive) cues and can involve intricate collections of beliefs, motivators, and other control states, for example, the selfless and selfish traits of the agents presented in Davis (1997). Similar control states are linked to personae in other work (Sloman, 2001). • Desires relate objects, agents, or events in an agent’s world to plausible states of that world. Impulses are transient desires that when acted on give rise to nontransient changes internal to the agent and in the external world. Psychological definitions of emotion can be found (Wilson & Keil, 1999) that refer to cognitive (appraisal) and physiological factors (reinforcers and valencing). The processes leading to the experience of emotions (in humans) are neither bottom-up nor top-down — they are both and more. Emotions are experienced as a result of the interactions within and with a synergistic information processing architecture (Davis, 2001a). In short, emotions are in part mental (appraisal) states and supporting and causal (valencing) processes. Any computational model of emotion must attempt to meet similar specifications. The stripped down architecture presented here makes use of causal (valencing) processes (i.e., drives) and appraisal processes (i.e., motivations). We can therefore use experimentation with a process architecture to answer the question of whether such a minimal architecture is capable of supporting recognizable emotive qualities. The final section of this chapter addresses this question. The architecture shown on the right-hand side of Figure 5 adds explicit motivation processing at the deliberative level to the drive model of Agent1. The signal strength associated with a drive means that calls to build explicit motivational constructs for specific drives can have a quantitative (signal) and qualitative (type) component. Motivational constructs can be likened to hybrid goals. The full motivator structure is shown in Table 1. This is implemented using the object-oriented paradigm with control and support processes modeled Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

40 Davis

Figure 5: Two action selection architectures. Agent0/Agent1 (left) with drives and Agent2 (right) with explicit motivational structure processing. Agent2 is an example of the tightly coupled processes shown in the center of Figure 1. Both are modeled in Figure 4.

Deliberative Level

Reactive Level

Attention Filters

Surfaced & emergent goals Reactive Level

Reflexive Level SENSE

ACT

Reflexive Level SENSE

ACT

using methods and rulesets in the SIMAGENT toolkit (rulesets RS6 and RS7 in Figure 4). Not all components of this computational object need to be instantiated for any specific agent class or drive type. Many of the components act as keys that influence processing elsewhere in the agent architecture of Figure 1. For example, the Emotion Key component can take single word values (e.g., Fear, Hate). This does not imply that emotion is a single linguistic term. Rather, these linguistic keys prime and influence related processing. Therefore, in this architecture, emotion is a control state distributed across a number of constructs and construct handling processes and any supporting processes. Consider the maundering scenario described above. The reactive agent will either flee from the predator (Signal Strength Model) or move toward the mean position of the three entities (Behavior Sum Model). The deliberative agent has other possibilities. Motivational structures can be instantiated that correspond to the following three drives based on Figure 3: • Flee from Predator at location (X,Y) with signal strength 0.75 • Move to Energy Source One at location (Xf1,Yf1) with signal strength 0.5 • Move to Energy Source Two at location (Xf2,Yf2) with signal strength 0.5 Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


Table 1: The components associated with motivator structures. Component

Meaning

Actors and Entities Belief Indicator

Other agents (actors) and objects referenced by this motivator

Commitment Status Decay Function Dynamic State Emotional Key Importance Value Insistence Value Intensity Management Information Motivational Attitude Plan Set Rationale Semantic Content Urgency Descriptor

Indication of current belief about the status of semantic content P: e.g. true, partially true, false. The current status of the motivator, e.g. adopted, rejected, undecided, interrupted, stalled, unconsidered, completed. Defines how insistence decreases while motivator is not adopted. The process state of the motivator e.g. being considered, nearing completion etc. Processing keys to the emotions and their situational triggers for the motivator. Importance (e.g. neutral, low, medium, high, unknown). This may be intrinsic or based on an assessment of the consequences of doing or not doing Heuristic value determining interrupt capabilities. This should correspond to a combination of the motivator’s importance and urgency. This influences the likelihood of (continuing) to being acted on. The state of relevant management and meta-management processes. The motivator’s attitude to semantic content P : make true, keep true, make false etc. Possible plan or set of plans for achieving the motivator. If the motivator arose from explicit reasoning – motivators need not. A proposition P denoting a possible state of affairs, which may be true or false How urgent is this descriptor – this may be qualitative (e.g. high, low) or quantitative (for example a time-cost function).

Table 2 gives examples of this structure as created for the Flee from Predator drive in the current situation with the initial values given in Column 2. Similar structures are created for the other two (Move to Energy Source) drives. The deliberative processes associated with motivation management are initiated by an alarm call from the posting of these structures to the agent’s motivator database. In this simple architecture, all these motivators are given an initial appraisal, leading to the changes in value as shown in Column 3 of Table 2. The motivator database is then ordered on the basis of the importance value for each motivator. Importance is given a fuzzy-valued descriptor (low, medium, or high) based on an appraisal of insistence, emotion key, and content. In the simplest motivation architectures, the topmost item from the database would be selected and Plan Set 5 (flee) adopted as behaviors to be activated. A more sophisticated approach can look to see how motivators can be Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

42 Davis

Table 2: The instantiated motivator structure for Flee Predator. Component Actors Belief Indicator Status Decay Function Dynamic State Emotional Key Importance Value Insistence Value Intensity Management Attitude Plan Set Rationale Content

Value (initial) Predator1 True Unconsidered Null Null Fear Null

Value (appraisal 1) Predator1 True Considered Null Postponed Fear High

Value (appraisal 2) Predator1, Food1 True Adopted, Merged Null Active Anxiety High

0.75 Null Null Make true Null Drive:Fear Flee(Predator1,x,y)

0.75 Medium Null Make true Null Drive:Fear Flee(Predator1,x,y)

Urgency

Null

Null

0.75 Medium Combined(M1,M2) Make true Plan(set10, bypass) Drive:Fear, Drive:Hunger Flee(Predator1,Xp,Yp) GoTo(Food1,Xf1,Yf1) High

combined, in a manner similar to goal conflation in teleoreactive planning (Nilsson, 1994). A planning module is used to find plans that combine plansets associated with pairwise combinations of motivators. The ordering on the motivator database dictates the combination order. Motivators that can be combined are subsumed into the highest motivator with the second deleted; in an agent with a more extensive motivator management architecture, the creation of new motivators does not delete existing ones. Any change to the motivator database results in further motivator appraisal. The fourth column in Table 2 shows the result of combining the Flee Predator with GoTo EnergySource1 motivators with a suitable new planset. This motivator is adopted and made active, and its planset is pursued as a series of movements in the environment that allows the agent to stay away from the predator but still move to and consume one of the energy sources. Such architectures for (single) agents have been subject to research for a number of years (Beaudoin & Sloman, 1993; Davis, 1997). A distributed version is now being redeveloped based on the sketches in Figures 1 and 5 (Davis, 2001b; Nunes, 2001). This builds upon the idea of loose and tight coupling of agents as described in earlier work. Rather than build many agents, each with their own motivation appraisal and management processes, agents in the same team (Nunes used five-aside-football as the application domain) initiate motivational structures that are passed to an abstract agent responsible for coordinating the motivations of alike agents. In effect, agents in the same Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


herd (or team) share the computationally expensive deliberative processing associated with motivator appraisal. There are a number of reasons why such a direction may prove beneficial. Agent communication overheads associated with motivation coordination are computationally less expensive than multiple agents each with their own motivation management capabilities. Furthermore, given that we require something other than emergent cooperation, these latter agents still need to communicate with each other about their adopted motivations.

DISCUSSION Extending the capabilities of implementations based on the earlier designs was problematic without reengineering the entire implementation — a process that could take years. With the distributed model, extra agencies could be introduced with these further capabilities. Furthermore, difficulty was experienced in introducing learning and adaptation into the design and implementation. It seemed that control mechanisms spanning the entire agent architecture would need changing. Other research has identified the same problem (Franklin, 1997). With the distributed model, specific agents can change as the overall architecture adapts. Learning associated with specific capabilities or knowledge is focused on those agents responsible for those capabilities or knowledge. An analogy can be drawn between this model and with changes observed to classifier rules in Holland’s architecture (Holland, 1975). The distributed agent requests those elements of itself (i.e., its component agents) that are associated with the current learning task to modify themselves. Changes to the global agent are in effect mapped onto changes to agents with specific process loci. There are many challenges to be faced with this direction. Currently, we are looking at the nature of communication between agents with shared motivations and are using distributed blackboards (Nunes, 2001) as a metaphor to the global workspace theory that Baars (1988) describes. This approach is being used by others, for example, IDA (Franklin, 2000). Figure 1 provides the perspective for the experiments with alternative designs for agents with the mechanisms necessary for us to address the stated research objectives. Although (ultimately) many different agent types will be required, no single agent type will be much more complicated than that described for the right-hand side of Figure 5. Deliberative agents are awoken by alarms when required. In one such architecture, agents develop motivator structures and pass them to a shared motivator appraiser agent. This agent has sufficient knowledge of the communicating agents that it can select motivators as Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

44 Davis

appropriate. Other agents can be called on to perform specific tasks by the reactive agents or by other deliberative agents as required. The motivator appraisal described above for the maundering reactive agent could, therefore, be applied to many agents, making use of the motivator-initiating reactive agent, one motivator appraisal agent, and a planning agent. The design philosophy described above is proving useful in developing agent applications. We are finding that developing design methodologies that can cope with the sometimes elusive concepts and ideas associated with cognitive science is of benefit to intelligent agent modeling in more constrained domains such as enterprise management (Davis, 2000), clinical diagnosis, and e-commerce (Luo et al., 2002). The applications implemented so far are making use of what we have learned from developing adaptive agents for cognitive science research. The cognitively oriented software agents described in this paper are helping us understand the concepts underlying drives and motivation. We firmly believe that no matter how eloquent philosophical or psychological theories and models, the plausibility of the information processing and other mechanisms embodied in these theories can only be validated through the development of computational models. The type of experiments described here and elsewhere (Davis, 1997, 2001a, 2001b; Nunes, 2001) exemplify this approach. We have designed and implemented agents that display motivational qualities and address important questions about the nature of emotion and autonomy. Metaagent architectures and adaptive agents provide the design and implementation tools necessary to pursue such lines of inquiry. The work in this chapter highlights the relation between agent architectures and the nature of drives and motivations. It shows that we can differentiate between drives and other forms of motivational control state at theoretical, design, and computational levels. With the advent of affective computation (Picard, 1997) and the growing sophistication of software systems, the types of control structures discussed in this chapter may well be required in future agent applications. We have shown that agents demonstrate sometimes unwanted qualities, i.e., control dithering and maundering. We have shown that by improving the processing structures associated with drives and goals (i.e., the use of motivations), deliberative control of such emergent and affective states can be achieved. Other experimentation has shown that oscillations in this deliberative control of emergent affective states can occur. The inclusion of the reflective (meta-agent) layer in the agent architecture provides the means to control this phenomenon. Earlier, it was argued that the simple Agent0/Agent1/Agent2 architecture is capable of demonstrating affective qualities such as maundering. Such results are in Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


agreement with other researchers (Scheutz & Sloman, 2001). Now consider whether we have demonstrated emotive qualities in our research agents. Oatley and Jenkins (1996) define emotion as “a state usually caused by an event of importance to the subject. It typically includes (a) a conscious mental state with a recognizable quality of feeling and directed towards some object, (b) a bodily perturbation of some kind, (c) recognizable expressions of the face, tone of voice, and gesture (d) a readiness for certain kinds of action.” Others, for example (Frijda, 1986), give similar definitions. This school of thought permits basic emotions, of which Fear is one. Fear is defined as the physical or social threat to self or a valued role or goal. Do any of the above agents satisfy these descriptions? Prey agents recognize the localized presence of a predator as a threat. Agents with motivations produce motivational states (structures and processes) to deal with this identified threat (Part a of the definition). Even the Agent0 base plinth displays a perturbation as a movement way from the threat (Part b of the definition), which is also an indication of readiness to respond to the threat. The agents with motivations also have plans to avoid the threat (Part d of the definition). It therefore follows that these agents are in fact capable of displaying a limited repertoire of emotive qualities, even if in a rather shallow manner. In pursuing this line of research, we found it necessary to revisit foundational principles in agent theory, such as autonomy, flexibility, and adaptability. Domain models of autonomy sometimes raise problems in the mapping from theory to design to implementation. A domain model of autonomy in economics, social theory, cognitive science, or artificial life may be at odds with or compromised by the notion of autonomy underpinning the agents in a computational toolkit. Such compromises can lead to the design of agent systems with inherent conflicts. This is an issue that needs to be addressed by research into agent software engineering. By designing agents with the qualities described in this chapter, an agent is given the means to represent and reason about these conflicts when they arise. This research continues to raises the questions: what is an agent, what is a mind, and what are emotion and motivation.

REFERENCES Agre, P. & Chapman, D. (1987). PENGI: An implementation of a theory of activity. Proceedings of AAAI-87, (pp. 268-272), Seattle, WA. Baars, B.J. (1988). A Cognitive Theory of Consciousness. Cambridge University Press.


46 Davis

Bates, J., Loyall, A.B., & Reilly, W.S. (1991). Broad agents. SIGART BULLETIN. 2(4). Beaudoin, L.P. & Sloman, A. (1993). A study of motive processing and attention. In A. Sloman, D. Hogg, G. Humphreys, D. Partridge, & A. Ramsay (Eds.), Prospects for Artificial Intelligence, IOS Press. Boden, M.A. (1996). The Philosophy of Artificial Life. Oxford: Oxford University Press. Brooks, R.A. (1991). Intelligence without representation. Artificial Intelligence. 47, 139–159. Castelfranchi, C. (1995). Guarantees for autonomy in cognitive agent architectures. In M. Wooldridge & N.R. Jennings (Eds), Intelligent Agents. Heidelberg: Springer-Verlag, pp. 56–70. Chapman, C.R. (1996). Limbic processes and the affective dimension of pain. In G. Carli & M. Zimmerman (Eds.), Towards the Neurobiology of Chronic Pain. 110, 63–81. Ciancarini, P. & Wooldridge, M. (2001). Agent-Oriented Software Engineering. Heidelberg: Springer-Verlag. Davis, D.N. (1997). Reactive and motivational agents. In J.P. Muller, M.J. Wooldridge, & N.R. Jennings (Eds.), Intelligent Agents III. Heidelberg: Springer-Verlag. Davis, D.N. (1998). Synthetic agents: synthetic minds? Frontiers in Cognitive Agents. IEEE Symposium on Systems, Man and Cybernetics. San Diego, CA. Davis, D.N. (2000). Agent-based decision support framework for water supply infrastructure rehabilitation and development. International Journal of Computers, Environment and Urban Systems, 24, 1–18. Davis, D.N. (2001). Multiple level representations of emotion in computational agents. AISB’01 Symposium on Emotion, Cognition and Affective Computing. University of York. Davis, D.N. (2001). Control states and complete agent architectures. Computational Intelligence. 17(4). Epstein, J.M. & Axtell, R. (1996). Growing Artificial Societies. Cambridge, MA: MIT Press. Erman, L.D. & Lesser, V.R. (1975). A multi-level organisation for problemsolving using many diverse cooperating sources of knowledge. Fourth International Joint Conference on Artificial Intelligence (IJCAI-75) (483–490). Ferber, J. (1999). Multi-Agent Systems. Reading, MA: Addison-Wesley.



Frankel, C.B. & Ray, R.D. (2001). Competence, emotion and self-regulatory architecture. AISB’0 Symposium on Emotion, Cognition and Affective Computing, University of York. Franklin S.P. (1997). Autonomous agents as embodied AI. Cybernetics and Systems. 28(6), 499–520. Franklin, S.P. (2000). A “consciousness” based architecture for a functioning mind, AISB’00 Symposium on How to Design a Functioning Mind, University of Birmingham. Frijda, N. (1986). The Emotions. London, New York: Cambridge University Press. Gibson, J.J. (1986). The Ecological Approach to Visual Perception, LEA Press. Hanks, S., Pollack, M.E., & Cohen, P.R. (1993). Benchmarks, test-beds, controlled experimentation, and the design of agent architectures. AI Magazine. 14(4), 17–42. Hayes-Roth, B. (1993). Intelligent control. Artificial Intelligence. 59, 213– 220. Holland, J.H. (1975). Adaptation in Natural and Artificial Systems. University of Michigan Press. Husbands P., Harvey, I., & Cliff, D. (1993). An evolutionary approach to situated AI. In A. Sloman, D. Hogg, G. Humphreys, D. Partridge, & A. Ramsay (Eds.), Prospects for Artificial Intelligence. IOS Press, pp. 61–70. Luo, Y., Davis, D.N., & Liu, K. (2002). Combining KADS with Zeus to develop a multi-agent e-commerce application. International Journal of Electronic Commerce Research (In Press) Nilsson, N.J. (1994). Teleo-reactive programs for agent control. Journal of Artificial Intelligence Research. 1, 139–158. Nunes, H.A. (2001). Investigation of motivation in agents using the simulation of 5-aside football. M.Sc. Research Thesis, Computer Science, University of Hull. Nwana, H., Ndumu, D., Lee, L., & Collis, J. (1999). ZEUS: A toolkit for building distributed multi-agent systems, Applied Artificial Intelligence Journal, 13(1), 129–186. Oatley, K. & Jenkins, J.M. (1996). Understanding Emotions. Oxford: Blackwell Scientific. Ortony, A., Clore, G.L., & Collins, A. (1988). The Cognitive Structure of Emotions. London, New York: Cambridge University Press.


48 Davis

Picard, R. (1997). Affective Computing. Cambridge, MA: MIT Press. Ryle, G. (1949). The Concept of Mind. University of Chicago Press. (Reprinted 1984). Scheutz, M. & Sloman, A. (2001). Affect and agent control: experiments with simple affective states. In N. Zhong, J. Liu, S. Ohsuga, & J. Bradshaw (Eds.), Intelligent Agent Technology: Research and Development. New Jersey: World Scientific, pp. 200–209. Schreiber, G., Weilinga, B., & Breuker, J. (1993). KADS: A Principled Approach to Knowledge-based Systems. New York: Academic Press. Simon, H.A. (1967). Motivational and emotional controls of cognition. In H.A. Simon, Models of Thought. Yale University Press. Sloman, A. (1993). The mind as a control system, In C. Hookway & D. Peterson (Eds.), Philosophy and the Cognitive Sciences. London, New York: Cambridge University Press. Sloman, A. (2001). Varieties of affect and the CogAff architecture schema. AISB’01 Symposium on Emotion, Cognition and Affective Computing. University of York. Sloman, A. & Croucher, M. (1987). Why robots will have emotions. IJCAI7 (197–202), Japan. Sloman, A. & Logan, B. (1998). Cognition and affect: architectures and tools. Proceedings of the Second International Conference on Autonomous Agents (Agents ’98), ACM Press. VanLehn, K. (1991). Architectures for Intelligence. LEA. Wheeler, M. (1997). Cognition’s coming home: the reunion of life and mind. Fourth European Conference on Artificial Life. Cambridge, MA: MIT Press, pp. 10–19. Wilson, R.A. & Keil, F.C. (1999). The MIT Encyclopedia of the Cognitive Sciences, Cambridge, MA: MIT Press. Wooldridge, M. & Jennings, N.R. (1995). Intelligent Agents. Heidelberg: Springer-Verlag, pp. 56–70.


SoFAR: An Agent Framework 49

Chapter III

SoFAR: An Agent Framework for Distributed Information Management Luc Moreau University of Southampton, UK Norliza Zaini University of Southampton, UK Don Cruickshank University of Southampton, UK David De Roure University of Southampton, UK

ABSTRACT SoFAR, the Southampton Framework for Agent Research, is a versatile multiagent framework designed for Distributed Information Management tasks. SoFAR embraces the notion of proactivity as the opportunistic reuse of the services provided by other agents, and it provides the means to enable agents to locate suitable service providers. The contribution of SoFAR is to combine ideas from the distributed computing community with the performative-based communications used in other agent systems: communications in SoFAR are based on the startpoint/endpoint paradigm, a powerful abstraction that can be mapped onto multiple communication


50 Moreau, Zaini, Cruickshank & De Roure

layers. SoFAR also adopts an XML-based declarative approach for specifying ontologies and agents, providing a clear separation with their implementation.

INTRODUCTION The volume of information available from the World Wide Web and corporate information systems has increased dramatically over the last few years. It is now recognized that users require assistance to avoid being overwhelmed by this wealth of information; it is also essential that information suppliers be provided with tools that help them in authoring and maintaining it (De Roure et al., 1996). Distributed Information Management (DIM) is the term used to describe the set of activities that allows users to manage the entire life-cycle of information in a distributed environment. The activities, also referred to as DIM tasks, involve, among others, document creation and publication, information space navigation, information discovery, and integrity maintenance. The large volume of highly dynamic information involved in DIM tasks is an ideal subject for agent-style processing. This has been exemplified in several research projects, such as Pattie Maes’ agents that reduce users’ overload (Maes, 1994) or the numerous agents applied to the Internet or the World Wide Web (Chen & Sycara, 1998; Lieberman, 1995). Over the last decade, a series of projects at Southampton addressed the issue of distributed information management. This activity began with the Microcosm system (Fountain et al., 1990), which pioneered the idea of building a hypertext system out of a set of loosely-coupled communicating processes. It was an example of an open hypermedia system, in which links are regarded as first-class citizens. By managing and storing links in specific databases, called linkbases, this approach allows users to customize their information environment by selecting the appropriate linkbases. The open hypermedia philosophy was brought to the WWW by the Distributed Link Service (Carr et al., 1995). In a project called Memoir (De Roure et al., 1998), the notion of navigation trails was used to recommend documents examined by users sharing similar interests. This work was further extended by using a notion of “user context” to suggest links that are relevant to users (El-Beltagy et al., 2001). We learned two important lessons from our practical experience with designing and building prototypes over the last decade. First, it became clear



that properties of weak agency identified by Wooldridge and Jennings (1995), namely, autonomy, social ability, reactivity, and proactivity, are also desirable for distributed information management systems. Second, we came to the conclusion that distributed information management may be regarded as the result of coordinating a multitude of simple DIM tasks. It is our belief that the functionality of the system can be the result of individual agents opportunistically exploiting services offered by other agents. Therefore, we have been working toward building a multiagent system in which numerous agents can interact to test our hypothesis. In the domain of distributed information management, the ubiquitous definitions of weak agency defined in Wooldridge and Jennings (1995) are applicable but require some qualification. We adopted the following terminology for our DIM agent framework: 1. Autonomy: The ability of an agent to effect elements of its behavioral repertoire without intervention or direct control from actors external to the agent system (e.g., the user) 2. Social ability: The capacity to communicate with other agents in the system—it is an existence criterion for our framework; an agent that does not communicate, by definition, is not a participant agent 3. Reactivity: The capacity to reach to changes in the environment 4. Proactivity: As part of their autonomy, agents must at least possess opportunism as a key goal-directed behavior; that is, they must actively search for and use the abilities of other agents to complete their tasks Therefore, a multiagent system is composed of agents with simple (usually singular) abilities who possess the above criteria. By their opportunism, an agent can potentially discover new functionalities through cooperation. We believe that the simplicity of each of the DIM agents will enable the principled engineering of global behavior more easily than if each agent is gifted with sophisticated functionality and behaviors—this is because the local interactions are simpler, enabling abstraction. By making use of other agents whenever possible, the real power of the system is realized as a result of the collective behavior of the agents. Over the last few years, part of our activity has concentrated on designing and building a framework for coordinating the activity of our DIM agents. The purpose of this chapter is to describe the outcome of this research, called SoFAR (SOuthampton Framework for Agent Research), its properties, its design, and its implementation. The framework has been used by some 60 researchers and has been the object of an undergraduate course attended by Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


20 students; it is also currently used in several research projects. The key contributions of SoFAR are as follows: •

To apply some successful ideas of the distributed computing community to multiagent systems: We adopt the same communication paradigm as Nexus (Foster et al., 1996), which is the communication layer that has been used to build the Computational Grid (Foster and Kesselman, 1998). This approach has been shown to be generic and scalable. From the agent perspective, the act of communication becomes independent of the mechanisms for communicating, which is a view that naturally extends to speech-act-based communications. We introduce the concept of a contract, similar to Jini leases. A contract fulfils multiple roles, including a proof of successful registration or subscription, the clearing of registries in the presence of failures, and a session identifier. Contracts are an essential element by which agents can control the flow of information that is being delivered to them.

•

To adopt an XML-based declarative style for defining agents and ontologies, allowing a separation of specification from implementation: XML declarations are compiled into Java classes, respectively, by an “agency compiler” and an “ontology compiler,” providing a uniform and high-level programming interface. Such automatic code generation avoids the programming of repetitive code by the programmers. XML declarations have a clearly specified semantics, which promotes interoperability between components. Compilers can be retargeted to other programming languages, hereby promoting openness in the system.

This chapter is organized as follows. In the next section, we discuss the requirements of the framework. In “SoFar Agent Framework: Description,” the framework is described, including its communication mechanism, the notion of ontology, and the architecture it provides. In “Implementation Overview,” we present some implementation details of the framework. In the final sections, we discuss related work and future work before concluding the chapter. Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


FRAMEWORK FOR DIM AGENT: REQUIREMENTS Our initial motivation is to build an advanced distributed information management system. Even though we can identify a vast number of tasks that such a system must perform, we are currently unable to define such a system precisely or to explain its behavior in terms of subcomponents. Instead, we adopted a bottom-up approach to building such a system. As we are able to engineer systems that perform the simple tasks that we have identified, we wish to promote the coordination of their activity, in particular, by opportunistically exploiting services offered by other agents. In this section, we present a list of requirements that we identified for the framework in order to satisfy that goal: 1. DIM tasks need to be coordinated in a distributed environment. The number of tasks is not known a priori and may evolve over time. The framework must be reactive, because it must accommodate new tasks as they appear online. 2. The framework must promote the opportunistic reuse of agent services by other agents. To this end, it must provide mechanisms by which agents may advertise their capabilities and ways of finding agents supporting certain capabilities. 3. There are potentially a large number of agents that must be coordinated by the agent framework. The framework must be lightweight and scalable. By lightweight, we mean that it must be possible to implement efficient communication mechanisms, and that the administrative overhead of the framework should not hamper the overall performance of the system. By scalable, we mean that we must be able to accommodate a high number of agents (in the thousands) and that we want to avoid centralized components that would create bottlenecks during execution. 4. In order to be generic, communication primitives provided by the framework must be independent of the actual means of communication. Indeed, there are many communication techniques that would be suitable, such as XML messages over sockets, or object-style communications based on CORBA, DCOM, or RMI. However, once an on-the-wire protocol has been chosen, it becomes difficult to adopt another communication mechanism. Therefore, the framework is required to provide an abstract way of communicating between agents, which may be mapped onto different on-the-wire protocols. 5. Sources of information, such as databases or http servers, typically provide synchronous interactions, where the issuer of a query is blocked



until the result is returned. Such a type of query-oriented communication differs from the asynchronous type of communication usually supported by agent communication languages KQML and FIPA (Finin et al., 1997; FIPA, 1999). We want to support both mechanisms, because queryoriented communications are a natural paradigm in distributed information management, whereas asynchronous communications are suitable for more loosely coupled interactions between agents. In our framework, it is not a requirement to be directly compliant with standard agent communication languages such as KQML (Finin et al., 1997) or FIPA (FIPA, 1999). However, we believe that these standards are the result of a long experience of building agent systems, and we adopt some of their essential ideas, namely, declarative communications based on speech act theory and the organization of knowledge into discrete ontologies.

THE SOFAR AGENT FRAMEWORK: DESCRIPTION In this section, we describe SoFAR, the Southampton Framework for Agent Research. Most of the requirements of the previous section are in fact standard distributed computing requirements, and therefore, we looked at that community to find a solution to be used in the context of multiagent systems. We present such a solution below, and we extend and adapt it to support proper agent communications, as prescribed by KQML and FIPA agent communication mechanisms, among others.

A Distributed Computing View The distributed programming community has investigated numerous communication paradigms for distributed environments, such as message-passing libraries (e.g., MPI or PVM), communication channels (e.g., CSP or πcalculus), remote procedure call (RPC), and its object-oriented variant, remote method invocation (java, 1996). Nexus (Foster et al., 1996) is a distributed programming paradigm that provides the essence of a distributed object system and has inspired the model of communication used in SoFAR. Its implementation as a library has been used in the Globus project (www.globus.org), the basis of the Computational Grid (Foster & Kesselman, 1998). Here, we solely focus on the principles underlying Nexus, which has proven to be a generic mode of communication, both



efficient and scalable. It provides two key ideas: startpoint/endpoint pairs to refer to remote objects and remote service requests to start computations on remote objects. In Nexus, communication flows from a communication startpoint to a communication endpoint. A startpoint is bound to an endpoint to form a communication link. Many startpoints can be bound to a single endpoint, in which case, incoming communication is merged as in typical point-to-point message passing systems. Both startpoints and endpoints can be created dynamically; the startpoint has the additional property that it can be moved between processors using the communication operations we now describe. A communication link supports a single communication operation: an asynchronous remote service request (RSR). A RSR is applied to a startpoint by providing a procedure name and some data. The RSR transfers the data to the process in which the endpoint is located and remotely invokes the specified procedure, providing the endpoint and the data as arguments. A local address can be associated with an endpoint, in which case, any startpoint associated with the endpoint can be thought of as a “global pointer” to that address. Each communication link defines a unique communication medium, with which a specific communication method can be associated. There may be several supported protocols such as TCP, UDP, and HTTP. Each endpoint is associated with a table of handlers, from which one handler, i.e., a method in object-oriented terminology, is selected upon reception of an incoming RSR. In Nexus, a remote service request is a one-way communication; if results need to be returned, a second RSR has to be used.

Communications as Performatives The Nexus programming model provides the essence of a distributed object system, with means to refer to remote objects and to activate computations on them. Jennings and Wooldridge (2001) convincingly argue that agents are different to objects. We agree with their view and observe further differences as far as communications are concerned. If we return to a message-passing view of object-oriented systems, the messages sent and received by objects typically combine the exchanged data with the intended action (a query, or perhaps a statement of change) to be performed with that data in a way that makes the two inseparable. In addition, in object-oriented systems, classes have few or no restrictions on the methods they may implement or call. By comparison, the approach taken by many agent systems is to separate intention from content in communicative acts, abstracting and classifying the former according to Searle’s speech act theory (Searle, Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


1969). An agent’s communications are thereby structured and constrained according to a predefined set of performatives, which together make up an agent communication language (ACL). The number of different performatives varies between different ACLs. The simplest, such as Shoham’s Agent-0 (Shoham, 1993), have less than half a dozen, while the more complex, such as KQML or FIPA, have more than 20. Our experience is that a frugal but careful choice of performatives allows our agents to interact in as complex ways as if they were using a more complex agent communication language. In particular, FIPA and KQML contain specialized performatives for tasks such as forwarding messages or issuing calls for proposals, which we respectively see as functions of the communication layer or as terms to be defined in an application ontology. At the other extreme, Agent-0 relies on the composition of basic acts to perform more complex messages, which FIPA and KQML consider primitive. Our minimal set of performatives and their intuitive descriptions (Figure 1) are an attempt to strike a compromise between these extremes, being chosen in order to avoid the complexity and communication cost that composition would entail in the most common scenarios. Although there are important differences between agents and objects, there are some fundamental similarities, namely, that both are communicative entities. If the predominant object-oriented paradigm has shifted from message-passing to method invocation, we can similarly adopt a Nexus-like approach to interagent communications: the performatives in Figure 1 become the names of the procedures invoked by an RSR. In addition, methods may return values to their caller in order to satisfy our requirements of query-

Figure 1: Supported performatives Does the recipient know facts which match the query? What facts does the recipient know which match the query? inform The sender tells the recipient that the content is true uninform The sender tells the recipient that the content is false subscribe The sender asks to be informed of changes to facts which match the query unsubscribe The sender cancels a subscription request The sender asks the recipient to perform an action register The sender advertises their capabilities with a broker unregister The sender withdraws an advertisement query_if

query_ref



oriented communications: simple query performatives such as query_if or query_ref return values directly, rather than through an extended message exchange involving an inform message sent back to the querent. We defined three query performatives in our ACL, each with different semantics and expected responses: query_ref is an exhaustive search of an agent’s knowledgebase that returns all terms which satisfy the query; query_if verifies that a statement holds; subscribe is a temporal query in which the querent requests that the receiver enter into a contract to inform the querent each time a statement satisfying the query changes value (e.g., is asserted or retracted).

An Agent View of Communication The separation of intention from content is not the only difference we observe between object- and agent-based systems. Jennings and Wooldridge (2001) also note that while objects control their state, they do not control their behaviors. A method invocation is an irresistible request that objects must perform. Agents do not have this compulsion and are able to discriminate between messages based on their beliefs and the context of the message. This communication context includes information about the act of communication, such as the sender, receiver, sent time, message identifier, and conversation thread. An agent may use this to reject a message, to discriminate between senders, or to determine which thread of conversation a message belongs to. This information is usually not available in object systems but should definitely be made available in an agent system. Therefore, our model of agent communication is defined in terms of startpoints and endpoints, communication context, and performatives. A communication is based on a communication link defined by a startpoint and an endpoint. An endpoint identifies an agent’s ability to receive messages using a specific communication protocol and extracts messages from the communication link and passes them on to the agent. An agent’s endpoint is located where that agent resides. A startpoint is the other end of the communication link, from which messages get sent to an endpoint. There may be several startpoints for a given agent, each acting as a representative of the agent at remote locations. A startpoint can be seen as a “proxy” for an agent. As far as implementation is concerned, agents are regarded as objects that implement a predefined set of methods, corresponding to the performatives displayed in Figure 1. Communication between agents is performed by method invocation. Such a method is invoked on a startpoint, which takes care of packaging the method call up as a message and transmitting it to the endpoint. Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


Startpoints and endpoints have a crucial role: startpoints define the different components of the communication context, such as time or sender; endpoints construct the communication context and make it available to the agent. An agent is defined as an object that implements a method for each performative. Such methods are binary: the first argument is the term that is the subject of the message, whereas the second argument is the whole message, with its complete communication context. Performatives such as queries are intended to return a result: the result is transmitted back to the sender agent using the communication link that carried the query; the result of such performatives is returned as the value of the method invocation on the startpoint. Usually, a startpoint is attached to a single endpoint, and communication is point-to-point. If a startpoint is attached to several endpoints, a multicast mode of communication becomes possible. Note that performatives used in multicast mode are not supposed to return a result. Such a mode of communication is particularly useful for the performative inform in order to propagate information to several agents using a single communication act. (The implementation may use underlying multicast primitives or simulate multicast.)

Ontologies Agents exchange messages to communicate information about their environment or some problem domain, and so the content of the messages must be a representation of their world. As it is unreasonable to expect that all problem domains can be represented consistently by a single model, the design of specialized ontologies that form computational models of particular domains is now commonplace. Like “agent,” the word “ontology” has of late become popular with the computing community, and its meaning has become more vague as its use has increased. Gruber stated one of the best known definitions: “An ontology is an explicit specification of a conceptualisation” (Gruber, 1993). He goes on: ...we can describe the ontology of a program by defining a set of representational terms. In such an ontology, definitions associate the names of entities in the universe of discourse (e.g., classes, relations, functions, or other objects) with human-readable text describing what the names mean, and formal axioms that constrain the interpretation and well-formed use of these terms. Formally, an ontology is the statement of a logical theory. (Gruber, 1993)



Such a definition highlights an essential property of ontologies: they are a shared understanding of some domain that can be communicated across people and computers. A practical consequence of this property is that ontologies can be shared and reused among different applications; in particular, we believe that they are attractive in agent-based distributed information management. Pragmatically, an ontology is constituted by a specific vocabulary used to describe a certain reality, plus a set of explicit assumptions regarding the intended meaning of the vocabulary words. In SoFAR, ontologies are organized along a hierarchy based on single inheritance. Terms of ontologies are defined by the unique parent they extend and a (possibly empty) set of typed fields they contain. Terms are defined using an XML syntax; for instance, a Person can be defined as an entity composed of three fields.

It is sometimes convenient to define a term as abstract, which essentially declares a type, for which there cannot be any instance. An example of an abstract term is Entity, which is extended by two terms Person and Group.

The root of the hierarchy is the type Term, which is also abstract: any concept or relation in a SoFAR ontology is an extension of Term. SoFAR supports the usual primitive types found in most programming languages, such as numbers, Booleans, and strings.



Contractual Registration and Subscription Registration is the action by which an agent declares to the registry agent its ability to handle some messages. If the registry answers positively to a registration act, it commits to advertise the registered capability and to return it to agents that ask matching queries. As proof of its commitment, the registry issues a contract as a result of the registration act. As long as the contract remains live, the registry will retain the advertised capability. Conversely, if the agent that registered the capability desires to stop its advertising, it just has to terminate the associated contract. A similar mechanism exists for subscriptions. If an agent decides to answer positively to a subscription act, it commits to honor such a subscription: whenever a fact changes, it informs the interested subscriber. For each successful subscription act, a contract is issued as proof of commitment. The subscriber just needs to terminate the contract in order to suspend the flow of inform messages. There are several ways by which an agent can find information. They differ by when the result is returned and by the agent’s ability to control the flow of information: • Exhaustive searches (performative query_ref) and specific queries (performative query_if) complete their execution with the requested information. • An agent A can advertise (performative register) its desire to be informed about a given topic. Any agent in the system may inform A on the topic. Agent A is given little control over the flow of information. It can certainly stop advertising its interest, but there is no requirement for the other agents to stop propagating information to A. • In order to gain more control of the flow of information, agent A can subscribe (performative subscribe) to those agents knowledgeable on the topic. In return, each of these agents issues a contract, which may be used to terminate the individual subscriptions.

XML Agent Specification The framework as it stands is powerful enough to support any form of interaction in a multiagent system. However, the programming interface is still rather low-level: the programmer is required to repeat identical code too often, which makes the programming tedious. For instance, many agents need to find out the Registry and to advertise their capabilities; many agents need to find



about other agents by interrogating the Registry and to query these agents; many agents implement some of the performatives of Figure 1 and need to check whether the arguments received have the required type, and so on. In order to facilitate the programming, maintenance, building, and dissemination of agents, SoFAR offers an “agency compiler,” which takes an XML specification of a set of agents and provides: • A Java template, ready to be subclassed, implementing many of the tedious operations that all agents have to implement • An agent definition ready to be stored in a database for future reference • Files identifying the permissions to be granted to agents • A Makefile able to compile the agents, bundle them in a jar file, and export them for download As an example, Figure 2 shows the declaration of an agent communicating with RMI and accepting the inform performative with predicate LinkRequestInfo. Experience has shown that such XML declarations offer a clear separation between agent specification and implementation, which facilitates the development of agents.

Figure 2: Agent declaration

Liza <package>sofar.magnitude.index An agent able to get summarize url and parse according to keywords. sofar.ontology.base.* sofar.magnitude.ontology.* rmi sofar.magnitude.ontology.Magnitude <method name="inform"> LinkRequestInfo



IMPLEMENTATION OVERVIEW SoFAR implementation is about 100,000 lines of Java code, which we briefly overview now. SoFAR provides the programmer with an abstract class NullAgent that must be subclassed by all SoFAR agents. The class is responsible for the initialization of the agent, its unique identity, and its communication protocols; it also provides default void implementation of the performatives. The agency compiler compiles XML agent specifications into classes, which themselves subclass the NullAgent class. Finally, the programmer can implement their agent’s behavior by subclassing these classes. SoFAR supports communications over various implementations of RMI, including a UDP-based version of it, SOAP (Gudgin et al., 2001) and SSL. For each communication protocol, there is a StartPoint class implementing the performatives of Figure 1 and an EndPoint class able to pass messages onto the agent. Constructors for such classes are not public and are only accessed by a Boot class, called during an agent initialization which is when startpoint/endpoint pairs are initialized. A specific role of the startpoint is to fill the envelope of a communication act, and in particular, to identify the sender of the message. Allowing an agent programmer to pass an agent identity to the startpoint would be vulnerable to identity forging. Instead, we require startpoints to be initialized only by one method provided by SoFAR, which is automatically called when startpoints are deserialized. In order to avoid the identity of an agent to be communicated to other agents when a startpoint is communicated, the field storing the identity is declared as volatile. Additionally, SoFAR provides a class Platform responsible for the dynamic uploading and startup of agents. It is used in combination with a Launcher class, which operates in batch mode or interactively with the user, and specifies what agent, from which code base, with which parameters should be started. User authentication based on public key infrastructure takes place between launchers and platforms.

DISCUSSION AND RELATED WORK The exact nature and requirements of agency are still contentious subjects, with some disagreement in the literature. We follow Jennings and Wooldridge (2001) for our view of agency, regarding it as a software engineering tool for managing the complexity of system development. Nwana and Ndumu (1999) raise several points, namely, that the standardized ACLs contain too many performatives, some of which are used only infrequently, and that the effects on



ontology design of the interactions between a problem domain and a task to be accomplished are underinvestigated. If, as they suggest, the short-term solution is to create only limited domain ontologies, we believe that our use of mixed ontology expressions is a useful approach to bridging the gap between limited ontologies and broader general-purpose ontologies. SoFAR is not the only Java-based agent framework; there exist a number of others, the most notable of which are Zeus (Nwana et al., 1999), JAFMAS (Chauhan, 1997), JATlite (Petrie, 1996), FIPA-OS (FIPA-OS, 1999), Ajanta (Tripathi et al., 1999), and JACK (Jack, 1999). Zeus and JAFMAS adopt a similar approach, providing both a FIPA- or KQML-based communications infrastructure and a planning engine for handling rule-based conversations by means of automata models, and as such, are representative of a “traditional AI” approach to agency. JATlite also provides KQML-based messaging, but is flexible: it is designed to support other ACLs as necessary and does not place any restrictions on the internals of the agents. FIPA-OS (FIPA-OS, 1999) is a FIPA-compliant platform, which necessarily relies on a CORBA-based communication substrate; our approach can use CORBA as well as other technologies. Ajanta uses a method invocation approach not unlike ours, but does not constrain the methods used in performatives. JACK is a Java-based BDI framework, which provides facilities for formulating plans and reasoning about goals, but does not consider the pragmatics of communication or distribution issues. In its parsimonious approach to its ACL and the simplicity of its agents, SoFAR is most like Agent-0 (Shoham, 1993) and the systems derived from it, such as AgentBuilder (AgentBuilder, 1999) or PLACA (Thomas, 1994), although SoFAR does not provide support for planning abilities at a framework/language level as this latter system does. AgentBuilder is noteworthy as a commercial framework based on Shoham’s notion of agent-oriented programming (Shoham, 1993), but using KQML as its ACL rather than the much simpler Agent-0.

FUTURE TRENDS We are now observing a convergence of different technologies. Web Services are a distributed architecture under consideration by the World Wide Web consortium, based on XML protocols (Gudgin et al., 2001), and are particularly targeted at e-Business applications. The Grid paradigm is meant to enable access to computing power and resources with the ease similar to



electrical power (Foster & Kesselman, 1998). The Computational Grid is analogous to the electric power Grid, because it offers a consistent access to geographically distributed resources, irrespective of their physical location or access point. A number of Grid services have already been implemented as Web Services, bridging the gap between these two technologies. Grid technologies are enabling e-Science, the type of science that is collaborative and multidisciplinary, involving teams spanning institutions, states, countries, and continents. The “Grid problem” is defined as a flexible, secure, coordinated resource sharing among dynamic collections of individuals, institutions, and resources, which is also referred to as virtual organizations. Multiagent systems are particularly suited for the design of flexible virtual organizations, though scalable protocols for forming, running, and disbanding virtual organizations are still needed in the context of the Grid. In order to provide higher-level multiagent protocols, an agent framework should provide mechanisms that allow their definition and experimentation in an easy manner. Using our XML approach for declarative definition, we have prototyped a protocol compiler that transforms an XML definition of a protocol into a transition table directly interpretable by an agent. A challenging aspect of such a framework is to deploy it in order to test its scalability. We are developing two DIM applications to this end: a recommender system and a media streaming system (Cruickshank et al., 2001). We are planning trials that will specifically evaluate the framework scalability. Additionally, the deployment of such systems in multiple institutions raises concerns about security, which we are also investigating.

CONCLUSION We designed and implemented an agent communication mechanism derived from distributed computing techniques, but taking on board the reality of agency. Our approach is general and abstracts away from the communication details, supporting several on-the-wire protocols; it is lightweight and a proven route to scalability. In order to promote opportunistic reuse of agent services by other agents, our framework provides mechanisms to advertise information, query agents, and automatically manage subscriptions. A set of ontologies has been defined in order to support distributed information management tasks. The SoFAR framework has been the focus of a tremendous activity involving up to 60 researchers in the Intelligence, Agents Multimedia Group at Southampton. Training sessions were organized about agents, ontologies, and the actual framework implementation in Java. On three occasions, a group Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


activity, called “agentfest,” took place: during a three-day session, those researchers developed agents. As a result, SoFAR has now been adopted by several researchers for their everyday research. The framework has also been used for undergraduate teaching. We welcome other researchers to try SoFAR, which is available from www.sofar.ecs.soton.ac.uk.

ACKNOWLEDGMENTS This research is supported in part by EPSRC and QinetiQ project “Magnitude” reference GR/N35816, EPSRC project “HyStream” reference GR/M8407/01, and EU Project IST-2000-26135 FEEL. Some 60 researchers and 20 undergraduate students have used and developed SoFAR agents, and we wish to acknowledge their contribution. We also wish to thank Nick Gibbins for his initial contribution to the design of SoFAR.

REFERENCES AgentBuilder: An Integrated Toolkit for Constructing Intelligent Software Agents. Reticular Systems, Inc. Available from http:// www.agentbuilder.com/. 1999. Carr, L.A., De Roure, D.C., Hall, W., & Hill, G.J. (1995). The distributed link service: A tool for publishers, authors and readers. In Fourth International World Wide Web Conference: The Web Revolution, Boston, MA, pp. 647–656. O’Reilly & Associates. Appears in World Wide Web Journal (1). Chauhan, D. (1997). JAFMAS: A Java-Based Agent Framework for Multiagent Systems Development and Implementation. PhD thesis, ECECS Department, University of Cincinnati. Chen, L. & Sycara, K. (1998). WebMate: a Personal Agent for Browsing and Searching. In Proceedings of the Second International Conference on Autonomous Agents (132–139). Cruickshank, D., Moreau, L., & Roure, D.D. (2001). Architectural Design of a Multi-Agent System for Handling Metadata Streams. In The Fifth ACM International Conference on Autonomous Agents, Montreal, Canada. De Roure, D., Hall, W., Davis, H., & Dale, J. (1996). Agents for distributed multimedia information management. In Practical Application of Intelligent Agents and Multi-Agent Systems (91–102), London, UK.



De Roure, D., Hall, W., Reich, S., Pikrakis, A., Hill, G., & Stairmand, M. (1998). An Open Framework for Collaborative Distributed Information Management. In Seventh International World Wide Web Conference (WWW7), Brisbane, Australia, 30, 624–625. Amsterdam, New York: Elsevier. Published in Computer Networks and ISDN Systems. El-Beltagy, S., Hall, W., Roure, D.D., & Carr, L. (2001). Linking in context. In Proceedings of the 12th ACM Conference on Hypertext and Hypermedia (Hypertext ’01) (151–160). ACM, ACM Press. Finin, T., Labrou, Y., & Mayfield, J. (1997). Software Agents, J. Bradshaw, Ed., KQML as an Agent Communication Language. Cambridge, MA: MIT Press. FIPA: Foundation for Intelligent Physical Agents. http://www.fipa.org. fipaos (1999). FIPA-OS. http://www.nortelnetworks.com/products/announcements/fipa. Foster, I. & Kesselman, C., Eds. (1998). The Grid: Blueprint for a New Computing Infrastructure. Morgan Kaufman Publishers. Foster, I., Kesselman, C., & Tuecke, S. (1996). The Nexus Approach to Integrating Multithreading and Communication. Journal of Parallel and Distributed Computing, 37:70–82. Fountain, A.M., Hall, W., Heath, I., & Davis, H.C. (1990). MICROCOSM: An Open Model for Hypermedia with Dynamic Linking. In Rizk, A., Streitz, N., & André, J., (Eds.), Hypertext: Concepts, Systems and Applications (Proceedings of ECHT’90) (298–311). London, New York: Cambridge University Press. Gruber, T.R. (1993). Toward principles for the design of ontologies used for knowledge sharing. Technical Report KSL-93-04, Knowledge Systems Laboratory, Stanford University. Gudgin, M., Hadley, M., Moreau, J.-J., & Henrik Frystyk Nielsen (Eds.). (2001). Soap version 1.2. Technical report, World Wide Web Consortium. JACK Intelligent Agents User Guide. Agent Oriented Software Pty, Ltd., 1999. Jennings, N.R. & Wooldridge, M. (2001). Agent-Oriented Software Engineering. In Handbook of Agent Technology, AAAI/MIT Press. Lieberman, H. (1995). Letizia: An agent that assists web browsing. In Proceedings of the International Joint Conference on Artificial Intelligence, Montreal, Canada. Maes, P. (1994). Agents that Reduce Work and Information Overload. Communications of the ACM, 37(7): 31–40. Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


Nwana, H. & Ndumu, D. (1999). A perspective on software agents research. The Knowledge Engineering Review. Nwana, H., Ndumu, D., Lee, L., & Collis, J. (1999). Zeus: A tool-kit for building distributed multi-agent systems. Applied Artificial Intelligence Journal, 13(1): 129–186. Petrie, C. (1996). Agent-based engineering, the web, and intelligence. IEEE Expert. Searle, J. (1969). Speech Acts: An Essay in the Philosophy of Language. London, New York: Cambridge University Press. Shoham, Y. (1993). Agent-Oriented Programming. Artificial Intelligence, 60: 51–92. Thomas, S.R. (1994). The PLACA Agent Programming Language. In Wooldridge, M.J. & Jennings, N.R. (Eds.), ECAI-94 Workshop on Agent Theories, Architectures and Languages, Vol. 890 of Lecture Notes on AI. Heidelberg: Springer-Verlag. Tripathi, A., Karnik, N., Vora, M., Ahmed, T., & Singh, R.D. (1999). Ajanta — A Mobile Agent Programming System. Technical Report TR98-016, Department of Computer Science, University of Minnesota. Weal, M., Hughes, G., Millard, D., & Moreau, L. (2001). Open Hypermedia as a Navigational Interface to Ontological Information Spaces. Proceedings of the 12th ACM Conference on Hypertext and Hypermedia (HT’01), Aarhus, Denmark. Wooldridge, M. & Jennings, N.R. (1995). Intelligent Agents: Theory and Practice. Knowledge Engineering Review, 10(2).


68

Kefalas, Holcombe, Eleftherakis & Gheorghe

Chapter IV

A Formal Method for the Development of Agent-Based Systems P. Kefalas CITY College, Greece M. Holcombe University of Sheffield, UK G. Eleftherakis CITY College, Greece M. Gheorghe University of Sheffield, UK

ABSTRACT Recent advances in testing and verification of software based on formal specifications of the system to be built have reached a point where the ideas can be applied in a powerful way in the design of agent-based systems. The software engineering research has highlighted a number of important issues: the importance of the type of modeling technique used; the careful design of the model to enable powerful testing techniques to be used; the automated verification of the behavioral properties of the system; and the need to provide a mechanism for translating the formal models into executable software in a simple and transparent way. This chapter presents a detailed and comprehensive account of the ways in which some modern software engineering research can be applied Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

A Formal Method for Development of Agent-Based Systems 69

to the construction of effective and reliable agent-based software systems. More specifically, we intend to show how simple agents motivated from biology can be modeled as X-machines. Such modeling will facilitate verification and testing of an agent model, because appropriate strategies for model checking and testing are already developed around the Xmachine method. In addition, modular construction of agent models is feasible, because X-machines are provided with communicating features, which allow simple models to interact.

INTRODUCTION An agent is an encapsulated computer system situated in some environment that is capable of flexible, autonomous action in that environment in order to meet its design objectives (Jennings, 2000). There are two fundamental concepts associated with any dynamic or reactive system, such as an agent, that is situated in and reacting with some environment (Holcombe & Ipate, 1998): • The environment, which could be precisely or ill-specified or even completely unknown, but nevertheless, involves identifying the important aspects of the environment and the way in which they may change in accordance with the activities of the agent • The agent will be responding to environmental changes by changing its basic parameters and possibly affecting the environment as well. Thus, there are two ways in which the agent reacts, i.e., it undergoes internal changes, and it produces outputs that affect the environment Agents, as highly dynamic systems, are concerned with three essential factors: • A set of appropriate environmental stimuli or inputs • A set of internal states of the agent • A rule that relates the two above and determines what the agent state will change to if a particular input arrives while the agent is in a particular state One of the challenges that emerges in intelligent agent engineering is to develop agent models and agent implementations that are “correct.” According to Holcombe and Ipate (1998), the criteria for “correctness” are as follows: • The initial agent model should match with the requirements • The agent model should satisfy any necessary properties in order to meet its design objectives


70

•


The implementation should pass all tests constructed using a complete functional test generation method

All of the above criteria are closely related to three stages of agent system development, i.e., modeling, verification, and testing. Although agent-oriented software engineering aims to manage the inherent complexity of software systems (Wooldridge & Ciancarini, 2001), there is still no evidence to suggest that any methodology proposed leads toward “correct” systems. In the last few decades, there has been a strong debate on whether formal methods can achieve this goal. Academics and practitioners adopted extreme positions for or against formal methods (Young, 1991). It is, however, apparent that the truth lies somewhere between, and that there is a need for use of formal methods in software engineering in general (Clarke & Wing, 1996), while there are several specific cases proving the applicability of formal methods in agent development, as we shall see in the next section. Software system specification has centered on the use of models of data types, either functional or relational models such as Z (Spivey, 1989) or VDM (Jones, 1990) or axiomatic ones such as OBJ (Futatsugi et al., 1985). Although these have led to some considerable advances in software design, they lack the ability to express the dynamics of the system. Also, transforming an implicit formal description into an effective working system is not straightforward. Other formal methods, such as Finite State Machines (Wulf et al., 1981) or Petri Nets (Reisig, 1985) capture the essential feature, which is “change,” but fail to describe the system completely, because there is little or no reference at all to the internal data and how this data is affected by each operation in the state transition diagram. Other methods, like Statecharts (Harel, 1987), capture the requirements of dynamic behavior and modeling of data but are rather informal with respect to clarity and semantics. So far, little attention has been paid in formal methods that could facilitate all crucial stages of “correct” system development, modeling, verification, and testing. This chapter will introduce such a formal method, namely, X-machines, which closely suits the needs of agent development, while at the same time, being intuitive and practical.

FORMAL METHODS IN AGENT-BASED SYSTEMS In agent-oriented engineering, there have been several attempts to use formal methods, each focusing on different aspects of agent systems develop-



ment. One of them was to formalize PRS (Procedural Reasoning System), a variant of the BDI architecture (Rao & Georgeff, 1995) with the use of Z, in order to understand the architecture in a better way, to be able to move to the implementation through refinement of the specification and to be able to develop proof theories for the architecture (D’Iverno et al., 1998). Trying to capture the dynamics of an agent system, Rosenschein and Kaebling (1995) viewed an agent as a situated automaton that generates a mapping from inputs to outputs, mediated by its internal state. Brazier et al. (1995) developed the DESIRE framework, which focuses on the specification of the dynamics of the reasoning and acting behavior of multi-agent systems. In an attempt to verify whether properties of agent models are true, work has been done on model checking of multi-agent systems with reuse of existing technology and tools (Benerecetti et al., 1999; Rao & Georgeff, 1993). Toward implementation of agent systems, Attoui and Hasbani (1997) focused on program generation of reactive systems through a formal transformation process. A wider approach is taken by Fisher and Wooldridge (1997) who utilize Concurrent METATEM in order to formally specify multi-agent systems and then directly execute the specification while verifying important temporal properties of the system. Finally, in a less formal approach, extensions to UML to accommodate the distinctive requirements of agents (AUML) were proposed (Odell et al., 2000). In this chapter, we intend to show how simple agents motivated from biology can be modeled as X-machines. Such modeling will facilitate verification and testing of an agent model, because appropriate strategies for model checking and testing are already developed around the X-machine method. In addition, modular construction of agent models is feasible, because X-machines are provided with communicating features, which allow simple models to interact. Finally, tools developed for X-machines are briefly presented in order to demonstrate the practicality of the approach.

X-MACHINE DEFINITION An X-machine is a general computational machine introduced by Eilenberg (1974) and extended by Holcombe (1988) that resembles a Finite State Machine (FSM) but with two significant differences: • There is memory attached to the machine • The transitions are not labeled with simple inputs but with functions that operate on inputs and memory values


72


Figure 1: An abstract example of an X-machine; ϕi: functions operating on inputs and memory, Si: states. The general format of functions is: ϕ(σ, m) = (γ, m') if condition MEMORY

m

m’ γ

σ

input stream

φ1 φ3

S4

φ2

S1 φ2 S2

φ5

S3

φ2

output stream

φ4

φ4

These differences allow the X-machines to be more expressive and flexible than the FSM. Other machine models like pushdown automata or Turing machines are too low level and, hence, of little use for specification of real systems. X-machines employ a diagrammatic approach of modeling the control by extending the expressive power of the FSM. They are capable of modeling the data and the control of a system. Data is held in memory, which is attached to the X-machine. Transitions between states are performed through the application of functions, which are written in a formal notation and model the processing of the data. Functions receive input symbols and memory values and produce output while modifying the memory values (Figure 1). The machine, depending on the current state of control and the current values of the memory, consumes an input symbol from the input stream and determines the next state, the new memory state, and the output symbol, which will be part of the output stream. The formal definition of a deterministic stream X-machine (Ipate & Holcombe, 1998) is an 8-tuple XM = (Σ, Γ, Q, M, Φ, F, q0, m 0), where: • Σ, Γ is the input and output finite alphabet, respectively • Q is the finite set of states • M is the (possibly) infinite set called memory Φ is the type of the machine XM, a finite set of partial functions ϕ that • map an input and a memory state to an output and a new memory state, ϕ:Σ × M→Γ × M • F is the next state partial function that given a state and a function from the type Φ , denotes the next state; F is often described as a state transition diagram, F: Q ×Φ → Q • q0 and m0 are the initial state and memory, respectively



X-machines can be used as a core method for an integrated formal methodology of developing correct systems. The X-machine integrates the control and data processing while allowing them to be described separately. The X-machine formal method forms the basis for a specification/modeling language with a great potential value to software engineers. It is rather intuitive, while at the same time, formal descriptions of data types and functions can be written in any known mathematical notation. Finally, X-machines can be extended by adding new features to the original model, such as hierarchical decomposition and communication, which will be described later. Such features are particularly interesting in agent-based systems.

FORMAL AGENT MODELING Many biological processes seem to behave like agents, as, for example, a colony of ants. Much research has been based on such behavior in order to solve interesting problems (Dorigo & Di Caro, 1999). An important task of some ants is to find food and carry it to their nest. This can be accomplished by searching for food at random or by following pheromone trails that other ants have left on their return back to the nest (Deneubourg et al., 1990). While moving, an ant should avoid obstacles. Once food is found, an ant should leave Figure 2: A Finite State Machine modeling an ant’s behavior AT NEST pheromone

space

nest pheromone

FOLLOWING TRAIL

pheromone pheromone obstacle

nest

MOVING FREELY

space

food

space

pheromone space

LIFTING FOOD

food space

obstacle obstacle

AT OBSTACLE


74


Figure 3: An X-machine that models an ant ignore food

stay_at_nest

AT NEST move

move_to_food find_nest

LOOKING FOR FOOD

move_to_nest

move_to_food

drop_food GOING BACK TO NEST

move

find_nest find_food

MOVING FREELY

move_to_nest lift_food

AT FOOD

lift_food

got_lost

a pheromone trail while traveling back to its nest, thus implicitly communicating with other ants the destination of a source where food may be found. When the nest is found, the ant drops the food. Clearly, this is a reactive agent that receives inputs from the environment and acts upon these inputs according to the state in which the agent is. Such reactive agents can be fairly easily modeled by a FSM in a rather straightforward way by specifying the states and the inputs (percepts) to be used for state transitions (Figure 2). The FSM lacks the ability to model any nontrivial data structures. In more complex tasks, one can imagine that the actions of the agents will also be determined by the values stored in its memory. For example, an agent may know its position, remember the position of the food source or the position of obstacles, thus building a map of the environment in order to make the task eventually more efficient. Using FSM or variants of it (Brooks, 1986; Rosenschein & Kaebling, 1995) for such agents is rather complicated, because the number of states increases in combinatorial fashion to the possible values of the memory structure. X-machines can facilitate modeling of agents that demand remembering as well as reactiveness. Figure 3 shows the model of an ant that searches for food but also remembers food positions in order to set up its next goals. The behavior of obstacle avoidance is omitted for simplicity. Formally, the definition of the X-machines requires all elements of the 8tuple (Σ, Γ, Q, M, Φ, F, q0, m0). First, the input set consists of the percept and the x and y coordinate it is perceived:

Σ = ({space, nest} ∪ FOOD) × COORD × COORD



where [FOOD] is a basic type and COORD is of type integer, COORD ⊆ Z. The set of outputs is defined as a set of messages:

Γ = {“moving freely,” “moving to nest,” “dropping food,” …} The states in which the agents can be are five: Q = {At Nest, Moving Freely, At Food, Going Back to Nest, Looking for Food}. The state “Moving Freely” applies to an agent that does not have a specific goal and searches in random for a food source. The state “Going Back to Nest” applies when the agent is carrying a food item and is on its way back to its nest. The state “Looking for Food” applies when the agent has a goal, i.e., remembers where food is found during previous explorations of the terrain. The memory consists of three elements, i.e., what the agent carries, the current position of the agent, and the sequence of positions where food is found during its exploration: M = (FOOD∪ {none}) × (COORD × COORD)× seq (COORD×COORD) where none indicates that no food is carried. The initial memory and the initial states are, respectively: m0 = (none, (0,0), nil) q0 = “At Nest” It is assumed that the nest is at position (0,0). The next state partial function is depicted with the state diagram in Figure 3. The type Φ is a set of functions of the form: function_name(input_tuple, memory_tuple)→ (output, memory_tuple’), if condition. For example, here are some function definitions: lift_food((f,x,y),(none,(x,y),foodlist)) → (“lifting food,” (f,(x,y) < (x,y) :: foodlist>)), if f ∈ FOOD ∧ (x,y) ∉ foodlist Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

76


find_food( (f,fpx,fpy), (food,(x,y),foodlist) ) → (“more food,” (food,(x,y), < (fpx,fpy)::foodlist >)), if f ∈ FOOD ∧ f ∉ foodlist drop_food( (nest,0,0), (food,(x,y),foodlist) ) → (“dropping food,” (none,(0,0),foodlist)) find_nest( (nest,0,0), (none,(x,y),foodlist) ) → (“found nest again,” (none,(0,0),foodlist))

VERIFICATION OF AGENT MODELS Having designed a model for an agent, it would be desirable to verify whether it corresponds to the requirements, i.e., at all circumstances during the existence of the agent modeled in some way, its required properties are true in that model. Model checking is a formal verification technique, which determines whether given properties of a system are satisfied by a model. A model checker takes a model and a property as inputs, and outputs a claim that the property is true or a counterexample falsifying the property. In order to use model checking, the most efficient way to express a model is any kind of state machine, a CCS agent, a Petri Net, a CSP agent, etc. The most common properties to check are something will never occur or something will eventually occur. In Temporal Logic Model Checking (Clarke et al., 1986) a property is expressed as a formula in a certain temporal logic. The verification can be accomplished using an efficient breadth first search procedure, which views the transition system as a model for the logic and determines if the specifications are satisfied by that model. This approach is simple and completely automated, but it has one major problem, namely, the state explosion. The latter can be handled to some extent by a model checking variant, the symbolic model checking (McMillan, 1993). Model checking is sometimes proved to be more efficient than other formal verification techniques, such as theorem proving (Burch et al., 1992). On the other hand, model checking is formal as opposed to simulation, which may reveal inconsistencies and misconceptions in a model but does not guarantee completeness of the model with respect to requirements. Bearing in mind the most usual definition of a model, i.e., a labeled state transition graph, also called a Kripke structure (Kripke, 1963), model checking utilizes algorithms that, given a temporal logic formula, verify whether properties hold in the model. A Kripke structure is a tuple , where:



• • •

Q is a nonempty set of states R is a binary relation on Q, i.e., R⊆ Q× Q, which shows which states are related to other states L: Q → 2Prop is a truth assignment function that shows which propositions are true in each state, where Prop is a set of atomic propositions

Temporal logic formulae, e.g., CTL* formulae, are constructed through the use of operators combined with the path quantifiers A (meaning “for all paths”) or E (meaning “there exists a path”). The five basic CTL* operators are as follows (Emerson & Halpern, 1986): • X (next time) requires that a property hold in the next state • F (eventually) requires a property to hold at some state on a path • G (always) requires a property to hold at every state on a path • U (until) requires a property p to hold in a path until another property q holds • R (release) as a dual operator of U The above logic could be extended to accommodate past, present, and future properties of an agent system, as in FML (Fisher & Wooldridge, 1997). Having constructed a model of an agent as an X-machine, it is possible to apply existing model checking techniques to verify its properties. That would require transformation of an X-machine into another model that resembles a Kripke structure. Such a process exists, called the exhaustive refinement of an X-machine to a FSM, and results in a model in which CTL* formula may be applied. However, exhaustive refinement suffers two major disadvantages: • The loss of expressiveness that the X-machine possesses • The combinatorial explosion The former has to do with the memory structure attached to the Xmachine. In an equivalent FSM resulting from the process of refinement, the memory will be implicitly contained in states. It would be therefore impossible to verify that certain properties are true for some memory instances of the states of the original X-machine model, because this information is lost during the refinement process. The latter has to do with properties contained in the model’s memory but do not play any role in model checking with respect to some other properties. Exhaustive refinement may result in a possibly infinite state space, if such properties are included in the equivalent FSM, thus making model checking impossible.


78


In order to apply model checking in X-machines, temporal logic is extended with memory quantifier operators: • Mx, for all memory instances • mx, there exist memory instances which together with the basic temporal operators of CTL*, can form expressions suitable for checking the properties of an agent model. The resulting logic XmCTL can verify the model expressed as X-machine against the requirements, because it can prove that certain properties, which implicitly reside in the memory of X-machine, are true (Eleftherakis & Kefalas, 2001). For example, in an agent whose task is to carry food to its nest as in the example of Figure 3, model checking can verify whether food will eventually be dropped in the nest by the formula: AG [¬Mx (m1 ≠ none) ∨ EFMx (m1 = none) ] where m1 indicates the first element of the memory tuple. The formula states that in all states of the X-machine, it is true that either the ant does not hold any food or there exists a path after that state where eventually the ant does not hold any food. Another example is the following formula: E [Mx (m1 = none) U Mx (m1 ≠ none) ] i.e., there exists a path in which the ant eventually holds food, and in all previous states, the ant holds nothing. Also, another useful property to be checked is:

¬EFmx [ (m1 ≠ none) ∧ (m3 = nil) ] i.e., if the ant holds something, then the food list is not empty. The new syntax and semantics facilitate model checking of X-machines in two ways: • Expressiveness suited to the X-machine model • Effective reduction of the state space through selective refinement of the original X-machine model

COMPLETE TESTING OF AGENTS In the previous section, we focused on modeling of an agent and verification of the models specified with respect to requirements. Having ensured that Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


the model is “correct,” we need to also ensure that the implementation is “correct,” this time with respect to the model. This can be achieved through testing, but only under one important assumption, i.e., testing is complete. To guarantee “correctness” of the implementation, one must be certain that all tests are performed, and the results correspond to what the model has specified. Holcombe and Ipate (1998) presented a testing method, which is a generalization of Chow’s W-method (Chow, 1978) for FSM testing. It is proved that this testing method finds all faults in the implementation (Ipate & Holcombe, 1998). The method works based on the following assumptions: • The specification and the implementation of the system can be represented as X-machines • The X-machine corresponding to the specification and the X-machine corresponding to the implementation have the same typeΦ • •

Assuming the above, the method also requires that: The X-machine satisfies the design for test conditions Its associated automaton is minimal

The associated automaton of an X-machine is the conversion of the Xmachine to a FSM by treating the elements of Φ as abstract input symbols. The design for test conditions states that the type Φ of the two machines is complete with respect to memory and output distinguishable. A function ϕ∈Φ is called complete with respect to memory if:

∀m ∈ M, ∃ σ∈ Σ such that (m, σ ) ∈ dom ϕ A type Φ is called complete with respect to memory M, if any basic function will be able to process all memory values, that is if:

∀ϕ ∈ Φ , ϕ is complete with respect to M A type Φ is called output distinguishable if any two different processing functions will produce different outputs on each memory/input pair, that is if:

∀ ϕ1,ϕ2∈ Φ if ∃m ∈ M, σ ∈ Σ such that for some m1',m2', ∈ M, γ ∈Γ ϕ1(m, σ ) = (γ , m1') and ϕ2(m,σ ) = (γ , m2'), then ϕ1 = ϕ2. If Φ is not complete, then additional input symbols may be introduced such as to make processing functions complete (Holcombe & Ipate, 1998). Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

80


Figure 4: An X-machine that satisfies the design for test conditions ignore_nest

stay_at_nest move

MOVING FREELY

AT NEST

ignore_food

move

move_to_food found_nest

ignore_space

lift_food

ignore_food

move_to_food

FOLLOW TRAIL

HAVE FOOD

ignore_space

lift_food

move_to_nest

move_to_food

ignore_nest

In Figure 4, the X-machine illustrates a model of an ant that looks for food at random or follows the pheromone trail to find food and nest. The input set is Σ = {space, nest, pheromone, food}. The X-machine satisfies the design for test conditions and its associated automaton is minimal. When these requirements are met, the W-method may be employed to produce the k-test set X of the associated automaton, where k is the difference of the number of states of the two associated FSMs. The test-set X consists of processing functions for the associated automaton, and it is given by the formula: X = S(Φ k+1 ∪ Φ k ∪ … ∪ Φ ∪ {ε })W where W is a characterization set, and S a state cover. Informally, a characterization set W⊆ Φ * is a set of processing functions for which any two distinct states of the machine are distinguishable. The state cover S⊆ Φ * is a set of processing functions such that all states are reachable by q0. The W and S sets in the agent X-machine in Figure 4 are as follows: W= {stay_at_nest, move move_to_food, found_nest} S= {ε , move, move move_to_food, move_to_food lift_food} The derived test-set X, for k = 0, i.e., model and implementation are considered as FSM with the same number of states, is the following: Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


X= {move move move_to_food, move move move move_to_food, move ignore_nest move move_to_food, move lift_food found_nest, move move_to_food lift_food found_nest, move_to_food lift_food found_nest, move_to_food lift_food ignore_space found_nest,…} The fundamental test function is defined recursively and converts these sequences into sequences of inputs of the X-machine. Let XM = (Σ, Γ, Q, M, Φ, F, q0, m0) be a deterministic stream X-machine with Φ complete with respect to M, and let q ∈ Q, m∈ M. A function tq, m: Φ *→ Σ * will be called a test function of M with respect to q and m, will be defined recursively as (Ipate & Holcombe, 1998): tq,m(ε ) = or tq, m(ϕ1…ϕnϕn+1 )=

ε (the empty input symbol) tq, m(ϕ1ϕ2…ϕn )σn+1 if ∃ a path q,q1,…,qn-1,qn, in M starting from q, where σn+1 is such that (mn, σn+1)∈domϕn+1 and mn is the final memory value computed by the machine along the above path on the input sequence tq, m(ϕ1ϕ2…ϕn) tq, m(ϕ1ϕ2…ϕn),otherwise

The test-set containing sequences of inputs for the ant X-machine is the following: {space space pheromone, space space space, pheromone, space nest space pheromone, space food nest, space pheromone food nest, pheromone food nest, pheromone food space nest, …} The test-set so produced is proved to find all faults in the agent implementation. The testing process can therefore be performed automatically by checking whether the output sequences produced by the implementation are identical with the ones expected from the agent model.

AGENTS AS AGGREGATION OF BEHAVIORS Agents can be modeled as a stand-alone (possibly complex) X-machine as shown in the previous section. However, an agent can also be viewed as a set of simpler components, which model various different behaviors of the Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

82


agent. This fits with the three principles of complex agent systems: decomposition, abstraction, and organization (Jennings, 2001). Another approach for reactive agents is described in the subsumption architecture (Brooks, 1991), in which behaviors can communicate with each other in order to result in a situated agent with the desired overall robust performance. Similarly, Collinot et al. (1996) developed the Cassiopeia method, in which agents are defined by following three steps: • Identifying the elementary behaviors that are implied by the overall task • Identifying the relationship between elementary behaviors • Identifying the organizational behaviors of the system A methodology of building communicating X-machines from existing stand-alone X-machine is developed so that modeling can be split into two separate activities: • The modeling of X-machine components • The description of the communication between these components • • • •

The approach has several advantages for the developer who: Does not need to model a communicating system from scratch Can reuse existing models Can consider modeling and communication as two separate distinct activities Can use existing tools for stand-alone and communicating X-machines

Let us discuss the above one by one. Certain approaches for building a communicating system require a brand new conceptualization and development of a system as a whole. This approach has a major drawback, i.e., one cannot reuse existing models that have been already verified and tested for their “correctness.” Often, in agent systems, components from other agent systems are required. A desirable approach would be to conceptualize the system as a set of independent smaller models, which need to communicate with each other. Thus, one does not need to worry about the individual components, in which model-checking techniques and testing are applied but only by appropriately linking those components. This would lead to a disciplined development methodology, which implies two distinct and largely independent development activities, i.e., building models and employing communication between them. Also, this means that existing languages and tools for modeling, model checking, and testing are still useful and can be further extended to support larger communicating systems. Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


Several theoretical approaches for communicating X-machines have been proposed (Balanescu et al., 1999; Cowling et al., 2000; Barnard, 1998). In this section, we will describe the one that focuses on the practical development of communicating systems but also subsumes all others (Kefalas et al., 2001). In this approach, the functions of an X-machine, if so annotated, read input from a communicating stream instead of the standard input stream. Also, the functions may write to a communicating input stream of another X-machine. The normal output of the functions is not affected. The annotation used is the solid circle (IN port) and the solid box (OUT port) to indicate that input is read from another component and output is directed to another component, respectively. For example, function ϕ in Figure 5 accepts its input from the model xm1 and writes its output to model x-m2. Multiple communications channel for a single X-machine may exist. Another example is a simple form of communication between two ants. Assume that one ant is responsible for notifying another ant about the position of the food source. In order to achieve communication, the X-machines should be modified as illustrated in Figure 6. The function lift_food of the X-machine model ant2 becomes: Figure 5: An abstract example of a communicating X-machine component X-machine standard

x-m2 x-m1

standard

input stream

output stream

channel for receiving message from x-m1

φ IN port

channel for sending message to x-m2

OUT port

Figure 6: Ant2 X-machine sends a message about food position in Ant1 X-machine, by utilizing a communicating port ANT1 drop_food

ant2

ANT2 ant1

GOING BACK TO NEST move_to_nest

MOVING FREELY

move_to_nest find_food

AT FOOD

lift_food

move

move_to_nest


84


lift_food( (f,x,y),(none,(x,y),foodlist) ) → (OUTx-m ant2(f,x,y), (f,(x,y),)), if f ∈ FOOD ∧ (x,y)∉ foodlist and the function find_food of X-machine model ant1 becomes: find_food(INx-m ant1 (f,fpx,fpy), (food,(x,y),foodlist) ) → (“more food,” (food,(x,y), < (fpx,fpy)::foodlist >)), if f ∈ FOOD ∧ (fpx,fpy) ∉ foodlist Function find_food of ant2 may be modified accordingly to write a message to the OUT port, if needed. The approach is practical, in the sense that the designer can separately model the components of an agent and then describe the way in which these components communicate. This allows a disciplined development of situated agents. Practically, as we shall see later, components can be reused in other systems, because the only thing that needs to be changed is the communication part. In the following, we will use communicating X-machines to model the collective foraging behavior of a colony of honeybees, as it is fully compatible with the rules used by foraging honeybees (Vries & Biesmeijer, 1998) that include specifications for: • Traveling from the nest to the source • Searching for the source • Collecting nectar from the source • Traveling back to the nest • Transmitting the information about the source (the dancing of the returning bee) • The reaction of a bee in the nest to the dancing of a nest mate A foraging bee can be modeled according to a set of independent behaviors, which constitute the components of the overall agent. Figure 7 shows some of the behaviors of a foraging bee modeled as simple X-machines, with an input set S and a memory tuple M. Each machine has different memory, inputs (percepts), and functions. Some states and functions were named differently to show the modularity of the approach. It is assumed that the bee perceives: • Empty space to fly (space) • The hive (nest) Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


Figure 7: The behaviors of a foraging bee modeled separately as Xmachine components Behavior Traveling from nest to source: Σ = {space, source} M = (bee_pos, source_pos) q0 = at source

X-machine model fly_to_source

AWAY FROM SOURCE

find_source

AT SOURCE keep_flying

keep_flying

Searching for the source: Σ = {space, source} M = (bee_pos, source_pos) q0 = flying

search detect_source

AT NECTAR SOURCE

FLYING fly_back fly_back

Collecting nectar from the source: Σ = {nectar, rbee} M = (nectar_amount) q0 = carrying nothing Traveling back to the nest: Σ = {nest, space} M = (nest_pos) q0 = at hive Transmitting information about the source (dancing): Σ = {fbee, space, nest, source} M = (bee_pos, source_pos) q0 = in the nest Reacting to the information transmitting by the dancing: Σ = {space, lost, source_pos} M = (status, source_pos) q0 = flying freely

• • • •

collect_nectar

CARRYING NOTHING

CARRYING NECTAR transfer_nectar detect_hive

OUT OF HIVE

AT HIVE fly_out

dancing

find_source fly_out

IN THE NEST

OUT OF NEST fly_in keep_fly_out

fly

loose_source_info get_info_from_dance

FLYING FLEELY

FLYING TO SOURCE fly

ignore_dance

The source of nectar (source) An amount of nectar (nectar) Other bees, i.e., foraging bees (fbee) or receiving bees (rbee) Understands when it has lost its orientation (lost)


86


The memory of each X-machine holds information on the bee, the source, and the nest positions (bee_pos, source_pos and nest_pos), the amount of nectar carried (nectar_amount), and its status (employed or unemployed). For example, consider the X-machine modeling the dancing behavior. Its functions are defined as follows (Gheorghe et al., 2001): dancing(fbee, (bee_pos, source_pos)) → (“dancing”, (bee_pos, source_pos)) fly_out(space, (bee_pos, source_pos)) → (“flying out”, (bee_pos’, source_pos)) fly_in(nest, (bee_pos, source_pos)) → (“flying in”, (bee_pos’, source_pos)) find_source(source,(bee_pos,source_pos))→ (“sourcefound”,(source_pos, source_pos)) keep_fly_out(space,(bee_pos ,source_pos)) → (“keep flying”,(bee_pos’, source_pos)) where bee_pos, bee_pos’, source_pos∈Set_of_positions. The bee position can be calculated by some external function or some other X-machine. Figure 8 shows in detail how communication can be achieved directly by various honeybees, e.g., an employed foraging bee sends the source position to another foraging bee through the dancing behavior: dancing(fbee, (bee_pos, source_pos)) → (OUTx-m reacting(source_pos), (bee_pos, source_pos)) while an unemployed foraging bee reads the source position by the function: Figure 8: An example of two communicating behaviors; an employed bee sends information about the source position to an unemployed bee x-m reacting dancing fly_out

OUT OF NEST fly_in

loose_source_info

get_info_from_dance

find_source

IN THE NEST

X-M DANCING

x-m dancing

fly

FLYING FLEELY

FLYING TO SOURCE fly

ignore_dance keep_fly_out

Behavior of a foraging employed bee

x-m dancing

X-M REACTING

Behavior of a foraging unemployed bee



get_info_from_dance(INx-m dancing(source_pos),(unemployed, nil)) → (“getting source info”, (employed, source_pos)). If the foraging bee is currently employed, it just ignores the message: ignore_dance(INx-m dancing(source_pos),(employed, source_pos)) → (“ignoring source info”, (employed, source_pos)). The same communication takes place when a foraging bee transfers the amount of nectar that it is carrying to a receiving bee waiting at the hive. The separate behaviors can be put together in a communicating X-machine model. Figure 9 shows the complete foraging bee system, which is made up of component X-machines, which communicate via channels. Each machine works separately and concurrently in an asynchronous manner. Each machine can read inputs from a communication channel instead of its standard input steam. Also, each machine can send a message through a communication channel that will act as input to functions of another component. The figure shows an extra component, i.e., the perception system of the bee, which provides percepts to various behaviors. In addition, more machines can be modeled, as for example, the X-machine that builds an environment map (positions of obstacles, nest, food items, etc.). Information held in the memory of this machine could be used to efficiently move around the environment, or even to model a proactive behavior for the agent. Thus, modeling of an agent can be incremental by providing components, which will advance further the level of intelligent behavior.

Figure 9: Communicating X-machine modeling agent bee through aggregation of behaviors FORAGING BEE

detect_space detect_nest

PERCEPTING

detect_bee

space, source

traveling from nest to source

space, source

searching for source

detect_source

rbee

collecting nectar from source and transferring it

detect_nectar

got_lost

nest, space

traveling back to nest

ubee, space, nest, source

dancing

space, lost

reacting to dancing


88


The whole system works as follows: an employed bee accepts inputs from the environment, which cause transitions in the X-machine components that model its individual behaviors. Having found a source and after collecting nectar (appropriate transitions have been performed in source and environment X-machines), the bee returns to the hive, and on the sight of another foraging bee, performs the dancing which, as shown earlier, transmits the information of the source position. An unemployed bee accepts the input from its communication port and changes its status to employed. It can then perceive inputs from the environment and travels to the source. The whole process may then be repeated. In parallel, other bees can do the same in an asynchronous manner. The approach is practical, in the sense that the developer can separately model the components of an agent and then describe the way in which these components-behaviors communicate. Also, components can be reused in other systems, because the only thing that needs to be changed is the communication part. For example, the behavior for avoiding obstacles is a component of any biology-inspired agent, e.g., foraging bees or ants. The major advantage is that the methodology also lends itself to modular model checking and testing strategies in which X-machines are individually tested as components, while communication is tested separately with existing methodologies, mentioned earlier.

MULTI-AGENT MODELS Modeling multi-agent systems requires the consideration of the means of communicating between agents, in order to coordinate tasks, cooperate, etc. Also, modeling of artificial environments in which agents act imposes the need of exchanging “messages” between agents and the environment. For example, a number of ants modeled as X-machines need to interact with their environment, which contains few seeds (food items) that are also modeled as Xmachines. These two ants, which may be instances of the same model class, can communicate with the environment in order to achieve the desired behavior, i.e., to lift a heavy seed that is far from the abilities of a single agent (Figure 10). Several behaviors are omitted for the sake of exposition. The ant is capable of lifting a food item only if the strength it possesses is bigger than the weight of a food item. In any other case, cooperation between ants is necessary, which can be achieved by communication of ants and the food item machine. The method used in the previous section to describe communicating X-machines can also serve this purpose.



Figure 10: Ants model cooperating in the lifting task through communication with the environment store

ANT1

AT NEST walk_to_nest

free_walk

free_walk

SEED

find_nest

CARRYING FOOD

lift find_nest continue_free_walk

MOVING FREELY

force_applied

BECOME LIGHTER

found lift_food

find_food

AT FOOD

attempt_lift

found lift

force_applied

ON GROUND

found

force_applied force_released

LIFTED

force_released

put_down

BECOME HEAVIER

ANT2

found

force_released put_down

In addition, one may require agents that resemble one another, i.e., they have a common set of behaviors, but extra individual behaviors that determine some task that characterizes their individuality. For example, in a colony of foraging bees, some bees are responsible for collecting nectar from a source as well as have the ability to “inform” others about the location of the source (the dance of the foraging bee), while other bees are responsible for storing the nectar in the hive (Seely & Buhrman, 1999). Nevertheless, all have the ability to fly, receive nectar, etc. Such situations can be modeled with X-machines, as long as there is a way to define classes of models and instances of these classes, which can inherit generic behaviors from the chain of hierarchy. Figure 11 demonstrates the whole multi-agent system that models the colony of the honeybees as well as the environment and its various components, such as the source and the nest. The same happens when coordination is achieved by some other agent through scheduling and decomposition of a large task into smaller tasks, which are manageable by individual agents. Ready-made components may be used to complete the multi-agent system, as discussed before. If, however, these components bear some incompatibility with the rest of the agents, communication and interaction protocols may be required. One can easily imagine Xmachines that act as a synthetic glue between agents, modeling, for example,


90


Figure 11: The model of the honeybees’ multi-agent system and its interaction with the environment. ENVIRONMENT space

nest

source nectar rbee ubee ebee lost

FORAGING BEE (employed) NEST

detect_nest

PERCEPTING

space, source space, source

traveling from nest to source searching for source

rbee

collecting nectar from other bee

detect_nectar

got_lost

detect_bee

RECEIVING BEE

SOURCE

detect_source

detect_space

collecting nectar from source and transfer it

nectar

FORAGING BEE (unemployed)

traveling back to nest

nest, space ubee, space, nest, source

dancing source_pos

space, lost

reacting to dancing

reacting to dancing

KQML parsers (Finin et al., 1997) or the Contract Net Protocol (Davis & Smith, 1983).

TOOLS X-machines modeling is based on a mathematical notation, which, however, implies a certain degree of freedom, especially as far as definition of functions are concerned. In order to make the approach practical and suitable for the development of tools around X-machines, a standard notation is devised, and its semantics are fully defined (Kapeti & Kefalas, 1999). Our aim was to use this notation, namely, X-machine Definition Language (XMDL), as an interchange language between developers who could share models



Figure 12: The use of XMDL in system development X-machine model

coding

XMDL model

check

Development Tool

Model Checking

Syntax Analyzer

Parser Tool

Completeness Checking Algorithms

Testing

Parser Tool Compiler

Parser Tool

written in XMDL for different purposes (Figure 12). To avoid complex mathematical notation, the language symbols are completely defined in ASCII code. A model developed with XMDL consists of the following: • The model for a component X-machine • The coding referring to possible communication of this component with other X-machines Briefly, a XMDL-based model is a list of definitions corresponding to the 8-tuple of the X-machine. The language also provides syntax for: • Use of built-in types such as integers, Booleans, sets, sequences, bags, etc. • Use of operations on these types, such as arithmetic, Boolean, set operations, etc. • Definition of new types • Definition of functions and the conditions under which they are applicable In XMDL, the functions take two parameter tuples, i.e., an input symbol and a memory value, and return two new parameter tuples, i.e., an output and a new memory value. A function may be applicable under conditions (if-then) or unconditionally. Variables are denoted by “?”. The informative where in combination with the operator “ [with: #init_state = . #init_memory ]. #communication of <model name>: Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


reads from <model name> writes <message> to <model name> [where <expression> from (memory|input|output) ]. For example, the following list represents the communication part of an ant agent: #model ant1 instance_of ant. #communication of ant1: find_food reads from ant2. #model ant2 instance_of ant. #communication of ant2: lift_food writes (?f,?x,?y) to ant1 where (?f,?x,?y) from input (?f,?y,?y). • • • •

The XMDL language is: Orthogonal, because there are rather few primitive structures that can be combined in rather few ways Mark-up, because it supplies the users with the freedom of a nonpositioning language Strongly typed, because it performs type checking, set checking, and function checking, as well as exception handling Very close to the mathematical notation

DISCUSSION We have demonstrated how formal methods and, specifically, X-machines can be used as the core method in intelligent agent engineering process. First, we argued that X-machines are better suited to the requirements of agent systems than simple automata as far as their modeling is concerned. The use of memory structure and functions provide a better mapping to the requirements of an agent model, and they are proven essential when simple automata fail to provide a finite model. Second, the relation of X-machines to FSM gives the opportunity to exploit existing techniques for verification and testing. Verification is achieved through a variant of temporal logic adopted for X-machines, namely, XmCTL, which can be utilized for model checking of agents, i.e., to verify whether certain desired properties hold in the agent model. Testing refers to the agent implementation, and it is complete, in the sense that a variant of the W-method can automatically generate all possible test cases for the agent. Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

94


Third, a methodology for building complex agent systems through aggregation of behaviors as well as multi-agent systems is discussed. Communicating Xmachines are demonstrated as a powerful extension to X-machines that facilitate modeling of large-scale systems, without the loss of the ability to use model checking and testing in each individual component. Finally, a notation that makes X-machine practical was presented. The XMachine Definition Language provides the appropriate syntax and semantics in order to make the formal method: • Practical, in the sense that complex mathematical definitions can be written using simple text • General, because the notation becomes standard, and therefore, the same model can be used throughout a variety of tools • Modular, because it can accommodate component-based communicating X-machine model development • Disciplined, because modeling of components and communication between them are regarded as two separate developing activities Currently, XMDL forms the basis for the development of various tools, such as automatic translations to programming languages such as PROLOG and JAVA (Kefalas, 2000) or transformations to other formal notations such as Z (Kefalas & Sotiriadou, 2000). Also, various other tools such as animators, test-case generators, and model checkers have XMDL as an interchange language. Finally, XMDL is recently extended to accommodate certain objectoriented features of X-machine modeling as well as cardinality of communication in a multi-agent system (Kefalas et al., 2001).

CONCLUSION Further work is required in developing the tools further and also in building more examples of agent systems and carrying out the testing and verification described. Because the X-machine method is fully grounded in the theory of computation, it is fully general and will be applicable to any type of computational task. The paradigm of the X-machine is also convenient when it comes to implementing the models in an imperative programming language. In fact, the translation is more or less automatic. The existence of the powerful testing method described lays the foundation for the method to be used in potentially critical applications. Finally, the model checking developments will lead to a situation in which one of the key issues in agent software engineering, namely,



how can we guarantee that the agent system constructed will exhibit the desired emergent behavior, can be solved, or at least substantial progress toward this goal will be achieved.

REFERENCES Attoui, A. & Hasbani, A. (1997). Reactive systems developing by formal specification transformations. Proceedings of the 8th International Workshop on Database and Expert Systems Applications (DEXA 97), (pp. 339–344). Balanescu, T., Cowling, A.J., Gheorgescu, H., Gheorghe, M., Holcombe, M., & Vertan, C. (1999). Communicating stream X-machines systems are no more than X-machines. Journal of Universal Computer Science, 5 (9), 494–507. Barnard, J. (1998). COMX: a design methodology using communicating Xmachines. Journal of Information and Software Technology, 40, 271– 280. Benerecetti, M., Giunchiglia, F., & Serafini, L. (1999). A model checking algorithm for multiagent systems. In J.P. Muller, M.P. Singh, & A.S. Rao (Eds.). Intelligent Agents V (LNAI Volume 1555). (pp. 163–176). Heidelberg: Springer-Verlag. Brazier, F., Dunin-Keplicz, B., Jennings, N., & Treur, J. (1995). Formal specification of multi-agent systems: a real-world case. Proceedings of International Conference on Multi-Agent Systems (ICMAS’95), (pp. 25–32). Cambridge, MA: MIT Press. Brooks, R.A. (1986). A robust layered control system for a mobile robot. IEEE Journal of Robotics Automation, 2 (7), 14–23. Brooks, R.A. (1991). Intelligence without reason. In J. Mylopoulos & R. Reiter (Eds.). Proceedings of the 12th International Joint Conference on Artificial Intelligence (pp. 569–595). Morgan Kaufmann. Burch, J.R., Clarke, E.M., McMillan, K.L., Dill, D.L., & Hwang, J. (1992). Symbolic model checking: 1020 states and beyond, Information and Computation, 98 (2), 142–170. Chow, T.S. (1978). Testing software design modeled by finite-state machines. IEEE Transactions on Software Engineering, 4 (3), 178–187. Clarke, E. & Wing, J.M. (1996). Formal methods: state of the art and future directions, ACM Computing Surveys, 28 (4), 626–643. Clarke, E.M., Emerson, E.A., & Sistla, A.P. (1986). Automatic verification of finite state concurrent systems using temporal logic specifications. ACM Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

96


Transactions on Programming Languages and Systems, 8 (2), 244– 263. Collinot, A., Drogul, A., & Benhamou, P. (1996). Agent oriented design of a soccer robot team. In Proceedings of the 2nd International Conference on Multi-Agent Systems, (pp. 41–47). Cowling, A.J., Gheorgescu, H., & Vertan, C. (2000). A structured way to use channels for communication in X-machines systems. Formal Aspects of Computing, 12, 485–500. Davis, R. & Smith, R. (1983). Negotiation as a metaphor for distributed problem solving. Artificial Intelligence, 20 (1), 63–109. Deneubourg, J.-L., Aron, S., Goss, S., & Pasteels J.-M. (1990). The selforganizing exploratory pattern of the Argentine ant. Journal of Insect Behavior, 3, 159–168. Dorigo, M. & Di Caro, G. (1999). The ant colony optimization meta-heuristic. In D. Corne, M. Dorigo, & F. Glover (Eds.), New Ideas in Optimization. New York: McGraw-Hill, pp. 11–32. Eilenberg, S. (1974). Automata, Machines and Languages. Vol. A. New York: Academic Press. Eleftherakis, G. & Kefalas, P. (2001). Model checking safety critical systems specified as X-machines. Matematica-Informatica, Analele Universitatii Bucharest, 49 (1), 59–70. Eleftherakis, G. & Kefalas, P. (2001). Towards model checking of finite state machines extended with memory through refinement. In G.,Antoniou, N.,Mastorakis, & O.,Panfilov (Eds.). Advances in Signal Processing and Computer Technologies, World Scientific and Engineering Society Press, pp. 321–326. Emerson, E.A. & Halpern, J.Y. (1986). Sometimes and not never revisited: On branching time versus linear time, Journal of the ACM, 33, 151–178. Ferber, J. (1996). Reactive distributed artificial intelligence: principles and applications. In Foundations of Distributed Artificial Intelligence. New York: John Wiley & Sons, pp. 287–314. Finin, T., Labrou, Y., & Mayfield, J. (1997). KQML as an agent communication language. In J.M. Bradshaw (Ed.). Software Agents, AAAI Press, pp. 291–316. Fisher, M. & Wooldridge, M. (1997). On the Formal Specification and Verification of Multi-Agent Systems. International Journal of Cooperating Information Systems, 6(1), 37–65. Futatsugi, K., Goguen, J., Jouannaud, J.-P., & Meseguer, J. (1985). Principles of OBJ2. In B. Reid (Ed.). Proceedings, Twelfth ACM Symposium on Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


Principles of Programming Languages, (pp. 52–66). Association for Computing Machinery. Georghe, M., Holcombe, M., & Kefalas, P. (2001). Computational models for collective foraging, Biosystems, Elsevier Science, 61, 133-141. Harel, D. (1987). Statecharts: A visual approach to complex systems. Science of Computer Programming, 8 (3). Holcombe, M. (1988). X-machines as a basis for dynamic system specification. Software Engineering Journal, 3 (2), 69–76. Holcombe, M. & Ipate, F. (1998). Correct Systems: Building a Business Process Solution. London: Springer-Verlag. Inverno, d’ M., Kinny, D., Luck, M., & Wooldridge, M. (1998). A formal specification of dMARS. In M.P. Singh, A. Rao, & M.J. Wooldridge (Eds.). Intelligent Agents IV (LNAI Volume 1365), Heidelberg: SpringerVerlag, pp. 155–176. Ipate, F. & Holcombe, M. (1998). Specification and testing using generalised machines: a presentation and a case study. Software Testing, Verification and Reliability, 8, 61–81. Jennings, N.R. (2000). On agent-based software engineering. Artificial Intelligence, 117, 277–296. Jennings, N.R. (2001). An agent-based approach for building complex software systems. Communications of the ACM, 44 (4), 35–41 . Jones, C.B. (1990). Systematic software development using VDM (2nd ed.). New York: Prentice Hall. Kapeti, E. & Kefalas, P. (2000). A design language and tool for X-machines specification. In D.I. Fotiadis & S.D. Nikolopoulos (Eds.). Advances in Informatics, World Scientific Publishing Company, pp. 134–145. Kefalas, P. (2000). Automatic Translation from X-machines to Prolog. Technical Report TR-CS01/00, Department of Computer Science, CITY Liberal Studies. Kefalas, P. (2000). XMDL user manual: version 1.6. Technical Report TRCS07/00, Department of Computer Science, CITY Liberal Studies. Kefalas, P., Eleftherakis, G., & Kehris, E. (2001). Modular modeling of largescale systems using communicating X-machines. In Y. Manolopoulos & S. Evripidou (Eds.). Proceedings of the 8th Panhellenic Conference in Informatics (pp. 20–29). Livanis Publishing Company. Kefalas, P. & Sotiriadou, A. (2000). A compiler that transforms X-machines specification to Z. Technical Report TR-CS06/00, Department of Computer Science, CITY Liberal Studies. Kripke, S. (1963). A semantical analysis of modal logic I: normal modal Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

98


propositional calculi. Zeitschrift fur Mathematische Logik und Grundlagen Mathematik, 9, 67–96. McMillan, K.L. (1993). Symbolic Model Checking. Dordrecht: Kluwer. Odell, J., Parunak, H.V.D., & Bauer, B. (2000). Extending UML for agents. In Proceedings of the Agent-Oriented Information Systems Workshop at the 17th National Conference on Artificial Intelligence. Rao, A.S. & Georgeff, M.P. (1993). A model-theoretic approach to the verification of situated reasoning systems. In R. Bajcsy (Ed.). Proceedings of the 13th International Joint Conference on Artificial Intelligence (IJCAI’93), (pp. 318–324). Morgan Kaufmann. Rao, A.S., & Georgeff, M. (1995). BDI Agents: from theory to practice. In Proceedings of the 1st International Conference on Multi-Agent Systems (ICMAS-95), (pp. 312–319). Reisig, W. (1985). Petri nets — an introduction. EATCS Monographs on Theoretical Computer Science, 4, Heidelberg: Springer-Verlag. Rosenschein, S.R. & Kaebling, L.P. (1995). A situated view of representation and control. Artificial Intelligence, 73 (1–2), 149–173. Seeley, T.D. & Buhrman, S.C. (1991). Group decision making in swarms of honey bees. Behavioral Ecology and Sociobiology, 45, 19–31. Spivey, M. (1989). The Z notation: A reference manual. New York: Prentice Hall. Vries, H. de & Biesmeijer, J.C. (1998). Modeling collective foraging by means of individual behavior rules in honey-bees. Behavioral Ecology and Sociobiology, 44 109-124. Wooldridge, M. & Ciancarini, P. (2001). Agent-oriented software engineering: The state of the art. LNCS (Vol. 1957, pp. 1-28) Heidelberg: Springer-Verlag. Wulf, W.A., Shaw, M., Hilfinger, P.N., & Flon, L. (1981). Fundamental Structures of Computer Science. New York: Addison-Wesley. Young, W.D. (1991). Formal methods versus software engineering: is there a conflict? In Proceedings of the 4th Testing, Analysis, and Verification Symposium, pp. 188–899.


Engineering Emotionally Intelligent Agents

99

Chapter V

Engineering Emotionally Intelligent Agents Penny Baillie University of Southern Queensland, Australia Mark Toleman University of Southern Queensland, Australia Dickson Lukose Mindbox Inc., USA

ABSTRACT Interacting with intelligence in an ever-changing environment calls for exceptional performances from artificial beings. One mechanism explored to produce intuitive-like behavior in artificial intelligence applications is emotion. This chapter examines the engineering of a mechanism that synthesizes and processes an artificial agent’s internal emotional states: the Affective Space. Through use of the affective space, an agent can predict the effect certain behaviors will have on its emotional state and, in turn, decide how to behave. Furthermore, an agent can use the emotions produced from its behavior to update its beliefs about particular entities and events. This chapter explores the psychological theory used to structure the affective space, the way in which the strength of emotional states can be diminished over time, how emotions influence an agent’s perception, and the way in which an agent can migrate from one emotional state to another. Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

100

Baillie, Toleman & Lukose

INTRODUCTION This chapter examines the affective, core mechanism of the Emotionally Motivated Artificial Intelligence (EMAI) architecture: the Affective Space. The design of the affective space is motivated by research into the affective agent domain and the identification of the shortage of agent architectures that have the capacity for decision making influenced by a mechanism that simulates human emotional intelligence (Goleman, 1995). Picard (1997) affirms the importance of this type of emotionally influenced decision making in computers. She suggests that if affective decision making were integrated into computers, it would provide a competent solution to emulating the intelligence of humans, where decisions are often made with insufficient knowledge, limited memory, and relatively slow processing speeds. Emotions are an integral part of human decision making, and by giving machines a similar mechanism, it could help in problem solving, where options cannot be fully explored, data is incomplete, and processing time is short. In recent times, there have been a number of architectures designed to produce artificial agents capable of expressing and processing emotions [Silas T Dog (Blumberg, 1997), PETEEI (El-Nasr, 1998), EBC Framework (Velasquez, 1999), Emotional Agents (Reilly, 1996), and Creatures (Grand et al., 1997)]. These models cover a wide range of affective phenomena and differ broadly between implementations. As a complete examination of these architectures would constitute a publication in its own right, a comprehensive review of these models will not appear in this chapter. This chapter begins by examining a brief overview of the EMAI architecture. This is followed with an in-depth examination of the affective space; the architecture’s primary emotion-producing mechanism. The chapter continues by examining how emotions are produced and processed by the affective space. Finally, an examination of some future trends for the use of emotional agents is given.

OVERVIEW OF THE EMAI ARCHITECTURE The EMAI architecture consists of several major processing and knowledge representation areas. These areas work together in a complex network of information gathering, manipulating, and updating. As shown in Figure 1, any agent implemented using the EMAI architecture receives external sensory data from its environment. It also processes internal sensory data from the Motivational Drive Generator in the Knowledge Area. Internal State Registers



101

simulate low-level biological mechanisms (such as hunger and fatigue). The sensory processor and the affective space integrate both types of sensory data into the agent’s belief system via an emotional filtering process. Sensory input (internal and external) received by the sensory processor may activate goals in the agent. The goals are processed by the agent’s constructive area, where plans are chosen that will satisfy these goals. These plans are generated by the Event Space Generator, which generates a series of competing events that could be performed by the agent to satisfy its goals. Before the agent schedules the events for execution in the Deliberate Area, each event is ordered by the Intention Generator in collaboration with the affective space and sorted from most liked to least liked. Once the agent has the list of events sorted by emotional affect, the behavior actuator begins executing them in order. The events executed by the behavior actuator at any moment in time become the EMAI agent’s outward behavior. Although the goal-orientated nature of the EMAI architecture is equally as important as the emotional mechanisms, the focus of this chapter is on the engineering of the emotional aspects of the architecture, therefore, goals and plans will not be discussed here. For further information on the goal setting and planning mechanisms in EMAI architecture, the reader is encouraged to see Baillie (2002). The affective space, shown in Figure 1, is the focal point of the agent architecture. The affective space acts as an emotional filter that influences an EMAI agent’s perception of its beliefs and the environment and, as a consequence, how it behaves. While there have been a number of emotional agents that have preceded EMAI, none have structured emotions in the unrivaled multidimensional sense as does the EMAI’s affective space.

THE AFFECTIVE SPACE The affective space is a new and unique concept in the domain of affective computing. Smith and Ellsworth’s experimentation and analysis (Smith & Ellsworth, 1985), identifies six orthogonal appraisal dimensions used to describe the scope of the affective space. These dimensions are pleasantness, P; anticipated effort, E; certainty, C; attentional activity, A; responsibility, R; and control, O; across 15 emotions: happiness, sadness, anger, boredom, challenge, hope, fear, interest, contempt, disgust, frustration, surprise, pride, shame, and guilt. Each emotion exists within the affective space, as a discrete emotion point comprised of values for each of the six appraisal dimensions, as shown in Table 1. Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

102


Figure 1: Emotionally motivated artificial intelligence architecture

Figure 1. Emotionally Motivated Artificial Intelligence Architecture

Table 1: Mean locations of emotional points Emotion

P

E

C

A

R

O

Happiness

-1.46

-0.33

-0.46

0.15

0.09

-0.21

Sadness Anger Boredom Challenge Hope Fear Interest Contempt Disgust Frustration Surprise Pride Shame Guilt

0.87 0.85 0.34 -0.37 -0.5 0.44 -1.05 0.89 0.38 0.88 -1.35 -1.25 0.73 0.6

-0.14 0.53 -1.19 1.19 -0.18 0.63 -0.07 -0.07 0.06 0.48 -0.66 -0.31 0.07 0

0 -0.29 -0.35 -0.01 0.46 0.73 -0.07 -0.12 -0.39 -0.08 0.73 -0.32 0.21 -0.15

-0.21 0.12 -1.27 0.52 0.31 0.03 0.7 0.08 -0.96 0.6 0.4 0.02 -0.11 -0.36

-0.36 -0.94 -0.19 0.44 0.15 -0.17 -0.13 -0.5 -0.5 -0.37 -0.97 0.81 1.13 1.13

1.51 -0.96 0.12 -0.2 0.35 0.59 0.41 -0.63 -0.19 0.22 0.15 -0.46 -0.07 -0.29



103

As the agent’s emotional state or mood is stored as six attributes, each representing a value in each of the appraisal dimensions of the affective space, the agent’s mood can be compared (by a distance measure) to any pure emotion (for example, happiness or sadness) in the affective space. The pure emotion physically closest to the agent’s emotional state is used to describe the agent’s mood. Through the implementation of the affective space, an EMAI agent cannot only synthesize a mood for itself, but it can also associate emotions with environmental elements.

ASSIGNING A MULTIDIMENSIONAL EMOTIONAL STATE In the EMAI architecture, an emotional state can be assigned to any entity, be it an event element (person, place, object, etc.), a whole event, or a partially ordered set of events (a plan). By assessing an element, event, or plan using the six appraisal dimensions a six coordinate point can be determined. This point, when plotted in the affective space can be compared to the 15 pure emotion points that exist within (see Table 1). For any element, e, an Ωe value can be determined that represents the emotional state, Ω evoked by the item. The Pe, Ee, Ce, Ae, Re, and Oe values are calculated for an item. The representation for Ωe is shown in Expression 1. (1)

Ω

= {Pe , E e , C e , Ae , Re , Oe}

e

Given Ωe for an element, the emotional state can be deduced for the purpose of expression in natural language by determining the distance that Ωe is from each of the 15 pure emotion points. To determine natural language terminology to best describe the emotion of Ωe, the distance between the Ω1…Ω Ω15) is element’s emotional state point and each of the pure emotions (Ω calculated using Expression 2. (2)

∆Ω = ( P e − P j) + ( E e − E j) + (C e − C j) + ( Ae − A j) + ( Re − R j) + (O e − O j) 2

2

2

2

2

2

i

where 15 values will be calculated for j = 1…15. The pure emotion for the element expressed in English, Eme, closest to Ωe is then determined by using Expression 3.


104

Baillie, Toleman & Lukose (3) e

= emotion_ name(min(U ∆ )) Ωi j =1

⊃

Em

15

where the function min returns the pure emotion point in closest proximity to Ωe, and the function emotion_name converts the pure emotion point into English terminology. For example, assume an element with an emotional state point of Ωe = [15, 87, 35, -30, 10, -50]. To find the name of the emotion that best describes the element’s associated emotional state, the first step is to find the distance between this point and the 15 pure emotion points in the affective space (shown in Table 1) using Expression 3. The results are shown in Table 2. In this example, Ωe is associated with the emotion shame. The multidimensional affective decision making model of the EMAI agent uses an emotional prioritizing procedure to select behaviors. However, before the prioritizing can begin, an event must be assigned an emotion. The emotion assigned to the event is calculated by considering the agent’s emotional responses to each of the elements in the event. As defined in Baillie et al. (2000), an event E is made up of a set of actions a, a set of objects o, occurs at a time t, and has context c, as in Expression 4. (4)

E

= {a , o , c , t }

Table 2: Distance between an item’s emotional state and pure emotions in affective space Emotion happiness sadness anger boredom challenge hope fear interest contempt disgust frustration surprise pride shame guilt

∆Ω 2.08 2.22 2.18 2.15 1.59 1.46 1.70 2.11 1.68 1.74 1.93 2.68 1.64 0.80 0.84



105

For each element e, in an event E, the emotional state, Ωe is defined as per Expression 1. Based on the outcome of event E, the agent will assign a weighting w, to the emotional state of each of the elements, e. As the weighting of an element and the resulting emotional state with respect to an event are dynamic, the time, t, at which the emotional state is being calculated, must also be taken into consideration. Therefore, the emotional state resulting from an event E, written as ΩE, t is calculated as Expression 5. (5)

n

ΩE ,t = ∑ we,t Ωe,t e =1

where n is the number of elements associated to event E, and n

0 ≤ w ≤ 1 and ∑ w = 1 e

e=1

e

Once an event has been completed, each of the elements involved in the event has its emotional states updated with respect to the change in the emotional state of the agent (or the agent’s mood) evoked by the outcome of the event, ΩO,t+1. ΩO,t+1 represents the emotional state of the agent after an event has occurred, where O stands for the outcome emotion and t + 1 is the time at which the event ended. This value is not the same as the emotional state assigned to the event after execution. A change in the emotional state of the agent occurs when the values for each of the appraisal dimensions (P, E, C, A, R, O) are updated during and after an event. While each of six appraisal values of an individual element involved in the event influence how these values are changed for the agent’s emotional state, it cannot be determined before the event occurs what the final emotional state will be. The agent can only speculate. For example, an event that the agent believes will make the agent happy may fail during execution, may take longer to complete than initially calculated, or may require extra effort. The resulting emotional state in this example may be sad rather than the expected happy. Therefore, ΩO,t+1 cannot be calculated by combining the appraisal dimensions of the elements of an event but can only be determined after the event has been executed. The change in the emotional state of an event is calculated using Expression 6.

∆ =Ω Ω

O ,t +1

− ΩE ,t

(6)

After this has been calculated, the emotional states for each element in the event set can be updated as Expression 7.


106


Ω

e,t +1

= Ωe,t + we,t +1 ∆Ω

(7) E

Instead of the element taking on the final emotional state of the event, the previous emotional state of the element is taken into account along with the weighting of effect the element had in the resulting emotional state for the event. For example, assume an event E with two elements A and B. At the beginning of the event, elements A and B have emotional states of happiness, and weightings of 0.2 and 0.8, respectively, thus:

Ω Ω

A, t

= [-146, - 33, - 46, 15, 9, - 21]

B ,t

= [-146, - 33, - 46, 15, 9, - 21]

w w

A, t

= 0 .2

B ,t

= 0.8

This would result in the emotional state for the event before execution as:

Ω

E ,t

= wA,t ΩA,t + wB,t ΩB,t = 0.2 × [-146, - 33, - 46,15, 9, - 21] + 0.8 × [-146, - 33, - 46,15, 9, - 21] = [-146, - 33, - 46,15, 9, - 21]

In other words, a happy event. Assuming after the event has occurred, the outcome results in an emotional state of happiness, and A and B are still weighted the same, then A and B can be updated as shown below.

Ω

A,t +1

= Ω A,t + w A,t +1 (ΩO ,t +1 − ΩE ,t )

= ([-146, - 33, - 46, 15, 9, - 21] + 0.2([-146, - 33, - 46, 15, 9, - 21] - [-146, - 33, - 46, 15, 9, - 21]) = [-146, - 33, - 46, 15, 9, - 21]

Ω

B ,t +1

= ΩB ,t + wB ,t +1 (ΩO ,t +1 − ΩE ,t )

= ([-146, - 33, - 46, 15, 9, - 21] + 0.8([-146, - 33, - 46, 15, 9, - 21] - [-146, - 33, - 46, 15, 9, - 21]) = [-146, - 33, - 46, 15, 9, - 21]



107

Figure 2: Emotional State of an Event

When the same event’s emotional state needs to be calculated in the future, it will again be evaluated to be happy. However, if the final emotional state of the event changes from its initial state, the emotional states for the elements will change. For example, imagine that the same event is to occur again. This time, the elements A and B are weighted initially as before; however, after the event, the emotional state is sad, and the weightings of A and B are now 0.4 and 0.6, respectively. Figure 2 graphically represents the elements A and B and the event E before execution with a vector representing the change in emotional state of the agent after the execution of the event E. Given the new emotional state of E and the weighting of A and B after the event, the new emotional states for A and B can be calculated as:

Ω

A,t +1

= Ω A,t + wA,t +1 (ΩO ,t +1 − ΩE ,t )

= ([-146, - 33, - 46, 15, 9, - 21] + 0.4([87, - 14, 0, - 21, - 36, 151] - [-146, - 33, - 46, 15, 9, - 21]) = [-146, - 33, - 46, 15, 9, - 21] + [93.2, 7.6, - 18.4, - 14.4, - 18, 68.8] = [−58.8,−25.4,64.4,6,−9,47.8]

Ω

B ,t +1

= ΩB ,t + wB ,t +1 (ΩO ,t +1 − ΩE ,t )

= ([-146, - 33, - 46, 15, 9, - 21] + 0.6([87, - 14, 0, - 21, - 36, 151] - [-146, - 33, - 46, 15, 9, - 21]) = [-146, - 33, - 46, 15, 9, - 21] + [139.8,11.4,-27.6,-21.6,-27,103.2] = [−6.2,−21.6,−73.6,−6.6,−18,82.2]

This moves the emotional point for each of the elements closer to the final emotional state for the event by using the weightings to add a portion of the vector from Et to Et+1 onto the emotional states for A and B. The result of this is represented graphically in Figure 3.


108


Figure 3: Emotional state of event elements after event execution

Once each element of the event has had its emotional state updated, the emotional state for the event is updated using the new values for A and B, thus:

Ω

E,t +1

= wA,t+1 ΩA,t+1 + wB,t+1 ΩB,t+1

= 0.4 × [−58.8,−25.4,64.4,6,−9,47.8] + 0.6× [−6.2,−21.6,−73.6,−6.6,−18,82.2] = [-27.24,- 23.12,- 18.4,- 1.56,- 14.4,68.44]

This results in a new emotional state that, in this example, lies between the elements of A and B as shown in Figure 4. In the case of elements A and B, from having found the emotional states (Ω values) for both, A can be said to be associated with surprise, and B can be associated with sadness. Once the agent has calculated the emotional points for each of the events and in turn each element involved in an event, the agent can use this emotional point in its affective decision-making process. Figure 4: Emotional state of event after event execution



109

AFFECTIVE DECISION MAKING WITH EMAI An EMAI agent can decide which of its planned events to execute by calculating the resulting emotional effect that performing the event will have on the agent’s mood. The agent’s mood ΩEMAI is the result of weighing the importance of an event wE and summating the m number of resultant emotional points for all episodes of all events performed by the agent E and defined by Expression 8. f

Ω

(8)

m

EMAI

= ∑ wE ΩE E =1

m

where 0 ≤ wE ≤ 1 and ∑ wE = 1 E =1

Given a number of competing events that have the same priority, the agent will select an event that will most likely update the agent’s emotional state to a more preferred emotional state. If the agent would prefer to have an emotional state closer to happy, it would select the event that would, when combined with its current emotional state, make the agent happy. During the course of affective decision making, the inherent nature of the affective space deals with two concepts that have been difficult to handle in other contemporary affective agent architectures: emotion blending and emotion decay. These are discussed in the following sections.

Emotion Blending Emotion blending is a concept devised by psychologists such as Plutchik (Lefton, 1994), in which it is stated that there exist several basic emotions and that more complex emotions can be created by blending different combinations of the basic emotions. The problem with any emotion synthesis in affective computing will be determining which emotions, when blended together, produce other emotions. In affective agent architectures such as Em (Reilly, 1996) and PETEEI (El-Nasr, 1998), emotions are represented by individual variables or gauges within the architecture. The problem occurs when the gauges need to be coordinated. This requires complex programming within the agent to describe the relationships that exist between the individual variables. The EMAI architecture eliminates the need for defining the complex relationships and programming coordination between emotion state variables because of the nature of the affective space. The relationships that exist among the pure emotions are defined by their separating distances. Rather than having Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

110


individual variables that represent the emotions integrated into the architecture and using these same variables to record the strength of the emotion of the agent, the EMAI architecture separates the two. The 15 pure emotions are fixed points within the affective space. They represent the relationship between the pure emotion and the appraisal dimensions but do not record the agent’s mood or the emotional state of any assessed item. The emotional state of an item or the agent is separate from each pure emotion. Because of this representation, theoretically, an infinite number of pure emotions could be added to the architecture with little effort. Furthermore, as the emotional state point for an item (or the agent) is independent of the pure emotion points, the value for the emotional state point can freely change and not affect the pure emotion points. Although a discrete point within the affective space represents the pure emotions, they exist within an affective continuum. Unlike other affective agent architectures that model pure emotions as individual and discrete variables with complex rules for the blending of the values, the affective space naturally defines blended emotions, as they are represented by the space that exists between the pure emotion points. While the emotional state for an item may fall close to the proximity of a pure emotion within the affective space, most often it will not fall exactly at the same spot as a pure emotion. Table 3 displays the results of blending equal parts of two of the 15 pure emotion points.

Emotional State Decay The same characteristics of the affective space that allow for a natural emotion blending process also address the issue of emotional state decay. Emotional decay is the mechanism by which an agent that is experiencing an emotional state will, over time, begin to have the strength of that emotional state reduced. This ensures that the agent does not stay in the same emotional mood. Contemporary affective agent models [PETEEI (El-Nasr, 1998), Em (Reilly, 1996), Silas T. Dog (Blumberg, 1997), Yuppy (Velasquez, 1999)] address the issue of emotional state decay using decay rate functions. Because these models represent emotions and their associated strengths as individual variables, emotional decay must be dealt with manually. This means coordinating the rate of decay of an emotion variable with other emotion variables. For example, an agent that is experiencing happiness will need to have the value of the happiness variable decreased as the value of the variable representing sadness is increased. Otherwise, the agent will be experiencing two opposing emotions at the same time. The closest that an emotional state point can get to the pure emotion is to be positioned at the exact same location in the affective space as the pure Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

Shame pride

shame contempt shame

shame hope shame

disgustingguilt

shame

hope

guiltypride

hope

shame

guiltyboredom

guilt

shamefulsadness challengingshame

contempt shame

fearfulshame hope

shamefulcontempt

shamefuldisgust

frustratingshame

hope

guilt

happypride

hope contempt

boredpride challengingpride hopefulpride

hope

happiness

hope

disgustingpride

hope

happiness

surprisedhappiness

hope contempt

hope interest hopefulsurprise

hope

interestingsurprise

hope

hope

frustratedsadness angryfrustration

disgust challengingfrustration

hope

disgustedhappiness

disgustedsadness contempt

boreddisgust challengingdisgust

hopefuldisgust

hope

hope

frustration angrycontempt disgust

frustration

hopefulcontempt

hopefulfrustration

interestedhappiness

hope contempt hope

challenginginterest

hopefulinterest

fearfulfrustration

hope

fearfulsadness frustration

disgust

fearfulchallenge

hopefulfear

hope

interest

hopefulsadness

contempt

hopefulboredom

challenginghope

frustratingcontempt

interest

fear

contempt

challenginganger

contempt

Pride fearfuldisgust

Surprise hope

Frustration disgustingcontempt

Disgust frustration

Contempt hope

Interest

guiltyshame

Fear disgust

Hope boredhappiness

Challenge boredsadness

Boredom disgust

Anger

hope

Sadness frustration

Happiness hope

Guilt

Shame

Pride

Surprise

Frustration

Disgust

Contempt

Interest

Fear

Hope

Challenge

Boredom

Anger

Sadness

Engineering Emotionally Intelligent Agents 111

Table 3: Resultant emotions from blending pure emotions

emotion. If this is the case, the agent is said to be experiencing a pure emotion. When the emotional state of the item being assessed changes, the position of the emotional state point in the affective space also changes. As the emotional state point moves away from one pure emotion in the affective space, it moves toward another. Therefore, the emotional state point for the assessed item

Table 3. Resultant Emotions from Blending Pure Emotions


112


cannot be representing two opposing emotional states at the same time. The affective space, by its nature, generates naturally decaying emotions without any extra programming or coordination effort. When the emotional state point for the item is at its most pleasant, the strength of the emotion happiness is the greatest. This is because the point for the pure emotion of happiness is closest to the pleasantness axis at the pleasant end. As the emotional state point moves toward, for example, sadness, the strength of the happiness emotion with respect to the item’s emotional state point reduces.

FUTURE TRENDS FOR AFFECTIVE AGENTS A number of application areas for affective agents have been developed in recent years. The majority of these areas focus on the computer’s ability to predict the emotional state of the user. Several projects undertaken by the MIT Media Laboratory focus on wearable computers and affective tutoring systems that rely on their ability to determine the user’s emotion state from a series of physiological changes. The research that has gone into the EMAI agent has taken a different perspective. Rather than predicting the emotions of the user, the computer is generating emotions that affect its own behavior. This type of behavior would be beneficial in the creation of an intelligent tutoring system that not only predicts the user’s emotional state but also could adapt to it and provide an emotional and personalized interface that could appear sympathetic to the user. Another obvious application area for the EMAI architecture is in the development of computer game and interactive fiction characters. Stern, a leading computer game developer, in Stern (1999) comments that what computer gamers are really seeking is affective AI and, in particular, the creation of believable artificial life or A-life. The EMAI architecture achieves a partial step in the direction toward achieving affective AI in computer characters by modeling an emotion-rich decision-making mechanism that gives a computer character a perceived personality and the ability to use emotions to adapt to its environment. Another area in which the EMAI architecture could be applied is that of e-commerce. E-commerce systems are increasingly recognizing the importance of giving additional value to users by providing personalized transactional experiences (Meech & Marsh, 2000). Two specific factors apparent in the impact of personalized transactions in e-commerce are those of trust and personality and how these may be integrated into Web-based interfaces. The principle behind a lifelike intelligent agent to act as a host at a Web site is the Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


113

same as for the creation of computer game characters. The aim is to create a suspension of disbelief for the user. The qualities necessary for this are personality, emotions, social relationships, and the illusion of life (Mateas, 1997). With the ability to create an artificial personality with emotions, the EMAI architecture could be used to develop a personalized face for an ecommerce business by providing a face for the company and a personal shopping guide. Another area worthy of research is that of the appraisal dimensions selected for constructing the affective space. The dimensions chosen for EMAI’s initial affective space belonged to an existing appraisal model by Smith and Ellsworth (Smith & Ellsworth, 1985). As the objective of the research herein was not to evaluate the appraisal dimensions but to evaluate the use of such a model in an intelligent agent architecture, different appraisal dimensions were not evaluated. It has become evident during this research that while the appraisal dimensions used for EMAI’s affective space have been successful in representing emotional states with respect to events, there are many other dimensions that could be equally as effective. Furthermore, things other than events can evoke emotional states, and in these cases, an entirely different set of appraisal dimensions may be more appropriate. Besides the six appraisal dimensions included in the Smith and Ellsworth (1985) model used for the definition of EMAI’s affective space, psychologists have suggested many others. These include legitimacy (Roseman et al., 1990), compatibility, novelty, and perceived obstacle (Scherer, 1982) and danger or threat (Smith & Lazarus, 1990). Because the Smith and Ellsworth model was developed through experimental procedures and evaluated as a successful theory for relating their chosen appraisal dimensions with 15 separate emotional states, it formed an excellent basis for the affective space of the EMAI architecture. As the objective of the engineering of the EMAI architecture was to develop an emotionally intelligent artificial being and not expand the psychological theory of emotion, further research into other appropriate appraisal dimensions and other emotional states has not been explored. This is not to suggest that this type of research would not be beneficial to the advancement of affective computing. In fact, the appraisals in the EMAI architecture are restricted to the emotional states evoked from events. However, emotional states can be evoked from a number of other cognitive appraisals. Music, for example, can evoke a variety of emotional states in the listener (Juslin & Madison, 1999). Tempo, timing, articulation, key, tone, and instrument choice influence the emotions experienced by the listener. Color is Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

114


another concept that can also elicit an emotional response from the viewer (Jacobson & Bender, 1996). Emotions experienced as the result of viewing a specific color differ culturally. For example, black is associated with funerals and mourning in the Western World, and white and purple are associated with funerals and mourning in the Eastern world. However, the research clearly shows that differing colors can evoke different emotional states. This suggests that emotional experiences are not restricted to the cognitive appraisals of Smith and Ellsworth’s model. Exactly what appraisal dimensions are appropriate in assessing concepts other than events is ill defined in current literature. It could be the case that in some domains, cognitive appraisal models of emotion theory are inadequate for the full realization of a reasonable, emotionally intelligent artificial being. For example, besides the Western World philosophy on emotion, there are also a number of Eastern views. The traditional Chinese medicine of acupuncture defined 12 main Qi channels that run through the body (Rogers, 2000). These channels relate to major organs and intestinal parts of the body, such as the stomach, lungs, and large intestine. The lungs are said to relate to the emotion of sadness, the stomach corresponds to pensiveness, and the heart happiness. In the Chinese martial art of Tai Chi, emotions relate to the universal five elements: wood, fire, earth, water, and metal. Each element corresponds to a negative and a positive emotional state. For wood, the positive emotion is humor, and the negative emotion is anger. Exactly how these views on emotional theories integrate into AI is unclear. However, it is worth further investigation. It may be that some Western philosophies are suited to a particular set of application areas and Eastern philosophies to others. As can be seen from the aforementioned application areas, the EMAI architecture is ideal for integration into systems that require an intelligent system that requires an interface with personality and emotional capabilities. Its emotional abilities allow it to create a personalized and nonthreatening interface for computer systems and Web interfaces. Its emotion development mechanisms allow it to not only to meet the current needs of the user but also to be able to perceive future needs.

CONCLUSION Although emotions were originally thought to impede rational thinking, new schools of thought have arisen that consider emotions a vital part of rational intelligent behavior and necessary not only for survival, but also intellectual Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


115

success. In recent years, important emphasis has been placed on the concept of emotional intelligence. This has led many researchers in the area of artificial intelligence to examine the contribution that theories on emotion could have in the development of intelligent agents. Thus, the domain of affective computing was born. The contributions presented in this chapter endeavor to further enhance the domain of affective computing by providing a different perspective on the way in which emotions are modeled in an artificial agent. The continuous affective space in the EMAI agent is used to replace the discrete variables used to represent individual emotions in EMAI’s predecessors. In short, the EMAI architecture dispenses with the current traditional view that emotional states are to be modeled as individual variables, and it integrates a new model of a continuous multidimensional affective space. The inherent nature of this unique representation addresses a number of enigmas associated with the traditional depiction. Namely, the significant programming effort and complex relational algorithms that existed between emotional states for updating their valences are no longer necessary. With the use of the affective space, this chapter has demonstrated the ability to scale up (increase) the emotional states of an agent with considerable ease. Furthermore, emotional decay and emotion blending are dealt with in an efficient and effective manner. What the future holds for the field of affective computing is unclear. The complexities of human emotions may be too extreme to include them holistically within an artificial intelligence at this time. Only those segments of emotional behavior that are advantageous to the goals of an artificial being should be considered. Further categorization of emotional behaviors is necessary to identify domains in which particular aspects of emotions are of an advantage. As more and more effort is put into small projects that contribute to the understanding of emotion, the field of affective computing comes closer and closer to a model for general, emotional, artificial intelligence.

REFERENCES Baillie, P. (2002). An agent with a passion for decision making. In Proceedings of Agents in Simulation 2002. Passau, Germany: University of Passau. Baillie, P., Toleman, M. & Lukose, D. (2000). Emotional intelligence for intuitive agents. In Proceedings of AISAT 2000, Hobart, Australia (pp. 134-139) Hobart: University of Tasmania. Blumberg, B.M., Todd, P.M. & Maes, P. (1996). No bad dogs: Ethological lessons for learning in Hamsterdam. In P. Maes, M.J. Mataric, J.-A. Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

116


Meyer, J. Pollack, & S.W. Wilson (Eds.), From Animals to Animats 4: Proceedings of the Fourth International Conference on Simulation of Adaptive Behavior, (pp. 295-304) Cambridge, MA: MIT Press/ Bradford Books. El-Nasr, M.S. (1998). Modeling Emotion Dynamics in Intelligent Agents. Master of Science Dissertation, American University in Cairo. Goleman, D. (1995). Emotional Intelligence. New York: Bantam Books. Grand, S., Cliff, D. & Malhotra, A. (1997). Creatures: Artificial Life Autonomous Software Agents for Home Entertainment. In Proceedings of 1st International Conference on Autonomous Agents (pp. 22-9) New York: ACM Press. Jacobson, N. & Bender, W. (1996). Color as a determined communicator. IBM Systems Journal MIT, Cambridge, 35(3), 526-538. Juslin, P. & Madison, G. (1999). The role of timing patterns in recognition of emotional expression from musical performance. In Music Perception (Vol. 17, pp. 197-221) New Haven: The University of California Press. Mateas, M. (1997, June). An Oz-centric review of interactive drama in believable agents. (Technical Report CMU-CS-97-156) School of Computer Science, Carnegie Mellon University, Pittsburgh, PA. Meech, J. & Marsh, S. (2000). Social factors in e-commerce personalization. In Proceedings of the CHI 2000 Workshop Designing Interactive Systems for 1-to-1 E-commerce. The Hague, NRC 43664. Picard, R. (1997). Affective Computing. London: The MIT Press. Reilly, W.S.N. (1996). Believable Social and Emotional Agents. PhD Dissertation, Carnegie Mellon University. Rogers, P.A.M. (2000). Acupuncture and homeostasis of body and adaptive systems. In The Web Journal of Acupuncture, found at http:// users.med.auth.gr/karanik/english/hels/helsfram.html. Roseman, I.J., Jose, P.E. & Spindel, M.S. (1990). Appraisals of emotioneliciting events: Testing a theory of discrete emotions. Journal of Personality and Social Psychology, American Psychologists Association, Washington, 59(5), 899-915. Scherer, K.R. (1982). Emotion as process: Function, origin and regulation. Journal of Experimental Psychology, American Psychologists Association, Washington, 29, 497-510. Smith, C.A. & Ellsworth, P.C. (1985). Attitudes and social cognition. In Journal of Personality and Social Psychology, American Psychologists Association, Washington, 48(4), 813-838. Smith, C.A. & Lazarus, R.S. (1990). Emotion and adaption. In L.A. Pervin Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


117

(Ed.), Handbook of Personality: Theory and Research, (pp. 609-637) Guilford, NY. Stern, A. (1999). AI beyond computer games. In Proceedings of 1999 AAAI Spring Symposium, Artificial Intelligence and Computer Games, (pp. 77-80). Menlo Park, AAAI Press. Velasquez. (1999). From affect programs to higher cognitive emotions: an emotion-based control approach. In Proceedings of Workshop on Emotion-Based Agent Architectures, (pp. 10-15) Seattle, WA.


118 Smith & Bonacina

Chapter VI

Evolutionary Computation as a Paradigm for Engineering Emergent Behavior in Multi-Agent Systems Robert E. Smith The University of the West of England, UK Claudio Bonacina The University of the West of England, UK

ABSTRACT In the multi-agent system (MAS) context, the theories and practices of evolutionary computation (EC) have new implications, particularly with regard to engineering and shaping system behaviors. Thus, it is important that we consider the embodiment of EC in “real” agents, that is, agents that involve the real restrictions of time and space within MASs. In this chapter, we address these issues in three ways. First, we relate the foundations of EC theory to MAS and consider how general interactions


Evolutionary Computation as a Paradigm

119

among agents fit within this theory. Second, we introduce a platform independent agent system to assure that our EC methods work within the generic, but realistic, constraints of agents. Finally, we introduce an agent-based system of EC objects. Concluding sections discuss implications and future directions.

INTRODUCTION With the advance of computational power and communications speed, we now live in a computational world where a large number of software agents may be acting on behalf of even the most casual user: searching for music, comparing pension schemes, purchasing goods and services, identifying chat partners, etc. Moreover, these agents may be collaborating with those of other users, while spawning and managing agents of their own. In more formal settings, a business, academic, or government user may simultaneously employ many software agents to manage workflow, trade goods or information, collaboratively solve problems, etc. In the future, even relatively simple household appliances may play a role in this churning system of interacting, computational agents. The behavior of a multi-agent system (MAS) is a result of the repeated (usually asynchronous) action and interaction of the agents. Understanding how to engineer adaptation and self-organization is thus central to the application of agents in the computational world of the future. Desirable self-organization is observed in many biological, social, and physical systems. However, fostering these conditions in artificial systems proves to be difficult and offers the potential for undesirable behaviors to emerge. Thus, it is vital to be able to understand and shape emergent behaviors in agent-based systems. Current mathematical and empirical tools give only partial insight into emergent behavior in large, agent-based societies. Evolutionary Computation (EC) (Back et al., 1997) provides a paradigm for addressing this need. Moreover, EC techniques are inherently based on a distributed paradigm (natural evolution), making them particularly well suited for adaptation in agents. In the MAS context, EC theories and practices have new implications. Agents that interact according to these theories are no longer locked inside the laboratory conditions imposed by EC researchers and users. Thus, it is important that we consider the embodiment (in a sense similar to that in Brooks et al., 1998) of EC in “real” agents, that is, agents that involve the real restrictions of time and space within a MAS. We address this issue in two ways. First, we have developed a platform independent agent system, to assure that we work within the generic, but Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


realistic, constraints of agents. Second, we have developed an agent-based system of EC objects. The prime thrust of our research with these tools is to facilitate understanding of EC within agents and understanding of more general agent interactions in the light of EC theories. The following sections describe the foundations of EC theory and practice and relate these foundations to MAS, both in terms of applying EC to MAS and in terms of understanding general MASs as evolving systems, through EC theory. The platform independent agent system is presented, along with the generic software framework for EC in MAS. Final sections include implications and future directions.

BACKGROUND The key qualities exhibited by software agents are autonomy, reactivity, and proactivity (Franklin & Graesser, 1997; Wooldridge & Jennings, 1996). Moreover, agents have the possibility of mobility in complex network environments, putting software functions near the computational resources they require. Agents can also explicitly exploit the availability of distributed, parallel computation facilities. However, these qualities ultimately depend on the potential for agent adaptation. For instance, if an agent is to operate with true autonomy in a complex, dynamic environment, it may have to react to a spectrum of circumstances that cannot be foreseen by the agent’s designer. Autonomous agents may need to explore alternative reactive and proactive strategies, evaluate their performance online, and formulate new, innovative strategies without user intervention. Areas where agents could benefit from adaptation are addressed by active research in machine learning (e.g., classification of unforeseen inputs, strategy acquisition through reinforcement learning, etc.). However, many machine learning techniques are focused on centralized processing of databases to formulate models or strategies. In contrast, EC techniques are inherently based on a distributed paradigm (natural evolution), making them particularly well suited for online, ongoing adaptation in agents. Note that in the following description, we will intentionally avoid providing the typical description of EC, to show that the connection to agents emerges from less method-specific rationale. Also note that (while explicitly avoiding entering the endless debate on what constitutes an agent), we refer to a finegrained system of many agents and assume no particularly detailed “intelligence” or “cognitive ability” in agents. Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


121

Instead, we merely assume that MASs are filled with self-interested agents, which can change, adapt, and learn. Given this, the agents can be expected to interact in ever-changing ways that range from the competitive to the cooperative. One should ask what models of system behavior have considered systems comprised of such agents? Clearly, the theories that have developed in the field of evolutionary computation qualify. This section will try to concisely restate some of the basic theories of EC, using an agent perspective, to indicate a direction along which EC theories can have implications for general agents. Many of the developments follow those in Goldberg (1989). Although these theories have been much debated in the EC literature (Radcliffe, 1997), we believe their conceptual conclusions provide a valuable perspective on self-organizing evolvable systems of general agents. Let us consider a set of agents, each of which selfishly seeks to exploit some limited set of resources. Imagine that each agent’s behavior is dictated by a set of (discrete) features, but assume an agent has no particular intelligence about how those features relate to its own success. Given this situation, a method of biased perturbations seems an obvious avenue for progress (evolution) for the individual agent. However, this model fails to consider how agents may benefit from one another. Let us assume that agents can exchange information in the form of messages (possibly in a market of information). How might a selfish agent exploit such information? Clearly, a viable strategy is for an agent to attempt to reason about the effectiveness and features of other agents and use any information obtained to bias its perturbation strategy toward more effective regions of the feature space. Given this outline of an agent’s localized perspective, let us consider the resulting global effects. Using notation similar to that in Goldberg (1989) let the expected proportion of existing agents containing some subset of features H at some time t be p(H,t). The expected number at time t + 1 is given by: p( H , t + 1) = Ps ( H ) 1 − Pd ( H ) where Ps(H) is the probability of any individual agent selecting feature H, and Pd(H) is the probability of feature being disrupted by the selection of other features or other effects of agents perturbing their feature sets. Note that each of these probabilities may be a function of the proportions of features. This simple expression makes no assumptions that can be said to be “genetic.” Our first assumption that vaguely relates to biological analogy is to cast the formula above in terms of a reproductive plan (Holland, 1975). That is, we will assume that the probability of selecting a feature is proportional (or Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


just directly related) to the proportion of agents that contain that feature. In other words, the more agents that persist in having a feature, the greater the likelihood that agents will adopt or retain that feature. In its simplest (proportional) form, this gives: Ps ( H ) = p ( H , t ) R ( H ) where R(H) is a reproductive rate related to the feature’s perceived utility across the population of agents. Note that this simple “proportional selection” form is often used in EC, but any increasing function of proportion would yield conceptual conclusions similar to those presented here. Substituting yields: p( H , t + 1) = p ( H , t ) R ( H ) 1 − Pd ( H ) which is a proportional form of Holland’s schema theorem (Holland, 1975). This formula does not depend explicitly on the form of the internal workings of the agents (i.e., the method of encoding features, or the operators within the agents). It only depends on the assumption of a reproductive plan. Why a reproductive plan? This “bandwagoning” onto apparently useful features in other agents is certainly not the only rational approach from an agent perspective. Agents may find it useful to run counter to what other agents do. However, a reproductive plan is certainly one reasonable strategy, and worthy of examination. Moreover, other plans not explicitly reproductive in character, but that use perceived utility of features in other agents to bias feature selection, may yield similar mathematical forms. Assume that the agent’s reasoning about desirable features is generally correct for some desirable feature H, and that R(H)[1-Pd(H)] remains greater than one for that feature. Ignoring constraints on proportions, this dictates an exponential increase in the p(H,t) with respect to t. Is this form of increase desirable? Holland’s k-armed bandit argument (Holland, 1975) shows that, regardless of the distributions of utilities of H and competing (mutually exclusive) features, a near-optimal rate of increase should be of exponential form with respect to time. A reproductive plan, like that stated above, yields this exponential form for certain features. This is an emergent effect at the system level, which only involves interactions at the agent level. The features that show this near-optimal, exponential effect are those with low rates of disruption,



123

Pd(H), relative to the magnitude of R(H). In EC, such features are often referred to as building blocks. At this point, note that the previous discussion is not inherently genetic and could as easily apply to memes as genes (Blackmore & Dawkins, 1999; Dawkins, 1990). Memes are replicators and are defined as units of cultural information, such as cultural practices or ideas, which are transmitted via communication and imitation between one entity and another. Clearly, memes are subjected to a reproductive plan, in the sense of Holland (1975). The primary difference between genes and memes is that we have an understanding of the underlying encoding of genes, but we have no such understanding (in general) of memes. Otherwise, the two entities behave in a conceptually identical fashion, that is, selfishly trying to maximize their reproductive success. The remaining technical discussion in this section concentrates on the assumption of some atomic encoding unit of an agent’s features. Although we may not understand what this unit is for agent memes, much of the reasoning will still ultimately hold. Moreover, the EC offshoot field of memetic algorithms (Krasnogor & Smith, 2000) can provide some insight in light of the perspective presented here. Because all interactions between agents could be categorized as potential transmissions of genes or memes, we believe that the EC-based perspective here can provide insight into general interactions between agents that may not be specifically genetic in character. All building blocks are treated in the emergent, yet near-optimal fashion indicated above, under a reproductive plan. Therefore, we should consider how many of these building blocks exist in a population of individuals. However, to maintain a general agent focus, we will do this without specific reference to EC details (e.g., genetic encoding). We only assume that there is a set of (discrete) atomic features, from which all other features are constructed. These atomic features are (roughly) analogous to genes in biological systems (or memes), but we are not assuming any particular, underlying encoding. With these assumptions, and the assumption of a population size that insures moderate sized building blocks have at least one copy in the population (Smith et al., 2000), one can show that the overall number of building blocks (Nbb) in the population has the following lower bound:  1  2 log(1/ p) 2 +1 N bb ≥   N   2



For binary atomic features, where p = 0.5, this gives a form of the N 3 lower bound often associated with genetic algorithms (Holland, 1975). However, regardless of the assumed form of atomic features, the general estimate shows that (under certain restrictive assumptions) a large number of building blocks are implicitly treated in the near optimal fashion indicated by the k-armed bandit argument, as an emergent phenomenon of reproductive plans. This key emergent effect of reproductive plans is referred to in EC as implicit parallelism. The discussion above only alludes to the simplest aspects of evolutionary models. More modern theories consider how features come into balance with respect to limited resources consumed, as another emergent consequence of selfish agents under reproductive plans. Some EC models have explicitly exploited this emergent “resource balancing” effect, as in multiobjective optimization problems (Deb & Goldberg, 1989). Such effects have also been observed as emergent artifacts of EC system dynamics (Horn, Goldberg, & Deb, 1994). EC models seem an appropriate way to gain insight on the propagation of features through a system of agents, and possibly to shape that propagation. Moreover, EC processes implicitly exploit parallelism, while remaining trivial to explicitly parallelize (as in an autonomous agent context). This is a strong argument for the advantages of MASs in general. However, the effects of EC in laboratory environments are not necessarily the same as the effects implied above for generalized agent systems. Thus, we must begin considering the EC analogy within these standards and within real agents based on these standards.

EC IN MAS A clear example of a system of agents, with evolutionary computation as an emergent effect, is Holland’s ECHO system (Holland, 1992). Another notable example is Tierra (Ray, 1990), which involves the coevolution of pieces of computer code. Other schemes suggest the idea of evolving agents (Adami, 1998; Balabanovic, 1997; Menczer & Monge, 1999; Moukas, 1996). However, these systems are not designed within general software agent frameworks. With the emergence of standards-based frameworks for general agents, it is becoming possible to consider EC as a form of interaction between such general agents. We suggest that by examining EC-specific interactions in such frameworks, one can gain insight into general agent interactions in the



125

appropriately embodied context. The following sections introduce a framework for examining embodied EC effects in general, standards-based agents.

A PLATFORM-INDEPENDENT AGENT SYSTEM To provide the appropriate context for general agent interactions, we first introduce our platform independent agent system, PIA. Because it is our desire to better understand EC when it is thoroughly embodied in real agents, and to also better understand general agent interactions through an EC perspective, we feel it is necessary to use a system of “real” agents, but one that is independent of the specifics of any given agent development platform. Different MAS platforms provide a large number of services. Different research communities are interested in different subsets of these services. We focused our attention on the basic characteristics that MAS should have, and with PIA, we provide a simpler interface with respect to the services provided by available platforms. Moreover, the PIA framework provides an easy way of developing and engineering research-grade applications using a prototyping approach and facilitating the applications migration from one platform to another. In order to achieve the platform independence, we modeled the PIA architecture using some appropriate and well-known design patterns (Gamma, 1995). We define PIA services that address basic characteristics of MAS in order to be able to mount them on the largest number of possible platforms.

Basic MAS Characteristics in PIA • •

•

•

In our modeling process, we identified the following actors: The Agent should be able to register itself in the system, obtaining a unique agent id, which is usable in a distributed environment; deregister itself from the system; send messages; and receive messages. The Message should provide these services: getting the sender and receivers IDs; getting the message’s send and receive times; setting the message’s receive time; getting the message’s type; and getting the message’s body. The Agent Factory should allow one to create an agent or destroy an agent. The agent factory is a concept present in all the platforms available to develop MAS. For example, in Aglets (the free development agent platform provided by IBM), it is possible to find several analogies between the Aglet Context and our agent factory concept. The Directory Service offers White and Yellow Pages and provides an interface for registering and deregistering agents to and from categories;



getting all agents registered in a category; getting all agents present in the system; getting all categories present in the system; testing if an agent and a category are present in the system; and testing if an agent belongs to a category. Obviously, it will often be necessary to define some rules that restrict agent creation and destruction. Such rules are application dependent and are grouped into PIA’s rule service. The Rule Service contains all the interaction and permission rules needed in the framework and in the applications using it. The generic framework contains just the following basic rules: permission rule to deregister an agent and permission rule to destroy an agent. The interface provided by the rule service is highly application dependent and will be specialized in particular applications.

•

Figure 1: Bridge and abstract factory patterns Bridged

Implementor

SetImplementor() GetImplementor() Imp

Agent Factory Abstraction

Uses >

Operation ()

Agent Factory Implementor OperationImp ()

Imp->OperationImp()

Refined Agent Factory Concrete Agent Imp A OperationImp ()

Concrete Agent Imp B OperationImp () Uses >

Obtain >

Uses >

Developing Platform A

Client Application

Uses >

Developing Platform B

Abstract Factory CreateAgentFactory ()

Concrete Factory A CreateAgentFactory()

Concrete Factory B CreateAgentFactory()



•

127

The Message Delivery Service provides agents with an asynchronous message manager. It can be compared with an Internet Mail Provider that manages messages regardless of whether the user is connected or not. The services are forwarding a message and taking in a message.

PIA does not deal with the features of the low levels of MAS architecture (e.g., the transport layers or scheduling processes). These functionalities are necessary; hence, PIA exploits the low-level services provided by an underlying platform, through a series of design patterns that connect the basic PIA actors to the platform. These are described in the following section.

Patterns in the PIA Framework We modeled the PIA architecture using the well-known Bridge Pattern (Gamma, 1995), which permits us to supply implementation independent services. All the bridged classes implement the interface Bridged. In Figure 1, we give an example of the agent factory service as a bridged class. PIA applies the same pattern to the concepts of agent, message, and directory services. Not all the services provided by the framework are implementation dependent, hence, not all of them have been bridged (e.g., the rule service is not bridged). The representation of the Abstract Factory Pattern appears at the bottom part of Figure 1. This is the only component of PIA with which the client application directly interacts. There must be a concrete factory class for each different platform PIA is able to exploit. To add a supported platform to PIA, the user adds a new subclass of AbstractFactory, implementing all the services it provides. AbstractFactory supplies a class method for the instantiation of the appropriate concrete factory. The client application calls this method (indicating which developing platform the user has decided to exploit) and retrieves the reference to the corresponding object. The AbstractFactory class supplies all the necessary methods to access all the framework services (e.g., like the agent factory’s services, directory services, and the creation of messages). A concrete factory provides a platform-related implementation of these services. The kind of agent to instantiate is defined by passing the class of the agent as a parameter to the agent creation method of the agent factory. Indeed, because the user can specialize the agent abstraction, PIA provides the possibility of creating an agent as an instance of any subclass of the Agent class. For instance, here is an example of a method call to create a HelloAgent instance: Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


HelloAgent anAgent = (HelloAgent)theFactory.getAgentFactory(). createAgent(Class.forName(“HelloAgent”)); There is a third pattern widely used in PIA: the Visitor Pattern; it represents an operation to be performed on the elements of an object structure. Visitors let one define new operations without changing the classes of the elements on which it operates. Subclassing and Visitor Pattern are the two main ways to increase the framework services with application-dependent features.

EC WITHIN A FRAMEWORK OF GENERAL AGENTS To implement an EC system, a programmer typically writes computer code that creates a population of software entities and manipulates those entities in a prespecified way. This is clearly unlike the intended design principles of MASs for the future. Figure 2: Structure of a typical, centralized genetic algorithm (GA), a basic form of EC Centralized GA program population

Evaluation/Interaction w/ external applications

Selection

Recombination

Mutation

Individuals are simple genetic encodings of parameters



129

Figure 3: Structure of the EC embodied within a general MAS General Agent Environment

IndividualAgent Agent’s evaluation of other agents

Messages Messages flow to and from the agent, other agents, and the “outside world”

Agent’s Recombination Strategy

Agent’s Mutation Strategy

“Child” Agent

In the authors’ view of an MAS/EC system, one must turn the typical EC software design on its head. In common EC software, a centralized program stores individuals as data structures and imposes the genetic operations that generate successive populations (Figure 2). In natural genetics, the equivalent of the centralized GA does not exist. Instead, the evolutionary effects emerge from individual agents. This makes designing an agent-based EC system a matter of straightforward analogy (see Figure 3). Note that there are two perspectives one can take on this figure. One is to see it as a literal design of EC objects and methods for a general agent environment. Another is as a viewpoint for examining general interactions of agents. By considering the former, one may be able to gain insight into the latter. However, to consider the possible complexities of general-purpose agents, one must embody those agents in a realistic context. We now introduce a framework like that pictured in Figure 3 (formerly called Egglets in Smith & Taylor, 1998, but now called EvoAgents). The EvoAgent framework is based on PIA. By using PIA, the EvoAgent framework can be used upon any MAS development platform. It is important to be clear about the purposes of this framework. It will serve as a platform for examining and exploiting explicit EC interactions of Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


autonomous, standards-based agents. However, it is hoped these examinations will allow for broader understanding of an EC model of general agent interactions, where EC data structures and operations may not be explicit. The most fundamental aspect of EvoAgents is the addition of EC packages. Some of them address basic EC features like, for instance, two interfaces that can be implemented for general objects. These are Randomizable and Mutatable. Simply stated, the system can be assured that any object that implements Randomizable has a randomize method, which generates a random instance of the object. The system can be assured that any object that implements the Mutatable interface has a mutate method, which randomly perturbs the object in some fashion. The two interfaces allow an agent to pursue a strategy of biased perturbation of data objects that implement these interfaces. To account for exchange of pseudo-genetic feature sets, there is an interface Sperm (which described genetic feature sets sent as messages between agents). To account for biased combination of Sperm with an agent’s own features, there is an interface Egg. These interfaces specify the behaviors suggested by their names but are also arbitrary data objects that can be structured in any fashion. However, note that the individual agent who creates an object that implements these interfaces determines the specific details of how an Egg operates, and what sort of Sperm data organization is understandable, and processable. An Egg need not simply recombine two feature sets, it may process arbitrary numbers of feature sets in arbitrary ways. Another key element of the framework is a Plumage interface, which agents use to advertise themselves for “mating.” Note that how an agent evaluates and selects mates (through evaluation of Plumage objects) is entirely an internal function of that agent. Moreover, the model allows for different agents to have entirely different mating and recombination strategies. A complete hierarchy of EC agent classes and interfaces has been developed, and part of it is shown in Figure 4. A BehavioralAgent is able to run a specific strategy. A GAgent can be genetically initialized, because it implements the GeneticInitAgent interface. The behavior of a GAgent may be driven by its genetic material, but the agent cannot mate. In order to reproduce itself, an agent must extend the class Bagent that implements the interface BreederAgent. This part of the class hierarchy defines all the services necessary to the agent for breeding. Note that the interfaces in this system could be applied to many different types of data objects, ranging from the simple strings of bits typically used in GAs, to tree structures, or more complex data objects. Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


131

Figure 4: The complete class and interface hierarchy to develop ECrelated agents Phenotype

Agent

GeneticInitAgent

BehaviouralAgent

BreederAgent

GAgent

BAgent

Note that in the embodied agent context, mating is not actively controlled from outside the agent. Instead, each agent undertakes its own internal mating strategy. The dynamics of this asynchronous (and possibly diverse) activity is an important aspect of embodied agents that deserves careful consideration.

RESULTS Results with the EC-based agents discussed above have already shown behaviors similar to that of conventional, centralized EC systems, while also showing that there are interesting, distributed effects of agent-based mater selection that deserve careful, analytical and empirical investigation (Smith et al., 2000). Also, results in a system of trading agents have shown the emergence of a complex interplay of agents, whose evaluations of genetic features are a dynamic function of their population’s composition (see Figure 5). For the sake of brevity, details of this simulation are offered, but each of the prices shown in this graph is controlled by the genetically determined production behavior of simple agents in an economy with the prices of three “goods” in the economy Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


Figure 5: Prices of three intermediate goods as a function of trading day in an evolving economy of producers and consumers (Smith et al., 2000) 18 16 14 Prices

12 10 8 6 4 2 2832

2683

2534

2385

2236

2087

1938

1789

1640

1491

1342

1193

895

1044

746

597

448

299

1

150

0

Trading Day

shown. The oscillations seen here are a by-product of ongoing coevolution of the agents. The agent’s tend to “chase” one another to profitable areas of their economy, deplete it, and then chase toward the new profitable area this depletion creates. In this sense, these complex behaviors can be seen as similar to those observed in the El Farol Bar Problem (). Our current work is extending these results into an EC/MAS-based workflow system. In this system, agents negotiate for work items (through a standardized bidding process), and, on a longer time scale, mate, and recombine through EC operations. The results will show the evolution of agents and the emergent effects on the workflow process.

FUTURE TRENDS • •

The key points we intend to emphasize are the following: General agent interactions, and related changes in agent behavior, can be viewed as EC-like interactions, whether or not explicit EC is being employed. This suggests that experience with “contained” EC theory and practice has important implications for the open, more universal, agent-based systems of the future.



•

133

However, to adequately consider these effects, we must begin to embody EC capabilities within “real” (standards-based) agents, such that realistic effects can be appropriately considered, utilized, and transferred to general understanding of EC-like agent behavior

The final point is key to future research in this area. When general agents exchange genetic/memetic-like information (or information that we wish to view as genetic/memetic-like), many agent-based systems issues must be directly considered. Specifically, market-based system issues are of importance (Kearney & Merlat, 1999), e.g.: • Perceived market value of the material exchanged and related issues of: - Trading protocols - Exchange contracts - Secondary information markets - Mechanisms for insuring trust In considering the extension of EC paradigms to general agent interactions, each of these facets will constitute important areas for future research.

CONCLUSION In conclusion, the EC theories need not be thought of as being limited to the specific class of contained algorithms we now know as evolutionary computation. With the possibility of a future where ubiquitous, semiautonomous agents interact in a vast computational sea, these theories have new meanings that require us to refocus consideration of EC within an agent context. With the emergence of fields of memetics and artificial societies (Gilbert & Conte, 1995), the importance of this connection between EC and agents is further emphasized.

ACKNOWLEDGMENTS Authors Smith and Bonacina gratefully acknowledge support provided through a research grant from British Telecom. The authors would also like to thank Lashon Booker and Rick Riolo for many useful comments that contributed to this paper.



REFERENCES Adami, C. (1998). Introduction to Artificial Life. Heidelberg: SpringerVerlag. Arthur, W.B. (1994). Inductive reasoning and bounded rationality (the El Farol problem). American Economic Review, 84, 406–411. Back, T., Fogel, D.B., & Michalewicz, Z. (1997). The Handbook of Evolutionary Computation. Oxford: Oxford University Press. Balabanovic, M. (1997). An adaptive web page recommendation service. In Proceedings of The First International Conference on Autonomous Agents. Blackmore, S.J. & Dawkins, R. (1999). The Meme Machine. Oxford: Oxford University Press. Brooks, R.A., Breazeal, C. (Ferrell), Irie, R., Kemp, C., Marjanovic, M., Scassellat, B., & Williamson, M. (1998). Alternate essences of intelligence. Proceedings of AAAI-98 Conference. AAAI Press. Dawkins, R. (1990). The Selfish Gene. Oxford University Press. Deb, K. & Goldberg, D.E. (1989). An investigation of niche and species formation in genetic function optimization. Proceedings of the Third International Conference on Genetic Algorithms (42–50). Eymann, T., Padovan, B., & Schoder, D. (1998). Simulating value chain coordination with artificial life agents. In Demazeau, Y. (Ed.), Proceedings of ICMAS’98, (423–424). IEEE Computer Society Press: Los Alamitos. FIPA Specifications. http://www.fipa.org/spec/fipa99/fipa99Kawasaki.htm. Franklin & Graesser. (1997). Is it an agent, or just a program?: A taxonomy for autonomous agents. In Proceedings of the Third International Workshop on Agent Theories, Architectures, and Languages (21– 35). Heidelberg: Springer-Verlag. Gamma, E., Helm, R., Johnson, R., & Vlissides, J. (1995). Design Patterns. Reading, MA: Addison-Wesley. Gilbert, G.N. & Conte, R. (Eds.). (1995). Artificial Societies: The Computer Simulation of Social Life. London: UCL Press. Goldberg, D.E. (1989). Genetic Algorithms in Search Optimization, and Machine Learning. Reading, MA: Addison-Wesley. Grand, M. (1998). Patterns in Java, Volume 1. New York: Wiley. Holland, J.H. (1975). Adaptation in Natural and Artificial Systems. Ann Arbor, MI: The University of Michigan Press. Holland, J.H. (1992). Adaptation in Natural and Artificial Systems (2nd ed.). Cambridge, MA: MIT Press. Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


135

Horn, J., Goldberg, D.E., & Deb, K. (1994). Implicit niching in a learning classifier system: Nature’s way. Evolutionary Computation, 2 (1): 37– 66. IBM Aglets. http://www.trl.ibm.co.jp/aglets/index.html. Kapsalis, A., Smith, G.D., & Rayward-Smith, V.J. (1994). A unified paradigm for parallel genetic algorithms. In Fogarty, T. (Ed.), Evolutionary Computing AISB Workshop. Heidelberg: Springer-Verlag, pp. 131– 149. Kearney, P. & Merlat, W. (1999). Modeling market-based decentralised management systems. BT Technology Journal, 17 October (4). Kearney, P., Smith, R., Bonacina, C., & Eymann, T. (2000). Integration of computational models inspired by economics and genetics. BT Technology Journal. 18 (4): 150–161. Krasnogor, N. & Smith, J. (2000) MAFRA: A Java memetic algorithms framework, Workshop on Memetic Algorithms, GECCO. Menczer, F. & Monge, A.E. (1999) Scalable Web search by adaptive online agents: an InfoSpiders case study. In M. Klusch (Ed.), Intelligent Information Agents. Heildelberg: Springer-Verlag. Moukas, A. (1996). Amalthaea: information discovery and filtering using a multiagent evolving ecosystem. In The Proceedings of the Conference on Practical Applications of Agents and Multiagent Technology, London. Parunak, H.V.D., Savit, R., & Riolo, R. (1998). Agent-based modeling vs. equation-based modeling: a case study and users’ guide. In Proceedings of Workshop on Multiagent Systems and Agent-Based Simulation (MABS’98), (10–25). Heidelberg: Springer-Verlag. Radcliffe, N.J. (1997). Schema processing. In Back, T., Fogel, D.B., & Michalewicz, Z. (Eds.). The Handbook of Evolutionary Computation. Oxford: Oxford University Press, pp. B2.5:1–10. Ray, T. (1990). An approach to the synthesis of life. In Langton, C., Taylor, C., Farmer, J.D., & Rasmussen, S. (Eds.). Artificial Life II. Reading, MA: Addison-Wesley, pp. 371–408. Rosin, C.D. & Belew, R.K. (1997). New methods in competitive coevolution. Evolutionary Computation. 5(1): 1–29. Smith, R.E. & Taylor, N. (1998). A framework for evolutionary computation in agent-based systems. In C. Looney & J. Castaing (Eds.) Proceedings of the 1998 International Conference on Intelligent Systems, (221– 224). ISCA Press.



Smith, R.E., Bonacina, C., Kearney, P., & Merlat, W. (2000). Embodiment of evolutionary computation in general agents. Evolutionary Computation. 8 (4): 475–493. Wooldridge & Jennings. (1996). Software agents. IEE Review. January, 17– 20.


The Logic Behind Negotiation

137

Chapter VII

The Logic Behind Negotiation: From PreArgument Reasoning to Argument-Based Negotiation Luís Brito Universidade do Minho, Portugal Paulo Novais Universidade do Minho, Portugal José Neves Universidade do Minho, Portugal

ABSTRACT The use of agents in Electronic Commerce environments leads to the necessity to introduce some formal analysis and definitions. A four-step method is introduced for developing EC-directed agents, which are able to take into account nonlinearites such as gratitude and agreement. Negotiations that take into account a multistep exchange of arguments provide extra information, at each step, for the intervening agents, enabling them to react accordingly. This argument-based negotiation among Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

138 Brito, Novais & Neves

agents has much to gain from the use of Extended Logic Programming mechanisms. Incomplete information is common in EC scenarios; therefore, arguments must also take into account the presence of statements with an unknown valuation.

INTRODUCTION The amount of ambiguity present in real-life negotiations is intolerable for automatic reasoning systems. Concepts present in each intervening party of a real-life negotiation need to be objectively formalized in order for an automatic approach to be reached. Logic, and especially, Extend Logic Programming (ELP) (Baral & Gelfond, 1994) poses itself as a powerful tool to achieve the desired formality without compromising comprehension and readability, and the ability to easily build an executable prototype for agents. Logical formulas are extremely powerful, unambiguous, and possess a set of interesting advantages (McCarthy, 1959): Expressing information in declarative sentences is far more modular than expressing it in segments of computer programs or in tables. Sentences can be true in a much wider context than specific programs can be used. The supplier of a fact does not have to understand much about how the receiver functions or how or whether the receiver will use it. The same fact can be used for many purposes, because the logical consequences of collections of facts can be available. However, in a dynamic environment such as the one found in electronic commerce (EC), the simple use of logical formulas is not enough. The use of nonmonotonic characteristics is self-evident (which is in some way found in ELP) (Neves, 1984). In general logic programs, negative information is provided by the closedworld assumption (i.e., everything that cannot be proven to be true is false), however, in extended logic programs, that is not so. In ELP, a query may fail due to the fact that information is not available to support it or, on the other hand, it may fail due to the fact that negation succeeds. The knowledge base (KB), which serves as the basis for the agent’s reasoning, can be seen has an extended logic program (P) that is a collection of rules with the form: L0 ← L1, ..., Lm, not Lm+1, ..., not Ln



139

where Li (0 ≤ i ≤ n) is a literal (i.e., formulas of the form p or ¬ p, where p is an atom). This general form is reduced to L0← (also represented as L0) in the case of facts. The strategy to get a consistent and sound approach for the use of agents in EC is based on Novais et al. (2001) and is composed of a four-step development methodology: • Architecture definition: Define and specify the agent’s modules or functionalities, design the flow of information [e.g., Experience-Based Mediator (EBM) agent (Novais et al., 2000), mobile agents for virtual enterprises (Brito et al., 2000b)] • Process quantification: Quantify each metric and subprocess that the agents may have to deal with; establish the mechanisms and protocols for an efficient approach to a wide range of problems (Brito & Neves, 2000; Brito et al., 2000a) • Reasoning mechanism: Each agent needs a formal (logical) set of rules that will serve as the main guidelines for the negotiation processes; the agent needs to reason about the surrounding world before it acts through argumentation (Brito et al., 2001a) • Process formalization: The process of (logical) argumentation needs to proceed via a formal specification to a consistent implementation in order to set the agents to act or react in a reasonable (logical) way; arguing during an EC negotiation has many similarities to legal arguing (Prakken, 1993; Sartor, 1994) and logic presents itself, once again, as a powerful specification and implementation tool This methodology stands as a particular case of the use of formal methods in Agent-Oriented Software Engineering (AOSE) (Wooldrige & Ciancarini, 2001). This chapter is disposed accordingly to the proposed four-step approach to the development of agent for EC. In the section Architecture Development, architectures for EC are presented. In the section on Process Quantification, examples of objective process quantifications are exposed. In the Reasoning Formalization section, the basic building blocks of the negotiation process (tools for preargument), such as theorem solvers, restrictions, and null values are introduced, aiming at a proper formalization, and the process of reasoning with incomplete information is extended to include temporality and priorities, giving way to the formalization of concepts such as delegation, gratitude, and agreement. In the section on Process Formalization, the process of argumentation is formalized. Finally, some conclusions are drawn, and future work is proposed. Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


The main contributions of this work are the definition of a common ground to situate the agent’s reasoning mechanisms in EC environments; the use of formal tools (logic) to describe the rational behavior of agents involved in EC; the description of a reasoning mechanism necessary for a consistent and sound development of agents for EC; the use of incomplete information in the reasoning process; the bridging of legal argumentation and argument-based negotiation; and the establishment of sound syntactic and semantic tools for argument-based negotiation.

ARCHITECTURE DEVELOPMENT The development of agent architectures for EC needs to take into account the particular reasoning characteristics to be addressed. The EBM agent (Novais et al., 2000) provides a logic-based framework (with a well-defined set of modules) for preargument reasoning and argument generation. However, this architecture is, in no way, the final solution. Agents oriented to price manipulation (and other econometric approaches) represent an interesting solution (although with limited reasoning capabilities).

The EBM (Experience-Based Mediator) Agent The EBM agent is a general module-oriented architecture aiming at the development of intelligent agents for EC. Taking previous experience as a starting point, the agent’s knowledge is complemented by general and introspective knowledge, the former comprises information about the system and the prices and rules practiced by counterpart agents, and the last embraces psychological values such as beliefs, desires, intentions, and obligations. Dynamic and static knowledge are therefore embedded at this level. An agent must be able to reason about general or even incomplete information, on the one hand, and it must also be able to explain its own behavior or acquire new knowledge, on the other hand. But, in the present context, these procedures are not enough. The ability to deal with the market’s specificities is paramount [e.g., the ability to form prices, to evaluate a good or service, or to cartelise (Brito & Neves, 2000; Brito et al., 2000a)].

Other Approaches This is a functional approach to agent architecture in EC. Agents in EC were primarily seen as information gatherers (price gatherers) and priceadapters (through mathematical or functional techniques). The use of agents in



141

EC scenarios has also been approached through theories of economic implication, i.e., economic models and theories condition the behavior of an agent. As the transition toward the information economy is taking place, in Kephart et al. (2000) two kinds of agents were proposed to enable this same transition: the pricebots and the shopbots. Shopbots (also called comparison shopping agents) are the answer to intelligent price comparison of online providers. Pricebots, on the other hand, are the provider’s counterpart to shopbots, i.e., they manipulate prices, taking into account the market conditions. These two kinds of agents are a step toward so-called “frictionless” commerce.

PROCESS QUANTIFICATION Through the mass media, EC has been, to the eyes of the public, indisputably reduced to a Business-to-Consumer (B2C) perspective; furthermore, this short-sighted vision was reduced even more to the publicized and catalog sales (Guttman et al., 1998). In spite of this, the Business-to-Business (B2B) perspective is also endorsed by EC, although the lack of well-established standards and reluctance on the managerial side has hindered its success. EC can be seen under two perspectives: Virtual Marketplaces (VMs) and Virtual Organizations. VMs fall into the popular view of the subject, i.e., buying or selling in auction or nonconcurrent dealings. VOs are traditionally seen as the network of commercial interests established among different businesses, in order to provide some sort of good or service. The VOs view can be extended beyond the common definition and asserts that a network of interests can be established within an organization, i.e., functional units or work areas can be found in enterprises, giving way to a network of internal interests driven by the ultimate goal of providing, with maximum quality and minimum costs and delivery time, the contracted goods or services. The simple VM that spawns from a company that tries to sell its products on the Internet may be seen as an atomic element of a wider VO. This recursive construction is made possible by the fact that agents, being similar to their realworld counterparts, should play a mediator role, i.e., an agent is either a buyer or a seller, depending upon the pending circumstances. The definition of a business strategy is of paramount importance for the future success of any company. Planning must rely on a series of tools that enables the elaboration of a short-, medium-, or long-term strategy. In the real world, the main tools are mediation, agreement, and gratitude. Mediation enables a company to play a dual part on the market, i.e., the experiences Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


gathered as buyer may be used to extrapolate future actions as seller, and vice versa. Agreement enables a feeling of trust in an agent, either on truthful or untruthful voting scenarios (Brito & Neves, 2000). Gratitude is important for the creation of interorganizational dependencies (debts) that condition future deals (Brito et al., 2000a). Typical approaches to EC are based on the assumption of one-to-one negotiation without any spurious influences on thirdparty entities, i.e., negotiations are conducted in a one-provider to onecustomer way, such that there is an absence of dialogue among providers and, therefore, all negotiations are statistically independent.

Gratitude One can establish gratitude as a tacit obligation that influences the decision-making process (Brito et al., 2000a) (e.g., in the real world, an agent may be forced to pull out from a negotiation if so requested by someone to whom it owes some value). Gratitude may arise from one of two main situations: a gift of some sort is given to someone (e.g., a Christmas present) — nonnegotiable gratitude — or, during the process to set up a settlement or agreement, an agent offers some form of compensation for the pull out of a competitor (e.g., monetary compensation for an unsure transaction) — negotiable gratitude (Brito et al., 2000a). The significance of this concept on VOs spawns from the fact that agents are now able to influence future decisions on the part of the other companies’ counterparts, i.e., the debt of gratitude is influenced by personal standards and does not configure itself as a universal value. Gratitude may be measured marginally in the form: valueoffer − Wstrategy ( NI , y ), NI .grt = non − negotiable  gm ( x, y, NI )(1 − α ) Fgains ( NI ), NI .grt = negotiable 0, otherwise  where gm(x, y, NI), Wstrategy(NI, y), valueoffer, a and Fgains(NI) stand, respectively, for the marginal gratitude of agent x toward agent y, taking into account the negotiable information NI; the function that weights the influence of NI and agent y in the strategy of agent x; the value attributed by x to the offer that develops a gratitude debt; the percentage of the gains to offer to y as compensation for a drop-out; and the forecast of the gains taking into account NI. NI is a composite structure that includes fields such as the kind of gratitude (grt).



143

The overall agent’s debt toward the community is given by: G ( x) =

∑ g ( x, i,⋅) m

i∈ Agents \{ x}

where G(x) and Agents stand, respectively, for the aggregate gratitude debt of agent x and the set of agents that represent the community. This aggregate value must be thoroughly controlled, so that an agent x does not enter a situation where the debt of gratitude is greater than the gains expected to be obtained in future dealings.

Agreement Inside an organization, one may see the need for agreement, but when using collaborative agents, one is faced with a truthful environment (Brito & Neves, 2000). In these environments, one is able to formalize agreements in order to provide an integrated picture of the system to the outside world. This image can be achieved by gathering all the agents’ opinions on a particular subject and by using a majority vote. The agreement strategy can, therefore, be rewritten in order to weight each agent’s specificity. This may be expressed as: agreementw(value)= majority[w1opinion1(value), w2opinion2(value), ..., wnopinionn(value)], value∈ρ. with 0 ≤ wi ≤ 1, ∀ i ∈ Agents where wi and Agents stand, respectively, for the weight of agent i in the making of value, and the community of agents. wi, on the other hand, may be a function of time, which can be expressed as wi(t) = β wi(t - 1) + (1 - β) compliance (vp,i(t),vr,i(t)) with 0 ≤ β ≤ 1 where β, vp,i(t), vr,i(t) and compliance(x,y) stand, respectively, for the weight of the historic information in the making of wi(t); the judgement of agent i at time t on the value; the value attained by agent i at time t; and a measure of the reciprocity or relation between vp,i(t) and vr,i(t). The higher the value of β, the higher the significance of historic values and the smaller the influence of sporadic noncompliances. However, on an open market, one cannot assume that the agents are always truthful, i.e., telling or expressing the truth.



For someone to reach an agreement in untruthful environments, a roundbased protocol is needed in order to isolate the pernicious influences of untruthful voting. These protocols often rely on the existence of a minimum number of intervening factors. Typically, one must have n ≥ 3m + 1, where n is the number of truthful agents, and m is the number of untruthful ones (Brito & Neves, 2000). This natural capability to generate agreements using round-based protocols in real-world environments makes a case to strategic planning in interbusiness relationships. One is able to form alliances among companies to forge better prices to specific products. On the other hand, it makes possible the gathering of information of vital importance for the definition of market strategies, determining the limits of the common ground on which the different enterprises stand.

Strategic Planning Every organization, in order to evolve in a sustained and sound way, must define strategic guidelines that will enable competitiveness and the definition of management goals. The concept of strategy is important in areas that range from the military to commercial organizations, either virtual or real, i.e., the necessity for feasible and clear planning is vital in every point of the production and consumption chain. On the other hand, the companies that position themselves closer to the consumer suffer from the impact of production defects and delivery delays, while being, at the same time, pressured by the consumers. Typical EC systems are unaware of these shortcomings, i.e., they function on a per-deal basis, which may render interesting profits on a deal but, in the longterm, may decrease the negotiable company rates. One may now formalize the difference between overall profit (spawning from strategy driven systems) and local profitability, in the form: Profit ( Agents) ≠

∑ ∑ profit (i, j )

i∈Agents j∈Deals

where Profit(Agents) and profit(i,j) stand, respectively, for the overall profit obtained by the set of agents Agents, and the marginal profit acquired in a perbusiness strategy executed by agent i for deal j. In order to define a sound strategy, one must be able to gather the counterparts and guarantee their standing on particular issues, i.e., agreement plays an important role in market placement. On the other hand, one must be



145

aware that the market evolves within truthful environments (where there is agreement among the parties that make an organization) or within untruthful ones (this is the case where typical real-world, self-interested entities may utter conflicting opinions). The definition of strategic lines must take into account punctual alliances rising from gratitude debts, which may be used in order to secure the expected behavior from counterparts.

REASONING FORMALIZATION Some of the most important features of preargument reasoning are temporality, priorities, delegation, gratitude, and agreement (already quantified in the previous section). An agent weights its knowledge base, its temporal validity, and its relative priorities, and then decides if delegation is in order. As for gratitude and agreement, reasoning takes into account the quantification provided at the previous stage of the present methodology. The general process of negotiation must be clearly distinguished from the argumentation stage (Brito et al., 2001b). The process of argumentation is tightly coupled with the process of logically founded attack on the arguments put forward by a counterpart. It deals with price-formation issues and deal finalization. On the other hand, negotiation is a wider concept that is coupled with specific forms of reasoning, dealing with the high-order, prearguing relationships that may be established among agents.

Right to Deal During a negotiation process, each agent, although being able to deal with a counterpart, may be inhibited from doing so. Therefore, a distinction must be established between capability (i.e., an agent has the necessary expertise to do something) and right (i.e., an agent has the capability to do something, and it can proceed that course of action) (Norman et al., 1998). In the case of an EBM agent, it is assumed that it has the ability to deal with every product, under any scenario. However, any EBM agent has its behavior conditioned by the right-to-deal premise. Consider predicates capability-todeal: Product, Conditions, Counterpart → {true, false} (representing the capability to deal), and right-to-deal: Product, Conditions, Counterpart → {true, false} (representing the right to deal), where Product, Conditions and Counterpart stand, respectively, for the product to be traded, the conditions associated with that operation, and the counterpart agent involved in the deal. It may now be stated that:



∀

Product

∀

Conditions

∀ Counterpart capability-to-deal

(Product, Conditions, Counterpart) i.e., the capability to deal is a tautology within EBM agents. Therefore, the presence of such knowledge in the KB of an agent can be taken as implicit. Therefore, the knowledge about the right to deal (right-todeal: Product, Conditions, Counterpart → {true, false}) rises in importance. A logical theory (on which the KB of each agent is based upon) is now possible to define: Definition 1 (A Logical Theory for Negotiation Agents) Alogical Theory for Negotiation Agents is defined as the quadruplet TNA = 〈R, C, BP, p 〉 where R, C, BP and p stand, respectively, for the set of predicates on the right to deal (right-to-deal: Product, Conditions, Counterpart → {true, false}), the set of invariants (A:+restriction::P), the set of behavioral predicates (including the theorem proffers) and a noncircular order relation that states that if P p Q, then P occurs prior to Q, i.e., having precedence over Q.

Using Incomplete Information Typically, commerce-oriented agents (such as the EBM one) act in situations where dealing with a given agent is forbidden or, in some way, the set of conditions to be followed in a deal is not completely defined. These situations involve the use of null values (Analide & Neves, 2000). A special theorem solver can be developed in order to cope with this kind of information. With the use of incomplete information with null values, a simple three-valued logic is set into place. Using this framework, it is now possible to assert the conditions under which a given product or service may be traded. The use of a null value from an unknown set of values (Baral & Gelfond, 1994) can state the ability to deal some product with some counterpart knowing only that the set of conditions that governs such deal belongs to an unknown set of values. For this case, the KB of an agent must contain clauses such as the following: exceptionrtd(P, -, CP)←

nullunknown-set(X), right-to-deal( P, X, CP).



147

¬ right-to-deal (P, C, CP)← not right-to-deal(P, C, CP), not exceptionrtd(P ,C, CP). The KB of an agent must contain an instantiation of nullunknown-set [e.g., nullunknown-set (cond)] and right-to-deal() clauses that may use the null value [e.g., right-to-deal(p4,cond,cp2)].

Temporality The concept of temporality is connected to the temporal validity of possible inferences over the KB of an agent; i.e., a fact may be valid only on a well-defined time period. Taking a nondestructive KB and a nonmonotonous logic, different conclusions may be reached when the temporal validity of information is taken into account (e.g., John has the right to deal with Paul but only from 10/05/2001 to 12/05/2001) (Neves, 1984). Taking set R (right-to-deal clauses) from logical theory TNA, an extension is to be made in order for these elements to encompass temporal validity. Therefore, an agent will reason about validity, taking into account the information present at the fact level. An example of validity, for a specific clause, is shown in Figure 1. Definition 2 (Clauses with Temporality) A factual clause, represented as P, where P is an atomic formula, is represented, in order to encompass temporal validity, as P::[i1,i2,...,in]., where ij=[ta, tb] is one of the following elements: 1. Temporal instant ta= tb, ta, tb ≥ 0 with ta, tb ∈ TD. TD = {t|t ∈ No} ∪ {forever}, where forever represents the end of times.

Figure 1: Example of time validity for a right-to-deal clause



2.

Temporal interval ta< tb, ta ≥ 0 with ta, tb ∈ TD. TD= {t|t ∈ No} ∪ {forever}, where forever represents the end of times.

In the case where positive and negative information is present in the KB, set R of theory TNA should be consistent, i.e., the following condition should be verified: ∃P :: T1 ∧ ∃P :: T2 → T1 ∩ T2 = ∅

Priorities In logic programming languages, such as Prolog, some priority is established through the ordering of clauses. However, this kind of priority is too weak, giving way to the definition of new priority rules with well-specified semantics. The necessity to establish priorities, within the set of clauses that composes an agent’s KB, arises either from computational reasons or from the necessity of establishing new semantics. The solution, for a feasible priority treatment, lies in the embedding of priority rules in the KB of each agent (Brito et al., 2001a, 2001b). Therefore, logical theory TNA is to be changed into a new logical theory (TNAP) in which the organization of factual clauses is given by the semantics of priority rules. Definition 3 (A Logical Theory for Negotiation Agents with Priorities) The logical Theory for Negotiation Agents with Priorities its defined as TNAP = 〈R, C, BP, p 〉 where, R, C, BP, PR, and p stand, respectively, for the set of predicates on the right to deal (right-todeal: Product,Conditions,Counterpart → {true, false, unknown}), the set of assertion restrictions/invariants (A:+restriction::P.), the set of behavioral predicates (including all demonstrators/theorem solvers), the set of embedded priority rules and the noncircular order relation established among the different clauses in a KB that derives from the time of their insertion. Relation p determines, in the case of P p Q, that P is earlier than Q, thus ordering the set of clauses, providing for a fail-safe priority mechanism under the one provided by the set PR.



149

Although priorities can be established between single clauses, it is usual, at least as a first-level approach, to consider priorities among bodies of knowledge (e.g., information about mary as priority over information about john). These bodies of knowledge are nothing more than a high-level classification of factual clauses (e.g., agy : bk1 : right-to-deal(p2,[c5], cp4):: [[0, 10]].). Notice, however, that this classification has variable granularity, giving way to a per-clause priority if so needed (with the consequent increase in complexity). The previous definitions on the use of incomplete information, temporal information, and priorities culminate in the creation of a theorem solver that enables reasoning preargumentative reasoning. Definition 4 (An LP Theorem Solver for Incomplete and Temporal Information with Priorities) Taking factual clauses with temporal validity and body of knowledge classification (represented by BK::P::[i1,i2,...,in].) and rule clauses (represented by P ← Q and being read as “P if Q”) as the components of the KB present in each agent, the predicate demoLPITP:T,CT,V → {true, false}, where T, CT, V and {true, false} stands, respectively, for a logical theorem, the current time, the theorem valuation (true, false, or unknown) and the possible valuations for the demoLPITP predicate, represents the LP theorem solver for incomplete and temporal information over the KB, governed by the following set of rules: demoLPITP(P, CT, true)← priority(BK1, BK2), testpriority(BK1, BK2, P,T), intime(CT, T). demoLPITP(P, CT, false)← priority(BK1, BK2), testpriority(BK1, BK2, P,T), ¬intime(CT, T). demoLPITP(P, CT, false)← priority(BK1, BK2), ntestpriority(BK1, BK2, P,T), intime(CT, T). demoLPITP(P, -, unknown)← priority(BK1, BK2), not testpriority(BK1, BK2, P,-), not ntestpriority(BK1, BK2, P,-). (BK1::P::T). testpriority(BK1, -, P,T) ← testpriority(-, BK2,, P,T) ← (BK2::P::T). Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


ntestpriority(BK1, -, P,T) ← ¬ (BK1::P::T). ntestpriority(-, BK2, P,T) ← ¬ (BK2::P::T). where predicates intime: CT, LT ®{true, false}, testpriority: BKa, BKb, P, T→ {true, false} and ntestpriority: BKa, BKb, P, T → {true, false} stand, respectively, for the verification of presence of time CT in the list of validity intervals LT, the prioritized demonstration of theorem P for the bodies of knowledge BKa and BKb and the prioritized demonstration of theorem P through negative information for the bodies of knowledge BKa and BKb.

Delegation Delegation can be seen as the delivery (assimilation) of a valid negotiation from one agent to another. Negotiation tasks may only be delivered to a third party if there is sufficient knowledge relating to the right to deal with that same agent. Delegation acts as a way to undertake indirect negotiations, i.e., use a proxy agent taking advantage of its particular characteristics, such as gratitude debts and agreements established among the proxy and the other agents (Brito et al., 2000a). Therefore, formalizing the delegation process is equivalent to formalizing the generation of a “middle-man” approach to business. A logical perspective is given by considering that the act of delegating deals involving product P, conditions C and counterpart CP to agent Y (considering time CT), is only possible if the delegating agent is able to deal the product with the final counterpart (valid for the present view over delegation); the delegating agent is able to deal with the proxy agent; and the proxy agent is able to deal with the final counterpart by itself. Formally: agx : delegate(P,C,CP,Y,CT)← agx : demoLPITP(right-to-deal(P,C,CP), CT, true), agx : demoLPITP(right-to-deal(P,-,Y), CT, true), Y: validassimilation(Y: right-to-deal(P,C,CP), CT).

Gratitude Although gratitude quantification is possible and even desirable in order to enable computational manipulation, it is still a subjective element and a nonlinearity that has decisive influence in the outcome of many business strategies (e.g., in strategic planning) (Brito et al., 2000a). A quantified characterization of the marginal gratitude concept is depicted in the section on Process Quantification. Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


151

When analyzing the two gratitude situations, it can be seen that the first (nonnegotiable gratitude) occurs with gratitude value (Value) when, taking into account the negotiation information (NI), the counterpart agent (Y), and the reasoning time (CT), a specific offer (e.g., gift) from the counterpart agent takes place. Dealing with that counterpart is authorized, and the “subjective” (although quantified) value of the offer is taken into account when updating the negotiation information through which a strategic evaluation (conditioning further action) is made. The next situation (negotiable gratitude) is viable when it is possible to deal with a specific counterpart agent, which in turn, states in its own KB that the agent is able to drop a specific negotiation (probably a competitive negotiation) in exchange for some compensation (reflected in terms of gratitude). Formally: agx:gratitude(Value,NI,Y,CT) ← agx:offer(Y,Description), agx:demoLPITP(right-to-deal(-,-,Y),CT,true), agx:evaluateoffer(Y,Description,CT,Valueoffer), agx:update(NI,Y,Description,Valueoffer,CT,NNI), agx:evaluatestrategy(NNI,Y,CT,S,Sweight), agx:gratitudemarginal(NNI,[Valueoffer,Sweight],Value), Value>0. agx:gratitude(Value,NI,Y,CT) ← agx:demoLPITP(right-to-deal(-,-,Y),CT,true), Y:dropcompensation(agx,NI,CT,α ), agx:forecastgains(Y,NI,CT,F), agx:gratitudemarginal(NI,[α ,F],Value), Value>0.

Agreement Like gratitude, agreement can be seen, in many cases, as a subjective element that introduces nonlinearities in the negotiation process (Brito & Neves, 2000; Brito et al., 2000a). The simplest case of agreement is reached through the use of a majority of votes. This democratic approach relies on the existence of fully veridic agents, i.e., agents that convey their opinion in a consistent manner to their peers. This majority approach is quantified in the section on Process Quantification. In logical terms, an agreement can only be reached among agents that are able to deal with each other, i.e., if an agent is unable to assert the right to deal with other agents, it can never establish some sort of commitment (agreement). Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


An agreement is reached on a specific subject (S), among a set of entities (E) with a set of opinions (O) at a specific time (CT). By definition, an agent is in agreement with itself in every subject. As for the other counterparts, an agreement with them is reached if the agent is authorized to deal with every one of them, their opinions are gathered, and, finally, a summary is produced, i.e., it is possible to establish an agreement situation weighing the set of opinions. Formally: agx:agreement(-,[agx],-,-). agx:agreement(S,E,O,CT) ← agx:can-deal-with(E,CT), agx:gatheropinions(S,E,CT,LO), agx:summarize(S,O,LO,CT). agx:can-deal-with([A],CT) ← agx:demoLPITP(right-do-deal(-,-,A),CT,true). agx:can-deal-with([A|T],CT) ← agx:demoLPITP(right-do-deal(-,-,A),CT,true), agx:can-deal-with(T,CT).

Example Assume the following KB, defined according to the noncircular theory TNAP. Reasoning about delegation will now involve the set of restrictions embedded into the KBs. The clauses are: agx : bk1 : right-to-deal(p1,[c1], cp2):: [[0, forever]]. agx : bk2 : right-to-deal(p2,[c3, c4], cp3):: [[0, 50]]. % exceptions agx % theorem proffers agx % priorities agx: priorities(bk1, bk2). agy : bk1 : right-to-deal(p2,[c3, c4], cp3):: [[0, 60]]. agy : bk1 : right-to-deal(p2,[c5], cp4):: [[0, 10]]. agy :money(900). % exceptions agy % theorem proffers agy % priorities agy: priority(bk1, bk2). Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


153

Agent agx is able to negotiate product p1, taking conditions [c1] with counterpart agent agy permanently. In the case of product p2, conditions [c3, c4] are established for counterpart agent cp3, but only for interval [0,50]. The knowledge about the right to deal with agy overpowers the knowledge about cp3. Agent agy is able to negotiate product p2, taking conditions [c3, c4] with counterpart agent cp3, but only on interval [0,60]. Furthermore, it is able to negotiate product p2, taking conditions [c5] with counterpart agent cp4, but only on interval [0,10]. Agent agy has 900 monetary units expressed in its KB, and a new assertion is conditioned to the existence of 1000 monetary units (due to an assertion restriction). Priority rules establish that the knowledge about cp3 overpowers that of cp4. The KB can be queried in order to determine the validity of a delegation process: ?agx: delegate(p1,[c1],cp2, agy,10). false ? agx: delegate(P,C,cp2, agy, 10).

P={p1},C={[c1]}

The second column expresses possible variable valuations or the valuation for the query. In the first query, although the right to deal is well established in agent agx, it is impossible to assert the necessary knowledge in the proxy agent (agy) due to the assertion restriction. In the second query, the delegation on agent agy for a negotiation with cp2, at time instant 10, is only possible for product p1 and conditions [c1].

PROCESS FORMALIZATION Argument-Based Negotiation The use of logic for the formalization of argument-based negotiation does not aim at the definition of the best dealing strategies (although the construction of problem-solving methods for that purpose may turn out to be more stable, taking into account the concepts stated in the formal theory). There are two main objectives: offers and counteroffers are logically justified, and the definition of conflict or attack among opposing parties is clearer. Without arguments, each agent has no way of ascertaining why their proposals and counterproposals are accepted or rejected, due to the limited amount of exchanged information (Jennings et al., 1998).



Global Versus Local Knowledge Each element that composes an argument may come from one of two main sources: global or local knowledge. Global knowledge is shared by the intervening entities and is, therefore, independent of a particular experience or local state. Local knowledge derives from sources that are not common to every agent, giving way to the possibility of contradictory conclusions upon confrontation. Contrary to the definitions found in logical formalizations in law (Prakken, 1993), the KB embedded in each agent may be quite different. The use of global or local knowledge conditions the capacity to determine the winner of a confrontation. As expected, local knowledge is not the best starting point for a premise denial attack (e.g., a claim such as “my experience tells me I sold item X for Y monetary units” is difficult to be stated as false by the counterpart agent, because he cannot say what the particular experiences are of the other agent). In many Business-to-Business (B2B) or Business-to-Consumer (B2C) argumentations, there is often no winner or loser, however, the exchange of arguments among agents is essential so that an acceptable situation for both parties can be reached (even if an agent decides to drop, at any time, the negotiation). Local knowledge is important for an agent to reach another agent’s acceptability region faster (Jennings et al., 1998).

Negotiation Arguments After a theory and a language have been established, in order to represent each agent’s knowledge and information (from which it will draw the justification for each offer and counteroffer), a definition for argument must be reached. An argument is to be constructed progressively, being the antecedent of each rule composed by the consequents of previous rules. This definition is, perhaps, the most important in the logical formalization of argument-based negotiation. Definition 5 (negotiation argument with an implicit meta theorem-solver) Taking ordered theory TNAP, a negotiation argument is a finite, nonempty sequence of rules 〈r1,...,demo(ri,Vi),...,r n〉 such that, for each sequence rule rj with P as a part of the antecedent, there is a sequence rule ri (i < j) on which the consequent is P. The use of such arguments, extended by a threefold logic, is important due to their informative nature, i.e., one of the advantages of using argument-based Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


155

negotiation lies in the fact that information is conveyed in such a way that the counterpart agents are able to evolve their counterarguments in a parallel way (reaching a cooperative usage of knowledge) (Brito et al., 2001b; Jennings et al., 1998). The conclusion of an argument relates to the consequent of the last rule used in that same argument. Formally, see Definition 6. Definition 6 (argument conclusion) The conclusion of an argument A1 = 〈rr1,...,rn〉 , conc(A1), is the consequent of the last rule (rn). As it has been stated, the nature of the knowledge each agent has (local and global) is relevant for arguments and counterarguments. By composing an argument with rules or facts that spawn from local knowledge (e.g., previous experiences), the attack or counterargument launched by the opposing agent during its round is conditioned (due to the fact that local knowledge is hard to deny). Taking into account the two forms of argument attack (conclusion denial and premise denial), a conflict among two opposing agents (e.g., buyer or seller) can be formally specified. Definition 7 (conflict/attack over negotiation arguments) Let A1 = 〈r1,1,...,r1,n〉 be the argument of agent 1 and A2 = 〈r2,1,...,r2,m〉 be the argument of agent 2. Then, (1) if r1,i ∈ A1 or r2,j ∈ A2 are local, the arguments are said to be in “probable conflict” (2) A1 attacks A2 iff A1 executes a conclusion denial attack or a premise denial attack over A2 (3) A1 executes a conclusion denial attack over A2 iff there is no local knowledge involved and conc(A1) is contrary to conc(A2) (4) A1 executes a premise denial attack over A2 iff there is no local knowledge involved, and conc(A1) is contrary to some r2,j ∈ A2 Having in mind the use of rational agents (i.e., those that do not undermine their own actions and are able to formulate coherent arguments), a proper definition of coherency must be formulated.



Definition 8 (argument coherency) An argument A1 = 〈r1,...,rn〉 is said to be “coherent” iff ¬∃ai,aj ai,aj ∈ subarguments(A) ∧ i≠ j: ai attacks aj. Taking into account the definition of conflict and attack and the concept of round, it is possible to logically define the victory/defeat pair. Definition 18 (victory/defeat of negotiation arguments) Let A1 = 〈 r1,1,...,r1,n〉 be the argument of agent 1 and A2 = 〈 r2,1,...,r2,m〉 be the argument of agent 2, and A2 is presented at a later “round” than A1. Then, A1 is defeated by A2 (or A2 is victorious over A1) iff (1) A2 is coherent and A1 is incoherent (2) A2 is coherent, executes a conclusion denial attack over A1 (coherent) and the conclusion rule of A2 is prioritary (taking into account the TNAP theory) over A1 (3) A2 is coherent, executes a premise denial attack over A1 (coherent) and the conclusion rule of A2 is prioritary (taking into account the TNAP theory) over A1

Example Some examples may be presented to illustrate the previous definitions. Let agents E and F be engaged in the process of buying or selling product p1 in an environment with priority rules embedded in the KBs. Agents E and F share general knowledge, market knowledge, and the set of priority rules. Agent E: PE : r5 : price(p1,143). %(experience) price for p1 is 143 %(market) price for p1 is 147 MK : r7 : price(p1,147). GK : r1 : price(p1,150). %(global) price for p1 is 150 PRIO : r4 : priority(PE,GK). %(priority) PE overpowers GK PRIO : r6 : priority(MK,PE). %(priority) PE overpowers GK Agent F: MK : r7 : price(p1,147). GK : r1 : price(p1,150). PRIO : r4 : priority(PE,GK). PRIO : r6 : priority(MK,PE).

%(market) price for p1 is 147 %(global) price for p1 is 150 %(priority) PE overpowers GK %(priority) MK overpowers PE



157

The argument given by agent E might be AE = 〈r4,r5〉, however, agent F might argue with AF = 〈r6,r7〉, representing a conclusion denial attack taking into account the priority rules shared by the community. Agent F is considered the winner due to the fact that it uses an higher priority rule on the set of priority rules.

CONCLUSIONS As previously stated, logic stands as an important tool for formalizing approaches to the development of agent-based software. Logic provides a way to eliminate (or at least reduce) ambiguity and, in the particular case of ELP, is close to a working prototype. EC is an area posing particular problems to the use of agent-based software. Though applications in this area are particularly suited to be solved by agents, no formal development process has been devised for such a field of expertise. However, as previously seen, building agents for EC purposes can be seen as a four-step approach. Starting with the definition of an agent architecture, the processes that take place within and among agents are to be quantified, the reasoning mechanisms formally stated, and the flow of knowledge must be stated. The processes involved in EC, which are difficult to assimilate into traditional systems, revolve around subjective business parameters. Parameters such as gratitude and agreement among parties are nonlinearities, which need to be considered in order to develop a feasible EC system. This information is to be taken into account when drawing up a strategic plan of action. However, once subjective parameters have been quantified, some reasoning must take place before any argument is exchanged with potential counterparts. This stage, which has been referred to as “prenegotiation reasoning” deals with the existence of incomplete information and delineates logical conclusions upon an agent’s KB (e.g., is agent A able to deal product P with agent B at time T). The exchange of arguments among agents serves the main purpose of information exchange. Exchanging justified information provides an agent’s counterpart with enough knowledge to try and reach a common understanding much faster. Formalizing the ways an agent can attack an argument (and which knowledge to use for an effective “victory”) culminates the most important steps in the development of EC-directed agent software.



REFERENCES Analide, C. & Neves, J. (2000). Representation of Incomplete Information, in CAPSI, Conferência da Associação Portuguesa de Sistemas de Informação, Universidade do Minho, Guimarães, Portugal. Baral, C. & Gelfond, M. (1994). Logic Programming and Knowledge Representation, Journal of Logic Programming, 19,20:73–148. Brito, L. & Neves, J. (2000). Agreement and Coalition Formation in MultiagentBased Virtual Marketplaces, Proceedings of the IEA/AIE-2000: The 13th International Conference on Industrial & Engineering Applications of Artificial Intelligence & Expert Systems, New Orleans, LA. Brito, L., Neves, J., & Moura, F. (2000b). A Mobile-Agent-Based Architecture for Virtual Enterprises, Proceedings of the PRO-VE 2000 — 2nd IFIP/Massyve Working Conference on Infrastructures for Virtual Enterprises, Florianópolis, Brasil. Brito, L., Novais, P., & Neves, J. (2000a). Mediation, Agreement and Gratitude in Strategic Planning for Virtual Organisations, Proceedings of the ICEIS2000 — 2nd International Conference on Enterprise Information Systems, Stafford, UK. Brito, L., Novais, P., & Neves, J. (2001a). Temporality, Priorities and Delegation in an E-Commerce Environment, Proceedings of the 14th Bled Electronic Commerce Conference, Bled, Slovenia. Brito, L., Novais, P., & Neves, J. (2001b). On the Logical Aspects of Argument-Based Negotiation among Agents, Proceedings of the CIA 2001 — Fifth International Workshop on Cooperative Information Agents, Modena, Italy. Guttman, R., Moukas, A., & Maes, P. (1998). Agent-Mediated Electronic Commerce: A Survey, Knowledge Engineering Review. Jennings, N.R., Parsons, S., Noriega, P., & Sierra, C. (1998). On Argumentation-Based Negotiation, Proceedings of the International Workshop on Multiagent Systems, Boston, MA. Kephart, J., Hanson, J., & Greenwald, A. (2000). Dynamic Pricing by Software Agents, in Computer Networks. McCarthy, J. (1959). Programs with Common Sense, Proceedings of the Teddington Conference on the Mechanization of Thought Processes, (75–91), London. Neves, J. (1984). A Logic Interpreter to Handle Time and Negation in Logic Data Bases, Proceedings of the ACM 1984 Annual Conference, (50– 54), San Francisco, CA.



159

Norman, T., Sierra, C., & Jennings, N. (1998). Rights and Commitment in MultiAgent Agreements, Proceedings of the 3rd International Conference on Multiagent Systems (ICMAS-98), (222–229). Paris, France. Novais, P., Brito, L., & Neves, J. (2000). Experience-Based Mediator Agents as the Basis of an Electronic Commerce System, Proceedings of the Workshop 2000 — Agent-Based Simulation, Passau, Germany. Novais, P., Brito, L., & Neves, J. (2001). Developing Agents for Electronic Commerce — A Constructive Approach, Proceedings of the SCI 2001 — 5th World Multiconference on Systemics, Cybernetics and Informatics, Orlando, FL. Prakken, H. (1993). Logical Tools for Modeling Legal Argument, Doctoral Dissertation, Free University, Amsterdam. Sartor, G. (1994). A Formal Model of Legal Argumentation, Ratio Juris, 7, 212–226. Wooldrige, M. & Ciancarini, P. (2001). Agent-Oriented Software Engineering: The State of the Art, Agent-Oriented Software Engineering, SpringerVerlag Lecture Notes in AI Volume 1957, Heidelberg: Springer-Verlag.


160

Debenham & Henderson-Sellers

Chapter VIII

Designing Agent-Based Process Systems— Extending the OPEN Process Framework J. Debenham University of Technology, Sydney, Australia B. Henderson-Sellers University of Technology, Sydney, Australia

ABSTRACT Originally a development methodology targeted at object technology, the OPEN Process Framework (OPF) is found to be a successful basis for extensions that support agent-oriented software development. Here we describe the process components necessary to agent-oriented support and illustrate the extensions by means of two small case studies that illustrate the extensions by means of two small case studies that illustrate both taskdriven processes and goal-driven processes. The additional process components for Tasks and Techniques are all generated from the OPF’s metamodel, which gives the OPF its flexibility and tailorability to a wide variety of situations—here agent-orientation.


Designing Agent-Based Process Systems

161

INTRODUCTION AND BACKGROUND Modern software engineering encompasses a wide range of areas of interest. Three of particular interest are object orientation, component-based development, and intelligent multiagent systems. While all three have a different genesis, they all have elements in common, notably, modularity and information hiding with a clear distinction between interface (or specification) and implementation. Intelligent agents have emerged from artificial intelligence research (Wooldridge, 1997), whereas intelligent agent software engineering and methodologies have a firmer relationship to object technology (e.g., Wooldridge et al., 2000). The majority of agent-oriented methodologies have an objectoriented (OO) heredity despite observations that with previously published OO methodologies there is inadequate support for agents; while others adapt knowledge engineering techniques (Wooldridge & Ciancarini, 2001). Of course, OO methodologies that are highly prescriptive and overly specified are hard to extend when a new variant or a new paradigm appears. What is required is a more flexible approach to building methodologies or processes. One such approach will be utilized here: OPEN (Object-oriented Process, Environment and Notation; Graham et al., 1997). Using OPEN, process components are selected from a repository, and the actual methodology (or process) is constructed using identified construction and tailoring guidelines. We identify here what support must be added to OPEN to fully support agents. Following this identification, we give an example application in the context of the development of business process management systems in which the processes may be decomposed into conditional sequences of goals. Goal orientation, as embodied in this class of applications, is readily accommodated in the existing and new tasks and techniques of the OPEN Process Framework (OPF; Firesmith & Henderson-Sellers, 2002) leading to a specification of the individual agents in the system.

DESIGNING AGENT-BASED SYSTEMS A multiagent system is a society of autonomous components, each of which may be constructed independently. They may also be mobile (across hardware platforms and networks)—this aspect is out of the scope of this paper and is not discussed further. Agents can be regarded as powerful versions of objects, constructed using intelligent machinery that supports their autonomous nature and their capability to “take the initiative.” The high-level


162


specification of a multiagent system then, additionally, requires the description of the roles of the agents in their society and the relationships they have with one another, with external agents (including humans and other systems), and with system resources. This high-level specification constrains, but does not necessarily determine, the agent interaction protocol. The specification of the agent interaction protocol is the second step in the design process. Following that, the final step is to specify the design of the agents. An advocated approach to designing a multiagent system (Jennings & Wooldridge, 2001) is sequenced: The System Objective: What is the system intended to do? The Environment: Of what does the system environment consist? To what extent will the agents be able to “see” the things in the environment and to “effect” them? Consideration should be given to people, computer systems, hardware and software sensors and effectors, and existing traditional components. The System Organization: What are the agents in the system, and what are their roles? To whom—in the system or in the environment—do they report and to what extent? To whom can they talk and about what? To what resources (e.g., databases) do they have access? Evaluating the system organization for a multiagent system is highly analogous to the problem of deciding how a (human) office works. The Interaction Protocol: Because the individual agents are autonomous, they require some form of motivation to act. The agent interaction protocol specifies the negotiation mechanisms considered acceptable, e.g., “an English auction with payment made in cash.” Another wellknown, general mechanism is “contract nets,” in which proposals are requested and then evaluated. The interaction protocol is crucial in that it facilitates the agents’ motivation for doing something. In some systems, the goal of the system may not be represented in any of the individual agents, in which case, it will be implicitly buried in the interaction protocol. The Communication Protocol: This is fairly mundane in comparison to the interaction protocol. It includes the actual form of the messages and the language used to encode them. In distributed systems, it also includes the specification of ontologies (e.g., Nowostawski et al., 2000) that determine what the messages actually mean. In multiagent systems, communication is usually asynchronous.



163

The Conceptual Architecture: What architectural components will the agents have? What form of reasoning will each agent require? Reactive (feedback) and deliberative (proactive or feed-forward), or both (called “hybrid”)? How will this reasoning be represented? For example, one form of deliberative reasoning is plans [= state charts of goals]. In this case, we can ask what form of plan is required? This is the start of a highly detailed design for the agents. The Control Architecture: How will the agents work? What mechanism will they use to manage their commitments? What mechanism will they use to schedule their actions? How will the whole thing go? If the system is widely distributed, then how do you turn it all on? The System Operation: How will the agents “learn” to improve their performance? How will they manage their belief revision (this is substantially involved with identifying beliefs that are no longer valid—a complex problem even for humans).

A STARTING POINT: THE OPEN PROCESS FRAMEWORK Having described the design needs for building a multiagent system, we now identify a suitable, existing process infrastructure upon which to build a facility to support the design of multiagent systems. As noted above, OPEN (Graham et al., 1997; Henderson-Sellers et al., 1998) provides a useful starting point, because it is not only defined at the metamodel level but is also itself componentized. Thus, adding further support for the design of intelligent agents is straightforward, such potential extensions having been a priori designed into the metamodel architecture of the OPEN Process Framework (OPF). The OPF is a process metamodel or framework from which can be generated an organizationally specific process (instance). Some of the major elements in this metamodel are Work Units (Activities, Tasks, and Techniques, wherein Activities and Tasks say “what” is to be done, and Techniques say “how” it will be accomplished), Work Products, and Producers. Together, Work Units and Producers create Work Products, and the whole process is structured temporally by the use of Stages (phases, cycles, etc.). Each process instance is created by choosing specific instances of Activities, Tasks, Techniques etc. from the OPF Repository (Figure 1) and specific configurations thereof (created by the application of the Construction Guidelines). OPEN thus provides a high degree of flexibility to the user organization.


164


Figure 1: A personalized OO development process is created from a class repository of process components using supplied construction guidelines. These are all created as instances of metalevel elements in the OPF metamodel (Henderson-Sellers, 2001).

M1 Project

M2

is tailored to meet the needs of a specific

Personalized OO Development Process selection and construction

OPEN Process Framework (metamodel)

generates instances for

offers advice on

Repository of Predefined Process Components describes how to use the Construction Guidelines

user

methodologist

Figure 1 A personalized OO development process is created from a class repository of process components using supplied construction guidelines. These are all created as instances of metalevel elements in the OPF metamodel (after Henderson-Sellers, 2001. Originally published in the Journal of Object-Oriented Programming, October/November 2001, pg. 10).

EXISTING OPEN SUPPORT FOR AGENTORIENTED SOFTWARE ENGINEERING

If we consider the similarities between object-oriented and agent-oriented development at the granularity of OPEN’s Activities, the Tasks relevant to the Activities of Project Initiation, Implementation Planning, and Project Planning will remain relatively unchanged. These are the same for any project. Business approval must be obtained, feasibility studies must be undertaken, and other general tasks must be completed. Activities such as Requirements Engineering and Build will be most affected, because this is where the project domain affects the process. Because OPEN is a full life-cycle process model, it takes into account business, training, and personnel issues. The Activities, Tasks, and



165

Techniques associated with these issues may vary but will not be considered in this paper. Rather, we seek specific existing support for agents. None appears in the original books on OPEN, but in more recent publications (HendersonSellers, 2001) there is a preliminary indication of the need to support agents. A newly proposed Task named Identify intelligent agents is described there. With some minor modification, it states: Identification of agents is in some ways an extension of “finding the objects.” Agents are autonomous entities that have many similarities to objects. A major difference is that whereas an object receiving a request for service must deliver that service, an agent is empowered to say “no.” Agents act when “they feel like it,” and not necessarily when they receive a communication or other stimulus. Agents play roles with responsibilities. These responsibilities are not only equivalent to those for objects (responsibilities for doing, knowing, and enforcing) but also towards achieving organizational goals (Jennings, 2001). They more closely mimic the behavior of people and their decision-making strategies than can objects. Consequently, there is a greater emphasis on the roles that are played by agents. Each role is defined by four attributes: responsibilities, permissions, motivations, and protocols (Wooldridge et al., 2000). Roles are already well supported in OPEN. Thus, agent modeling has parallels with the use of roles in object modeling. An overall “community of agents” can therefore be well modeled at the highest level of abstraction using techniques of role modeling, collaboration diagrams, and other standard OO structuring techniques and diagrams. Some extensions to the Unified Modeling Language have recently been proposed by Odell et al. (2000) in order that the modeling language may be considered applicable for agent modeling. These standard modeling techniques and design notations will therefore not be discussed further here, because we will instead focus on the new additions to the developer’s suite of tools and techniques.

EXTENDING OPEN SUPPORT FOR AGENT ORIENTATION In this section, we outline the various Tasks and Techniques proposed as additions and modifications to the OPF, and especially its Repository, to better facilitate the support of intelligent agent software engineering. These are given mostly in summary form due to space limitations and the need for further refinement as additional research results become available. In particular, the


166


proposed Techniques for Market mechanisms, Activity scheduling, Commitment management, Learning Strategies, and Belief revision are all unresolved research issues. Nonetheless, it is clear that it is highly appropriate to create at least a placeholder of an OPEN Technique for each of these leading-edge techniques.

New Tasks Task: Identify Agents’ Roles A good metaphor for assisting in agent design is to identify what roles they are intended to play. Roles are applied to agents by, e.g., Wooldridge and Ciancarini (2001), in much the same way as they are applied to objects in OOram (Reenskaug et al., 1996) and supported in OPEN (Henderson-Sellers et al., 1998). In Gaia (Wooldridge et al., 2000), roles are defined by four attributes. Responsibilities are similar to those defined for objects but classified by liveness and safety properties. A set of permissions is then allocated to each agent to support these responsibilities. Computations are performed by a set of internal Actions (also called Activities1), and there are a number of associated Protocols that govern external Interactions. This task focuses on the identification of the roles per se together with this associated attribute set. Task: Undertake Agent Personalization In process management applications, agents often represent actual people. People may undertake tasks in their business with different styles (say within an office environment). The software agent must reflect the style of the person they are representing. Because these differences may be significant, this is an important task, which contributes to the overall agent design. Task: Identify Emergent Behavior Emergent behavior is that which exist at the system level but which cannot be predetermined by inspection of the behavior of the individual elements of the system. For example, a colony of ants has an overall behavior not reflected in the behavior of any one individual, whose behavior when studied appears to be almost random. Since these emergent properties are not deterministic, there is a challenge for the designer in identifying likely emergent behavior. Such emergent behaviors are often critical for the success of a system. Task: Model the Agent’s Environment Agents, unlike objects, are situated in an environment: they interact with it by observing and by changing it. The “robot” paradigm is often used to illustrate Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


167

the difference between agents and objects—an agent’s “sensors” and “effectors” are physical or virtual devices by which it interacts with its environment. Task: Determine Agent Interaction Protocol The agent interaction protocol determines how the agents in a system may converse (Kraus, 2001). It specifies what they are allowed to “say” to each other. Agents may cooperate or compete with each other (Durfee, 2001). For competitive agents, the objective of the interaction protocol may be to maximize the utility of the individual agents. Agents are cooperative. A key issue in designing the interaction protocol for cooperative agents is to ensure that the whole system behaves in a coherent way without stifling the autonomy of the individual agents. Another issue for cooperative agent interaction is how to coordinate their activities, particularly when access to resources is limited; this is achieved with a coordination protocol. In process management applications, a coordination protocol is required for personal diary management of the human users (Wobcke & Sichanie, 2000). A common strategy for cooperative interaction protocols is the decomposition and distribution of tasks. This is achieved in the process management applications described here by a “delegation strategy” coupled with the contract net mechanism with focused addressing. Each agent manages the work of its user and deals with the delegation of responsibility for subprocesses to other agents. This delegation is achieved by inviting a selected set of nodes to bid for work. A bid from a node contains information on the firm and preferred constraints that that user presently has, together with information about that user’s work—including an estimate of the cost, in time, that that user may incur. This employs the OPEN Technique: Contract Nets, and is documented by the use of Agent UML (Odell et al., 2000). Task: Determine Agent Communication Protocol Communication protocols underpin the interaction protocols by determining the actual form of messages plus languages and ontologies to be used for their coding. KQML (Knowledge Query and Manipulation Language) is an example of an agent communication language (ACL) (Finin et al., 1997), a second being the Foundation for Intelligent Physical Agents (FIPA)’s ACL (Odell, 2000, 2001). Both are based on speech acts. If Agent A wishes to tell Agent B something, it does so by posting a message to Agent B’s message area (direct interagent communication is also possible, but infrequently utilized; Odell, 2001).


168


Task: Determine Delegation Strategy The delegation strategy is part of the interaction protocol; it determines “what should be delegated to whom.” The delegation strategies described here use the performance knowledge parameters described below. The material in this section and the next are nonstandard and are described in some detail. In process management applications, delegation may involve forming a group (e.g., a committee). Estimating the effectiveness of every possible group of individuals in every possible situation and maintaining the currency of those estimates is not feasible. To measure the performance of groups, the effectiveness of individuals at forming and managing groups is estimated instead; this is feasible. In this way, to form a group, an individual is selected to whom responsibility of forming a group is delegated. In the applications, selection of what to do next may be handled manually by the user or automatically by the system. Delegation of what to whom may be handled manually, semimanually (when the user provides a short list of people to be approached), or automatically. Given a subprocess, suppose that we have some expectation of the payoff Di as a result of choosing the ith individual (i.e., agent and user pair) from the set of candidates {X1,...,Xi,...,Xn} to take responsibility for it. A delegation strategy at time τ is specified as S = {P1,...,P i,...,P n}, where Pi is the probability of delegating responsibility at time τ for a given task to individual Xi chosen from {X1,...,Xi,...,Xn}. For example, the delegation strategy best maximizes expected payoff: Pi =

such that Pr(X maximal  11// m ififXX i isi is such that Pr(X i>>)i ») is is maximal   0 0 otherwise otherwise

where Pr(Xi ») means “the probability that Xi will have the highest payoff.” There are m individuals for whom Pr(Xi ») is maximal. Another strategy prob (defined by Pi = Pr(Xi »)) also favors high payoff but gives all individuals a chance, sooner or later. It is an admissible delegation strategy that has the following properties: • if Pr(Xi ») > Pr(Xj ») then Pi > P j • if Pr(Xi ») = Pr(Xj ») then Pi = P j • Pi > 0 (for all i) So, the strategy best is not admissible. The strategy prob is admissible and is the default delegation strategy. It provides a balance between favoring Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


169

individuals who perform well and giving occasional opportunities to poor performers to improve their performance. The strategy prob is not based on any model of user improvement, and so it cannot be claimed to be optimal in that sense, although it has socially desirable properties. Task: Gather Performance Knowledge To deal with selection and delegation, performance knowledge is gathered. This comprises performance statistics on the operation of every agent, plan, and activity. A set of basic parameters is described here. More recent work includes observations on the relationships between the human users in the system; in particular, the extent to which one human can trust or rely on another. These higher-level parameters will be included in future versions of the method. In the case of a parameter, p, that can reasonably be assumed to be normally distributed, an estimate for the mean of p, µp, is revised on the basis of the ith observation obi to: µpnew = (1 – α) * obi + α * µpold which, given a starting value µpinitial, and some constant α, 0 < α < 1, approximates the geometric mean of all observations to date. Similarly, an estimate for times the standard deviation of p, σp, is revised on the basis of the ith observation obi to: σpnew = (1 – α) * | obi – µpold | + α * σpold which, given a starting value σpinitial, and some constant α, 0 < α < 1, approximates the geometric mean of the modulus of difference of the observations and the mean to date. The constant að is chosen on the basis of the stability of the observations. Each individual agent or user pair maintains estimates for the three parameters: time, cost, and likelihood of success for the execution of all of its plans, subplans, and activities. “All things being equal,” these parameters are assumed to be normally distributed—the case when “all things are not equal” is considered below. Time is the total time taken to termination. Cost is the actual cost of the resources allocated; for example, time used. The likelihood of success observations are binary (i.e., “success” or “fail”) and so the likelihood of the success parameter is binomially distributed, which is approximately normally distributed under the standard conditions. Unfortunately, value is difficult to measure in process management. The system does not attempt to measure value; each individual represents the perceived value of each other individual’s work as a constant for that individual. Finally, the delegate parameter estimates the amount of work delegated to each individual in each discrete time period. The delegate parameter is not normally distributed. The delegate and value estimates are associated with individuals. The Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

170


time, cost, and likelihood of success estimates are attached to plans and activities. The three parameters time, cost, and likelihood of success are assumed to be normally distributed. If working conditions are reasonably stable, then this assumption is acceptable, but the presence of external environmental influences may invalidate it. One virtue of the assumption of normality is that it provides a statistical basis on which to query unexpected observations. If an observation lies outside the expected confidence interval, then there are grounds, to the chosen degree of certainty, to ask why it is outside. Inferred reasons Γ for why an observation is outside expected limits may sometimes be extracted from observing the interactions with the users and other agents involved. If the effect of such a reason can be quantified (perhaps by simply asking a user), then the perturbed values of {obi} are corrected to {obi | Γ}. Performance knowledge is historical. If it is used to support future decisions, then some allowance should be made for how those performance estimates are expected to have changed in time. For example, if A was good yesterday and B was bad 6 months ago, then how should we rate their expected relative performance tomorrow? The probability of A being better than B will be greater than 0.5. The standard deviation of a parameter can be interpreted as a measure of lack of confidence in its mean. It may be shown that if ρ is the expected range of values for A and B and if σB = ρ, then the probability of A being better than B will be less than 0.79 no matter what µB, µA, and σA are. If σB = 2 * ρ, then this probability is less than 0.66. Thus, to allow for the historical estimate of B, determine a period by which the estimates should be “moderately useless,” say 1 year, and increase σB linearly by a half of the difference between its value and 2 * ρ (because 6 months is half of one year). This has the effect of giving B the “benefit of the doubt,” as B has not been given an opportunity for 6 months. Task: Determine Conceptual Architecture The agents used here for process management are three-layer, BDI hybrid agents (Müller, 1996); they exhibit deliberative and reactive reasoning. In other applications, it may be possible to use just one of these forms of reasoning. For example, in agent terms, an expert system is a reactive system. Task: Determine Control Architecture The three-layer, BDI hybrid agents used here employ the nondeterministic procedure: “on the basis of current beliefs—identify the current options, on the basis of current options and existing commitments—select the current commitCopyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


171

ments (or goals), for each newly committed goal, choose a plan for that goal, from the selected plans choose a consistent set of things to do next (called the agent’s intentions).” Task: Determine System Operation Determining the system operation for agents remains a research issue. By including the Task here into the OPEN Process Framework, we are acknowledging the need for such a Task and affiliated Techniques, even though the details of these are not yet fully understood or described, currently being the focus of much research worldwide. We can, however, identify a set of new OPEN Techniques that will be necessary to support this Task. These are Activity scheduling, Commitment management, Learning strategies for agents, Belief revision of agents, and Task selection by agents. Furthermore, documenting such internal designs can be accomplished by the use of the emerging agent modeling language AUML (Agent UML) as described by Odell et al. (2000). In particular, their description of agent-focused state charts and activity diagrams will likely be found useful. Task: Determine Security Policy for Agents Distributed systems attract additional security risks2. Risks include unauthorized disclosure or alteration, denial of service, and spoofing (Odell, 2001). A policy must be determined on how to access resources in a secure manner. Most policies are based on agent identity, itself based on the notion of credential. Task: Code The need to code is already identified within the OPF as OPEN Task: Code. While this was originally intended to describe coding of object-oriented classes, it is equally applicable to agents.

New Techniques Technique: Market Mechanisms OPEN includes various market mechanisms that are useful in the construction of the interaction protocol for competitive agents. These mechanisms include voting mechanisms, negotiation mechanisms, and various auctionbased mechanisms. The negotiation and auction mechanisms are useful for building the interaction protocol for economically rational agents. (Note. This Technique is not used in our subsequent case study, but is included here for completeness.) Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

172


Technique: Contract Nets Contract nets with focused addressing (Chapter 3 by Durfee in Weiss, 1999) are used here to manage semimanual or automatic delegation and have been a mainstay in handling heterogeneity (Durfee, 2001). In the second case study herein, a bid consists of the five pairs of real numbers (Constraint, Delegate, Success, Cost, Time). The pair constraint is an estimate of the earliest time that the individual could address the task, i.e., ignoring other nonurgent things to be done, and an estimate of the time that the individual would normally address the task if it “took its place in the in-tray.” The Constraint estimates require reference to the user’s diary (e.g., Wobcke & Sichanie, 2000). The Delegate pair represents delegations “in” and “out.” The pairs Success, Cost, and Time are estimates of the mean and standard deviation of the corresponding parameters, which are described below by the Task: Gather performance knowledge. The receiving agent then: • Attaches a subjective view of the value of the bidding individual • Assesses the extent to which a bid should be downgraded—or not considered at all—because it violates process constraints • Selects an acceptable bid, if any, possibly by applying its “delegation strategy” If there are no acceptable bids, then the receiving agent “thinks again.” Technique: Commitment Management Another aspect of agents is their need to manage their commitments, of which there are two forms: • The goals that an agent is permitted to achieve—These are commitments that an agent makes to itself. • The things that an agent promises to do for another agent (via the contract net protocol)—These commitments are made only if the agent that asks for bids accepts the other agent’s bid. Then the bidding agent is committed to the other agent to do something. In the process applications described here, a key feature of commitments is their value relative to other commitments. This value typically changes substantially, as new process instances are created and as circumstances change. Unfortunately, this value can only be assessed by a (human) principal, and that is how it is done. The system, though, keeps track of the cost (in time and effort) applied to a process. This enables the system to determine priorities that, for example, ensure that a (now) low value instance in which substantial Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


173

investment has been made and which is near to completion is not overlooked. Tools to estimate the parameters that support this prioritization of commitments are to be provided in OPEN. Technique: Activity Scheduling An agent’s commitments lead to the selection of plans for those commitments. Achievement of the goals and subgoals in those plans requires that activities and tasks be performed. The members of the resulting set of tasks may conflict with each other. Furthermore, they may conflict with the current intentions of other agents, in which case, a coordination protocol is required to resolve such conflict. Thus, in order to schedule these tasks, conflicts are resolved, and priorities are set. This technique is used to support the OPEN Task: Determine system operation. In the process applications described here, the scheduling of activities is substantially governed by the ever-changing availability of the humans in the system. This availability is hard to estimate, as the activities of the humans are not restricted to involvement in processes managed by the system. Each user’s agent maintains an estimate of their user’s capacity and the priorities of the items being dealt with. The problem of scheduling activities is a finer-grained view of the commitment management technique. Technique: Deliberative Reasoning: Plans In the goal-driven process management applications described in the case study, a plan cannot necessarily be relied upon to achieve its goal, even if all of the subgoals on a chosen path through the plan have been achieved. On the other hand, if a plan has failed to execute, then it is possible that the plan’s goal may still have been achieved. So, a necessary subgoal in every high-level plan body is a subgoal called the “success condition.” The success condition (SC) is a procedure with a goal to determine whether the plan’s goal has been achieved. The success condition (a procedure) is the final subgoal on every path through a plan. The execution of the procedure may succeed (9), fail (8), or abort (A). If the execution of the success condition fails, then the overall success of the plan is unknown (?). So, the four possible plan exits resulting from an attempt to execute a plan are as shown in Figure 2. A plan body is represented as a directed AND/OR graph or as a statetransition diagram, in which some of the nodes are labeled with subgoals. The plan body may contain the usual conditional constructs such as if...then and iteration constructs such as while…do... A plan body has one start state (activation condition “ac,” and activation action “α”), and stop states labeled Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

174


Figure 2: The process agent plan

Plan Name[ Plan Goal ] start

• [ ac] / α

Plan body /σ ✓

/υ

/φ ✖

?

[ab] / ω A

Figure 2 The process agent plan as success states “9” (success action “σ”), fail states “8” (fail action “φ”), unknown states “?” (unknown action “υ”), or abort states “A” (abort condition “ab”; abort action “ω”). Each plan contains an optional abort condition [ab] as shown in Figure 2. These abort conditions are realized as procedural abort triggers that are activated when their plan is active. Active abort triggers scan the agent’s beliefs for the presence of their abort condition. Abort triggers are only active if the goal of the plan to which they are attached is a goal that the agent is presently committed to achieving. If a plan is aborted, then any active subgoals of that plan are also aborted. Technique: Reactive Reasoning In reactive reasoning, a trigger is activated by the agent, which, for example, might lead to the continued execution of an active but stalled plan. As well as firing predetermined procedures, the plan can also be aborted should an abort action be activated. The basis of reactive reasoning is a rule of the format: if and then and where the is a device to determine whether the trigger is active or not, and is something that the agent may believe; may be simply to transfer some value to a partly executed plan, or may be more profound such as to abort a plan and decommit a goal.



175

If an agent A has an active plan P that requires input from its user or another agent B, then a procedure sends a request message directly to B with a unique identifier #I, and a reactive procedure trigger is activated (i.e., made “active”): if active and believes B’s response to #I is Z then pass Z to P and not active In this way, data are passed to partly executed plans using reactive triggers. Reactive triggers of this form are associated with belief states of the form “B’s response to #I is known.” Such a procedure trigger is active when its associated subgoal is committed to but has not been realized. The abort triggers have a higher priority than reactive triggers. So, if a plan’s abort trigger fires, and if an active subgoal in that plan is the subject of a reactive trigger, then that subgoal will be deactivated, thus preventing that reactive trigger from firing, even if the required belief is present in the world beliefs. This leads to a more detailed design [OPEN Task: Determine conceptual architecture]. Technique: Learning Strategies for Agents A multiagent system should “learn” to improve its performance. In the process management applications described here, the agents are continually revising their estimates of the performance of the other agents by re-estimating values for each other agent’s performance parameters. Thus, the agents are continually adapting to the changing behavior of the other agents. In other words, each agent is “watching” all the others. In this way, they learn to adapt to changes in circumstances. This technique is a placeholder for future research results to be used to support the OPEN Task: Determine system operation. A set of basic parameters used in our applications are described under Task: Gather performance knowledge. Technique: Control Architecture The control architecture used here is essential to the INTERRAP control architecture (Müller, 1996). In brief, the deliberative reasoning mechanism employs the nondeterministic procedure: “on the basis of current beliefs (identify the current options), on the basis of current options and existing commitments [select the current commitments (or goals), for each newly committed goal, choose a plan for that goal], from the selected plans, choose a consistent set of things to do next (called the agent’s intentions).” If the Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

176


current options do not include a current commitment, then that commitment is dropped. So, if agent A sends agent B a message M asking agent B to do something, agent B may commit to do this. If agent A then removes M from B’s message area, at B’s next deliberative cycle, B should decommit to that task. Technique: Belief Revision of Agents In the BDI (beliefs, intentions, desires) model of agents, the beliefs of agents are derived from reading messages passed to it by a user, from reading documents involved in the process instance, or by reading messages in an allocated message area. This message area is analogous to an office worker’s in-tray, in that messages placed there can be read, put in abeyance, or simply ignored. These beliefs are often modified with time, because they identify facts that are no longer true. There is thus a need for specifying the appropriate technique by which this belief revision is undertaken. This is still to a large extent unresolved, because there is no obvious solution to the problem of deleting beliefs that are no longer true. Agents are just like humans in this respect. In the context of the impact of beliefs on an agent’s parameters, this is not too problematical, though, because estimates will eventually be revised. This technique is often used to support the OPEN Task: Determine system operation. Belief revision has been managed in the case studies described below through making the sender of any message responsible for the removal of that message when it is no longer useful or valid. This is achieved conceptually by each agent having a “notice board” that is freely available to any agent in the system. An agent may then choose not to import the contents of a message into its beliefs if there are inconsistencies. Messages sent in the context of a particular process instance are removed by a garbage collection process when that instance is fully resolved. This simple method works well as long as these responsibilities are clear. The default message format in OPEN includes slots that are used to achieve this. Technique: Task Selection by Agents There may be a number of different tasks that can be invoked to achieve a goal. The selection of tasks by agents is dealt with by the plans and, if there is a choice, by applying a selection strategy. A default selection strategy here is “the probability of choosing a task is the probability that that task is the best task.” This strategy relies on a mechanism to determine the probability that a task is the best task. Mechanisms have been described for estimating the expected cost and value of tasks coupled with a set of human contributors. This Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


177

allows the system to pick appropriate tasks to progress the job at hand given the availability and compatibility of staff, the potential value of the task to a process instance, and its estimated cost. This technique is used by default to support the OPEN Task: Determine system operation. Tools that manage the parameters that underpin these mechanisms are to be made available in OPEN. In the following sections, we exemplify the use of several of the newly proposed Tasks and Techniques in the context of business processes.

TWO CATEGORIES OF BUSINESS PROCESS Business process management is a suitable area for multiagent systems (Jennings et al., 1998, 2000; Koudouridis et al., 2001). The two categories of business process are as follows: • A task-driven process can be associated with a, possibly conditional, sequence of activities such that execution of the corresponding sequence of tasks “always” achieves the process goal. Each of these activities has a goal and is associated with a task that on its termination “always” achieves this goal. Production workflows are often task-driven processes. • A goal-driven process has a process goal and can be associated with a, possibly conditional, sequence of subgoals, such that achievement of this sequence “always” achieves the process goal. Achievement of a subprocess goal is the termination condition for that subprocess. Each of these subgoals is associated with at least one activity and so with at least one task. Some of these tasks may work better than others, and there may be no way of knowing which is which. A task for an activity may fail outright or may be otherwise ineffective at achieving its goal. In other words, unpredictable task failure is a feature of goal-driven processes. If a task fails, then another way to achieve its subgoal may be sought. Figure 3 shows a simplified view of the management of goal-driven processes, in which the primitives are goals and plans. Some goals are associated with executable activities and so with tasks. If a goal is not associated with an activity, then it should be the subject of at least one plan. Figure 3 presents a simplified view, because a subgoal of a goal-driven process goal will not necessarily be goal-driven; aborting plans is also ignored. Two case studies follow. The first considers a task-driven process management system, and the second considers a goal-driven process management system. Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

178


Figure 3: Goal-driven process management (simplified view) Plan

Select Process Knowledge (knowledge of how much the instance has/should cost etc)

Identify ?not SC and not activity goal? Next-Goal (what to try to achieve next) ?activity goal?

Performance Knowledge (knowledge of how effective plans are)

Initialise Process Goal (what we are trying to achieve over all) ?SC?

Back-up

Add to

Select

Identify

Evaluate it

Procedure Add to

Do it

New Performance Knowledge

New Process Knowledge

Figure 3. Goal-driven process management (simplified view)

CASE STUDY 1 To avoid a lengthy description, the subject of both case studies involves the processing of a loan application by a bank. This is typically a workflow or task-driven process. The same example is used for the design of a single-agent system and the design of a multiagent system (see second case study). State charts [OPEN Technique: State modeling] are used to model taskdriven processes (Figure 4). To do this, first construct a node labeled with the activity that creates that process. From that node, directed arcs lead to other nodes labeled with activities, so that every possible sequence of activities that leads to a node that destroys the process is represented. If more than one arc follows a node, then those arcs are labeled with the condition under which they Figure 4: Statechart for a task-driven process

A

α (C) / D

B

Figure 4. Statechart for a task-driven process Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


179

Figure 5: Partial statechart for a loan application process application checked( OK ) / remove from checker and application enter on assessor being assessed assessment timed out( ) / remove from assessor and enter on supervisor application assessed urgently

assessment complete( risky ) / remove from assessor and enter on scrutiniser

application being scrutinised

assessment complete( not risky ) / remove from assessor and enter on loans officer offer being made

Figure 5. Partial statechart for a loan application process

should be followed. No arcs lead from a node that destroys a process. Then, relabel the arcs as α (C)/D, where α is the event that the activity that precedes the arc has terminated, C is the arc condition, if any, and D is the actions that the management system should perform prior to the activity following the arc. Some of what a Web-based process management system has to do is to add or delete pointers to virtual documents to or from the user’s work area. Operations of this sort are represented as actions D on the state chart. For example, Figure 5 shows part of a state chart for a loan application, where the primitives “remove” and “enter” add and delete pointers in this way. For a taskdriven process, the completion of an activity is equated to the realization of the activity’s goal. Thus, the only way that a process instance will not progress is if its activity instance is aborted for some reason, such as time constraints. In Figure 5, the event “assessment timed out” deals with such an eventuality. The resulting state chart is implemented simply as event–condition–action state-transition rules of the form: if in state A and event að occurs and condition C is true then perform action D and enter state B Task-driven process management can be effected using a single reactive agent or expert system containing rules of this form. If the rules in such a knowledge base are indexed by their “from” state, then the maintenance of the knowledge base is quite manageable. For example, the “top” transition in Figure 5 is:


180


Figure 6: State label to manage a “circulating” document

circulate to:

Fred John Jane Mary Peter Michael

considering sent 6 Feb not yet sent verdict: “no” verdict: “yes” considering sent 8 Feb not yet sent

Figure 6. State label to manage a “circulating” document if in state(application being assessed) and event(assessment complete) occurs and condition(application assessed risky) is true, then perform action(remove from assessor’s “Out Tray” and add to scrutinizer’s “In Tray”) and enter state (application being scrutinized). The state label can be quite complex. For example, a state label for a process that is to be circulated among n people, two at a time, until some event occurs, can be represented as an n * 2 matrix—an example is shown in Figure 6.

System Objective and Environment and Organization The system objective is to manage the processes modeled above. The environment [OPEN Task: Model the agent’s environment] consists of users, assumed to have personal computing equipment with a Web browser. The system organization consists of a single-agent system simply managing the (task-driven) processes.

The Conceptual Architecture, the Control Architecture, and System Operation The conceptual architecture is a reactive architecture, i.e., it does not support proactive, feed-forward reasoning, because there is no such need in task-driven processes. All that the system has to do is to implement the event– condition–action state-transition rules described above. The control architecture is to trigger rules in the order in which the triggering events occur. In this simple example, there are no issues to decide for system operation; whereas in Case Study 2, the system operation receives considerably more attention.



181

CASE STUDY 2 Goal-driven processes [OPEN Task: Identify agents’ goals] may be modeled as state and activity charts (Muth et al., 1998). The primitives of that model are activities and states. An activity chart specifies the data flow between activities. An activity chart is a directed graph in which the arcs are annotated with data items. A state chart is a representation of a finite state machine in which the transitions annotated with event–condition–action rules (see Figure 4). Muth et al. (1998) show that the state and activity chart representation may be decomposed to pre-empt a distributed implementation. Each event on a state chart may be associated with a goal to achieve that event, and so a state chart may be converted to a plan with nodes labeled with such goals. Unlike task-driven processes, the successful execution of a plan for a goal-driven process is not necessarily related to the achievement of its goal. One reason for this is that an instance may make progress outside the process management system—two players could go for lunch, for example. That is, when managing goal-driven processes, there may be no way of knowing the “best” task to do next. Each high-level plan for a goal-driven process should terminate with a check of whether its goal has been achieved. To represent goal-driven processes, a form of plan is required that can accommodate failure. This is discussed below. Thus, goal-driven process management has a requirement both for a software architecture that can cope naturally with failure and for some technique for intelligently selecting which is the “best” task to do next (Debenham, 2000) [OPEN Task: Determine conceptual architecture; OPEN Technique: Task selection by agents]. Any general-purpose architecture can achieve this first requirement, but the process architecture described below is particularly suitable.

The System Objective, Environment, and Organization In the goal-driven process management system, an agent supports each (human) user. These agents manage their users’ work and manage the work that a user has delegated to another user/agent pair. Subprocess delegation is the transfer of responsibility for a subprocess from one agent to another [OPEN Technique: Delegation analysis]. A delegation strategy decides who should be given responsibility for doing what. Delegation strategies in manual systems can be quite elementary; delegation is a job that some humans are not very good at. A user of the system may specify the delegation strategy and may permit the agent to delegate or may delegate manually. In doing this, the user has


182


Figure 7: A system node in the multiagent system

Work area

other agents Agent

User Diary

virtual documents

Figure 7. A system node in the multiagent system

considerable flexibility first in defining payoff and second in specifying the strategy. A delegation strategy may attempt to balance some of the three conflicting principles: maximizing payoff, maximizing opportunities for poor performers to improve, and balancing workload (Wooldridge & Jennings, 1998). The objective of the system is to manage goal-driven processes with a specified interaction protocol and communication protocol [OPEN Tasks: Determine agent interaction protocol; Determine communication protocol]. The system’s organization consists of one agent for each (human) user; the role of each agent is that of an assistant to its user. The components of each node in this system are illustrated in Figure 7. The user interacts with a virtual work area and a virtual diary. The work area contains three components: the process instances awaiting the attention of the user, the process instances for which the user has delegated responsibility to another agent, and the process instances that the agent does not understand. The diary contains the scheduled commitments of the user. The agent manages the work area and may also interact with the diary.

The Conceptual Architecture The conceptual architecture of the agents belongs to a well-documented class. Wooldridge describes a variety of architectures (Chapter 1 in Weiss, 1999). One well-documented class of hybrid architectures is the three-layer, BDI agent architectures. One member of this class is the INTERRAP architecture (Müller, 1996), which has its origins in the work of Rao and Georgeff (1995). In the goal-directed process management system, the agent’s concepCopyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


183

Figure 8: Conceptual architecture AGENT other agents

documents

user

looks at seen by reads

BELIEFS

GOALS

s INTENTIONS

c Cooperative Cooperative t plans r h p are i e l and/or g Self Local Local a d lattices g n in e u Procedure Procedures World terms r l of reads reactive procedure triggers send import data from beliefs e messages message to messages manager Social

Figure 8. Conceptual architecture

tual architecture differs slightly from the INTERRAP conceptual architecture; it is intended specifically for business process applications. This conceptual architecture is shown in Figure 8. It consists of a three-layer BDI architecture together with a message area, managed by a message manager. Access to the message area is given to other agents in the system who may post messages there and, if they wish, may remove messages that they have posted. The idea behind the message area is to establish a persistent part of the agent to which the other agents have access. This avoids other agents tampering directly with an agent’s beliefs and enables agents to freely remove their messages from a receiving agent’s message board if they wish. The message area is rather like a person’s office “in-tray,” into which agents may place documents and from which they may remove those documents if they wish. The agents’ world beliefs are derived from reading messages received from a user or from reading the documents involved in the process instance or from reading messages in the message area. These activities are fundamentally different in that documents are “passive”; they are read only when information is required. Users and other agents send messages when they feel like it. Beliefs play two roles. First, they may be partly or wholly responsible, activating a local or cooperative trigger that leads to the agent committing to a goal and may thus initiate an intention (e.g., a plan to achieve what a message asks, such as “please do xyz”). This is part of the deliberative reasoning mechanism [OPEN Task: Determine reasoning strategy for agents]. Second, they can be partly or wholly responsible for activating a reactive procedure trigger that, for example, enables the execution of an active plan to progress. This is part of the reactive reasoning mechanism.


184


Figure 9: Plan for assessment example illustrated in Figure 5 Assess_application[ Application X has been dealt with in time τ ] start

•

Application X assessed (RESULT = X ) by time ( NOW + τ – ∆ ) ✓ RESULT = OK

✖ RESULT = ¢ OK

Acknowledgement for application X received from Loans agent by time ( NOW + ∆ ) ✓

✓

/ AdU( #1 )

✖

✖

Acknowledgement for application X received from Supervisor agent by time ( NOW + ∆ )

Acknowledgement for application X received from Scrutiniser agent by time ( NOW + ∆ ) ✓

/ AdU( #2 ) ✖

✓

✖

/ AdU( #3 ) ✖

✓

✖

✓

Figure 9. Plan for assessment example illustrated in Figure 5 Deliberative Reasoning

The form of plan used here is slightly more elaborate than the form of agent plan described in Rao and Georgeff (1995), where plans are built from singleentry, triple-exit blocks. Though that approach is powerful, it is inappropriate for process management, because in this case, whether a plan has executed successfully is not necessarily related to whether that plan’s goal has been achieved. In the first case study, we described the inclusion of time and space constraints in a state-chart-based process model. The inclusion of these constraints substantially increases the complexity of that model. Consider now the management of the same process (Figure 5). Consider the agent for the Assessor. That agent may have a plan similar to that shown in Figure 9, which shows how constraints are dealt with in that formalism. That plan may appear to be conceptually no simpler than the representation shown in Figure 4, but it also has to cope with the interagent communication [OPEN Task: Determine communication protocol]. Three “fail actions” are shown in Figure 9: “AdU” is a hard-wired action that means “advise the agent’s user,” with the corresponding error message signaling that some calamity has occurred. In this example, the three hard-wired actions indicate that no acknowledgment was received from the three respective agents within some preset time Dð. No “?” or “A” states are shown in Figure 9; for each of the four subgoals, the “8” should be entered in lieu of an abort or unknown state.



185

Reactive Reasoning Reactive reasoning plays two roles in this case study: first, if a plan is aborted, then its abort action is activated; second, if a procedure trigger fires, then its procedure is activated—this includes hard-wired procedure triggers that deal with urgent messages such as “the building is on fire!” Of these two roles, the first takes precedence over the second.

The Control Architecture The control architecture is essentially the INTERRAP control architecture. In outline, the deliberative reasoning mechanism employs the nondeterministic procedure: “on the basis of current beliefs—identify the current options, on the basis of current options and existing commitments— select the current commitments (or goals), for each newly committed goal, choose a plan for that goal, from the selected plans choose a consistent set of things to do next (called the agent’s intentions).” If the current options do not include a current commitment, then that commitment is dropped. So if agent A sends agent B a message M asking agent B to do something, agent B may commit to do this. If agent A then removes M from B’s message area, then, at B’s next deliberative cycle, B should decommit to that task. The reactive reasoning mechanism takes precedence over the deliberative reasoning mechanism. The reactive frequency is the frequency at which an attempt is made to fire all active reactive triggers. The reactive frequency here is 30 seconds. The deliberative frequency is the frequency at which the deliberative reasoning mechanism is activated. To maintain some stability in each user’s work area, the deliberative frequency is 5 minutes.

The System Operation For goal-directed processes, there may be no way of knowing what the “best” thing to do next is, and that next thing may involve delegating the responsibility for a subprocess to another agent (Lind, 2001). This raises two related issues. The first issue is selection; that is, given a goal, select the “best” plan or activity for that goal. The second issue is delegation; that is, the problem of deciding whom to ask to take responsibility for what and then following up on the delegated responsibility to make sure that the work is done. The sense in which “best” is used here does not mean that selection and delegation are optimization problems. A process management system is one part of an organization. Ideally, the goal of a process management system


186


should be that of its organization, such as “to maximize corporate profits.” But, unless measurements on the full range of corporate activity are available to it, the management system is unable to address such a global goal. On the other hand, attempts to optimize the performance of the process management system only can lead, for example, to overuse of the best performing staff. So, if the only measurements available are derived from within the process management function, then the meaning of “best” should take note of global implications, such as the equity in the working environment [OPEN Task: Model the agent’s environment], as well as the quality of the process output. A definition of “best” in functional process management terms that attempts to address corporate priorities may lead to conflicting principles, such as maximizing payoff, providing opportunities for poor performers to improve, and balancing workload. In the absence of a satisfactory meaning of “best” and with only the performance knowledge to guide the decisions, the approach taken to plan/ activity selection is to ask the user to provide a utility function defined in terms of the performance parameters. If this utility function is a combination of (assumed) normal parameters, then a reasonable plan/activity selection strategy [OPEN Task: Identify agents’ goals] is given a goal to choose each plan (or activity) from the available plans (activities) with the probability that the plan (activity) has the highest expected utility value. Using this strategy, even poor plans have a chance of being selected, and, maybe, performing better than expected. Contract nets [OPEN Technique: Contract nets] with focused addressing are often used to manage semimanual or automatic delegation.

ASSESSMENT The applications built are distributed multiagent systems. This enables the management of complex tasks to be handled, as each node is individually responsible for the way in which it goes about its business. That is, the plan in each agent only has to deal with the goals the agent has to achieve. For example, a complete high-level plan for an Assessor agent is shown in Figure 9. This simplifies the design of plans for the agents. As a system built from autonomous components, each node in the goal-driven system has to cope with the unexpected failure of other nodes. This complicates the design of the plans. The delegation strategy was discussed above. An overriding principle is required to determine how delegation is to be dealt with no matter what parameters are used to support it. For example, if A delegates the responsibility for a subprocess to B, who, in turn, delegates the same subprocess to C, then



187

should B advise A of this second delegation—thus removing B from the responsibility chain—or should B remain in the responsibility chain? Distributed, goal-driven systems are considerably more expensive (approximately four times the programming effort) to build than task-driven systems. Having made this investment, dividends flow from the comparative ease by which new processes are included, in that only those agents involved in a process need to develop plans to cope with that process. There is also a negative here. The system has grown around a principle of personalization, i.e., each individual is responsible for deciding how their agent operates. This means that similar plans may be constructed at a number of nodes by the users at those nodes to deal with the same subprocess. One way of managing this is to publish solutions as they are constructed, but that has not been considered. The application of the extended OPEN Process Framework has been exemplified only in terms of its support for particular tasks and techniques, mostly at the design level. This worked well. Future evaluations need to create a full life-cycle development process, which will include not only technical but also management issues, including a well-specified life-cycle model, such as the Contract-Driven Life-cycle (CDLC) model generally advocated for use in OPEN process instances. The CDLC (or other) life-cycle model adds concepts such as stages, phases, builds, milestones as project management focused sequencing superimposed upon the technical Activity/Task interfacing and sequencing discussed here.

FUTURE TRENDS Agents are beginning to enter the mainstream of software engineering (Jennings, 2001), emerging from the research laboratory into commercial utilization. In order to build real agent-oriented systems for commercial applications, their design must encompass all aspects of traditional software engineering, not just focus (as in the past) on internal design issues. The type of methodological approach described here and exemplified by the OPEN Process Framework (Firesmith & Henderson-Sellers, 2002) or by the Gaia approach (Wooldridge et al., 2000), which binds together agent technology and solid OO-based processes and methodologies, should provide, in the near future, the required substantive support. Value is also seen in codeveloping the “theory” of the process-focused approach and its application and evaluation in case studies are undertaken in parallel.


188


SUMMARY A first attempt to augment the OPF with support for agents has been described. While the initial results are promising, there is still much methodology refinement that is necessary before full methodological support for agentoriented software development becomes commercially available. Proposed extensions (new Tasks and Techniques for OPEN) are illustrated in two case studies of business processes.

ENDNOTES 1

2

Note that this is an entirely different meaning to word Activity from that as used in OPEN. This is particularly true for an agent-oriented system, when agents are (potentially at least) mobile [a future OPEN Task: Model agent mobility].

REFERENCES Debenham, J.K. (2000). Supporting strategic process. In Proceedings Fifth International Conference on The Practical Application of Intelligent Agents and MultiAgents (237–256). Manchester, UK. Durfee, E.H. (2001). Scaling up agent coordination strategies. IEEE Computer, 34(7), 39–46. Finin, F., Labrou, Y., & Mayfield, J. (1997). KQML as an agent communication language. In J. Bradshaw (Ed.), Software Agents. Cambridge, MA: MIT Press. Firesmith, D.G. & Henderson-Sellers, B. (2002). The OPEN Process Framework. An Introduction. Harlow, UK: Addison-Wesley. Graham, I., Henderson-Sellers, B., & Younessi, H. (1997). The OPEN Process Specification. Harlow, UK: Addison-Wesley. Henderson-Sellers, B. (2001). Enhancing the OPF repository. JOOP, 14(4), 10–12, 22. Henderson-Sellers, B., Simons, A.J.H., & Younessi, H. (1998). The OPEN Toolbox of Techniques. Harlow, UK: Addison-Wesley. Jennings, N. (2001). Agent of change. Application Development Advisor, 5(3), 6. Jennings, N.R. & Wooldridge, M. (2001). Agent-oriented software engineering. In J. Bradshaw (Ed.), Handbook of Agent Technology (in press). Cambridge, MA: AAAI/MIT Press. Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


189

Jennings, N.R., Faratin, P., Norman, T.J., O’Brien, P., & Odgers, B. (2000). Autonomous agents for business process management. Int. J. Applied Artificial Intelligence, 14 (2), 145–189. Jennings, N.R., Sycara, K., & Wooldridge, M. (1998). A roadmap of agent research and development. Int. Journal of Autonomous Agents and MultiAgent Systems, 1 (1), 7–38. Koudouridis, G., Corley, S., Dennis, M., Ouzounis, V., Van Laenen, F., Garijo, F., & Reynolds, H. (2001). Communications management process integration using software agents: a specification of a framework for agent oriented workflow management systems. EURESCOM Project P815. http://www.eurescom.de/public/projectresults/P800-series/ 815d1.htm. Kraus, S. (2001). Strategic Negotiation in Multiagent Environments. Cambridge, MA: MIT Press. Lind, J. (2001). Iterative Software Engineering for Multiagent Systems: The Massive Method. Berlin, German: Springer-Verlag. Müller, J.P. (1996). The Design of Intelligent Agents. Berlin, Germany: Springer-Verlag. Muth, P., Wodtke, D., Weißenfels, J., Kotz, D.A., & Weikum, G. (1998). From centralized workflow specification to distributed workflow execution. J. Intelligent Information Systems, 10(2). Nowostawski, M., Bush, G., Purvis, M., & Cranefield, S. (2000). Platforms for agent-oriented software engineering. In Proceedings 7th Asia Pacific Software Engineering Conference (480–488). Los Alamitos, CA: IEEE Computer Society Press. Odell, J. (2000). Objects and agents: how do they differ? JOOP, 13(6), 50– 53. Odell, J. (2001). Key issues for agent technology, JOOP, 13(9), 23–27, 31. Odell, J., Van Dyke Parunak, H., & Bauer, B. (2000). Extending UML for agents. In G. Wagner, Y. Lesperance, & E. Yu (Eds.), Proceedings Agent-Oriented Information Systems Workshop, 17th National Conference on Artificial Intelligence (3–17). Austin, TX. Rao, A.S. & Georgeff, M.P. (1995). BDI agents: from theory to practice. In Proceedings First International Conference on Multiagent Systems (312– 319). San Francisco, CA. Reenskaug, T., Wold, P., & Lehne, O.A. (1996). Working with objects. the OOram software engineering manual. Greenwich, CT: Manning. Weiss, G. (Ed). (1999). Multiagent Systems. Cambridge, MA: The MIT Press. Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

190


Wobcke, W. & Sichanie, A. (2000). Personal diary management with fuzzy preferences. In Proceedings Fifth Int. Conf. on The Practical Application of Intelligent Agents and MultiAgents. Manchester, UK. Wooldridge, M. (1997). Agent-based software engineering, IEE Procs. Software Eng., 144, 26–37. Wooldridge, M. & Ciancarini, P. (2001), Agent-oriented software engineering: the state of the art. In Agent-Oriented Software Engineering. P. Ciancarini & M. Wooldridge (Eds.). Berlin, Germany: Springer Verlag, pp. 1–22. Wooldridge, M. & Jennings, N.R. (1998). Pitfalls of agent-oriented development. In Procs. 2nd Int. Conf. on Autonomous Agents (385–391). Minneapolis/St. Paul, MN. Wooldridge, M., Jennings, N.R., & Kinny, D. (2000). The Gaia methodology for agent-oriented analysis and design. J. Autonomous Agents and MultiAgent Systems, 3, 285–312.


Toward an Organization-Oriented Design Methodology

191

Chapter IX

Toward an OrganizationOriented Design Methodology for Agent Societies Virginia Dignum Utrecht University, The Netherlands Hans Weigand Tilburg University, The Netherlands

ABSTRACT In this chapter, we present a framework for the design of agent societies that considers the influence of social organizational aspects on the functionality and objectives of the agent society and specifies the development steps for the design and development of an agent-based system for a particular domain. Our approach will provide a generic frame that directly relates to the organizational perception of the problem. The framework specifies the development steps of the design and development of an agent-based system for a particular domain. Based on the coordination characteristics of a domain, the methodology provides three frameworks for societies (market, hierarchy, and network). These frameworks relate to the organizational perception of a problem and allows for existing methodologies to be used for the development, modeling, and formalization of each step. The methodology supports the development of increasingly detailed models of the society and its components.


192

Dignum & Weigand

INTRODUCTION In an increasing number of domains, organizations need to work together in transactions, tasks, or missions. Work relationships between people and enterprises are shifting from the “job-for-life” paradigm to project-based virtual enterprises in which people and organizations become independent contractors. These considerations lead to an increasing need for a transparent representation and implementation of work processes. In such settings, the ability to organize and maintain business processes, the support of communication and collaboration, and the management of knowledge are issues that are increasingly more important to insure the survival and sustainable advantage of organizations. The fact that business processes are highly dynamic and unpredictable makes it difficult to give a complete a priori specification of all activities that need to be performed, which are their knowledge needs, and how they should be ordered. In organizations, there is often a decentralized ownership of data, expertise, control, and resources involved in business processes. Different groups within organizations are relatively autonomous, in the sense that they control how their resources are created, managed, or consumed, and by whom, at what cost, and in what time frame. Often, multiple, physically distributed organizations (or parts hereof) are involved in one business process. Each organization, or part of an organization, attempts to maximize its own profit within the overall activity. There is a high degree of natural concurrency (many interrelated tasks and actors are working simultaneously at any given point of the business process), which makes it imperative to be able to monitor and manage the overall business process (e.g., total time, total budget, etc.). Software agents, characterized as autonomous entities with reasoning and communicative capabilities, are utmost suitable to implement, simulate, or represent autonomous real-life entities and, therefore, are an ideal means to model organizations. It is commonly accepted that agents are an effective solution in situations where the domain involves a number of distinct problemsolving entities, data sources, and other resources that are physically or logically distributed and that need to interact to solve a problem. Therefore, because of the proactive and autonomous behavior of agents, it is natural to design organizational support systems using agent societies that mimic the behavior and structure of human organizations (Zambonelli et al., 2001). In order to make agent technology widely accepted and used in industry, it is necessary to clearly specify the types of problems suitable for an agent approach and the benefits of agents above other technologies. Furthermore, it



193

is necessary to develop engineering methodologies that focus not only on the internal organization of each of the intervening agents but also on the social aspects of the domain (Omicini, 2001). However, as yet, there is no wellestablished and all-encompassing agent-oriented methodology that covers the whole development process of agent systems from requirement acquisition to implementation and testing. Most existing methodologies concentrate on just one part of the total picture or are too formal to be applicable in practice. A methodology for designing multiagent systems must be specific enough to allow engineers to design the system and generic enough to allow the acceptance and implementation of multiagent systems within an organization, allowing for the involvement of users, managers, and project teams.

Objectives We propose a framework that describes all the stages of development of a multiagent system, takes an organizational perspective on systems design, and specifies all the development steps for the design and development of an agentbased system for a particular domain. Specific agent-oriented methodologies can be used for the development and modeling of each of the development steps. We believe that such a generic framework, based on the organizational view, will contribute to the acceptance of multiagent technology by organizations. Following the development criteria proposed by Sycara (1998), we define a social framework for agent communities based on organizational coordination models that “implement” the generic interaction, cooperation, and communication mechanisms that occur in the problem domain. The proposed methodology allows a generic coordination model to be tailored to a given application and its specific agent roles and interactions to be determined.

BACKGROUND Social Concepts in Agent Research Nowadays, there is a rising awareness that multiagent systems and cybersocieties can best be understood and developed if they are inspired by human social phenomena (Artikis et al., 2001; Castelfranchi, 2000; Zambonelli et al., 2001). This is in many ways a novel concept within agent research, even if sociability has always been considered an important characteristic of agents. Until recently, the relation between environment and agent has been considered from an individualistic perspective, that is, from the perspective of the agent, in terms of how it can affect the environment or be affected by it.


194

Dignum & Weigand

In an individualistic view of multiagent systems, agents are individual entities socially situated in an environment, that is, their behavior depends on and reacts to the environment and to other agents in it (Dautenhahn, 2000). It is therefore not possible to impose requirements and objectives to the global aspects of the system, which is paramount in business environments. When multiagent systems, or agent societies, are considered from an organizational point of view, the concept of desirable social behavior becomes of utmost importance. In a business environment, the behavior of the global system and the collective aspects of the domain, such as stability over time, predictability, and commitment to aims and strategies, must be considered. Organization-oriented agent societies take a collectivist view that considers agents as being socially embedded (Edmonds, 1999). If an agent is socially embedded, it needs to consider not only its own behavior but also the behavior of the total system and how they influence each other. Multiagent systems that are developed to model and support organizations need coordination frameworks that mimic the coordination structures of the particular organization. The organizational structure determines important autonomous activities that must be explicitly organized into autonomous entities and relationships in the conceptual model of the agent society (Dignum et al., 2001). Furthermore, the multiagent system must be able to dynamically adapt to changes in organization structure, aims, and interactions.

Society Models—A Brief Overview The term society is used in a similar way in agent society research as in human or ecological societies. The role of any society is to allow its members to coexist in a shared environment and pursue their respective roles in the presence of or in cooperation with others. Main aspects in the definition of society are purpose, structure, rules, and norms. Structure is determined by roles, interaction rules, and communication language. Rules and norms describe the desirable behavior of members and are established and enforced by institutions that often have a legal standing and thus lend legitimacy and security to members. A further advantage of the organization-oriented view on designing multiagent systems is that it allows for heterogeneity of languages, applications, and architectures during implementation. AALAADIN (Ferber & Gutknecht, 1998) is a model for agent societies based on the organizational perspective. This model is based on the basic notions of agent, role, and group. Groups in AALAADIN are defined as atomic sets of agents and do not incorporate the notion of goal, which we feel is an important aspect of societies, because usually, societies are created and Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


195

maintained to realize a certain objective, dependent on the domain goals and requirements. Artikis et al. (2001) provide a formal characterization of agent societies that views societies as normative systems and describes agent behavior and society rules in terms of the normative consequences of the agent role in the society. This society is neutral to the internal architecture of the agents and explicitly represents the communication language, norms, and behavior rules, and agent ownership as parameters of the society model. Recently, Davidsson proposed a classification for artificial societies based on the following characteristics (Davidsson, 2001): • Openness—Describing the possibilities for any agent to join the society • Flexibility—Indicating the degree agent behavior is restricted by society rules and norms • Stability—Defining the predictability of the consequences of actions • Trustfulness—Specifying the extent to which agent owners may trust the society Based on this classification, two new types of agent societies, semiopen and semiclosed, are introduced that combine the flexibility of open agent societies with the stability of closed societies. This balance between flexibility and stability results in a system in which trust is achieved by mechanisms that enforce ethical behavior between agents.

Coordination Relating society models to the organizational perception of the problem can facilitate the development of organization-oriented multiagent systems. That is, a common ground of understanding must be found between agent engineers and organizational practitioners. Coordination is the ideal candidate as common ground. It is generally recognized that coordination is an important problem inherent to the design and implementation of multiagent systems (Bond & Gasser, 1998), but the implications of coordination models for the architecture and design of agent societies are not often considered. Based on ideas from organizational science research, we propose a framework for agent societies that considers and reflects the implications of the coordination model of the real-life organization being modeled. In the following, we highlight some views on coordination that currently hold in economics and organizational sciences and in computer science and distributed artificial intelligence. Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

196

Dignum & Weigand

Table 1: Comparison of organizational forms Co-ordination Relation form Primary means of communication Tone or Climate Conflict Resolution

MARKET Price mechanism Competition Prices

NETWORK Collaboration Mutual interest Relationships

Precision/ suspicion

c Open-ended / mutual Formal/ bureaucratic bureaucratic benefits Reciprocity Supervision (Reputation)

Haggling (Resort to courts)

HIERARCHY Supervision Authority Routines

Coordination and Organizational Forms Economics and organizational theory consider that relationships between and within organizations are developed for the exchange of goods, resources, information, and so on. Transaction costs and interdependencies in organizational relationships determine different models for organizational coordination, such as markets, hierarchies, and networks (Nouwens & Bouwman, 1995). Coordination in markets is achieved mainly through a price mechanism that allows independent actors to search for the best bargain. Hierarchies are mainly coordinated by supervision, that is, actors involved in power-dependent relationships act according to routines. Networks achieve coordination by mutual interest and interdependency. Table 1 summarizes the characteristics of these organization forms. Coordination and Interaction Forms In computer science and distributed artificial intelligence, coordination is usually defined as the art of managing interactions and dependencies among activities. Coordination languages are a class of programming notations that offer a solution to the problem of specifying and managing the interactions among computing agents. From this point of view, coordination models can be divided into two classes: control-driven and data-driven (Papadopoulos & Arbab, 1998). Control-driven models are systems made up of a well-defined number of entities and functions, in which the flow of control and the dependencies between entities need to be regulated. The data-driven model is more suited for open societies, in which the number of entities and functions is not known a priori, and cooperation is an important issue. Where the classification of cooperation provided by organizational theory stems from social considerations and transaction costs, this classification is concerned with the way interactions between agents happen. Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


197

Agent-Oriented Software Engineering Methodologies The application of the agent paradigm to the development of different applications calls for a development methodology that focuses not only on the internal organization of each of the intervening agents but also on the social aspects of the domain. Such methodology should provide models and methods for all types of activities throughout all phases of the software life cycle. Because of similarities between agents and objects, it has often been claimed that existing object-oriented (OO) methodologies can be used for the development of agent-based systems. However, it has also been noted that agents possess specific characteristics that are not covered by traditional OO methodologies (Omicini, 2000; Jennings et al., 1998). One fundamental difference between agents and objects is autonomy, which refers to the principle that agents have control over their own actions and internal state. That is, agents can decide whether or not to perform a requested action. Objects have no control over their own methods, that is, once a publicly accessible method is invoked, the corresponding actions are performed (Wooldridge, 1997). Another characteristic of multiagent systems that calls for specific methodological approaches is openness. Because components and relationships in an open system can change at any time, designers cannot be certain of the system’s behavior at design time. Frederiksson defends that a methodological cycle for the engineering of agent societies must comprise principles of observation and construction of systems (Frederiksson & Gustavsson, 2001). From an organizational point of view, the behavior of individual agents in a society can only be understood and described in relation to the social structure. Therefore, the engineering of agent societies needs to consider the interacting and communicating abilities of agents as well as the environment in which agent societies are situated. Furthermore, in open societies, the “control” over the design of participating agents lays outside the scope and design of the society. That is, the society cannot rely on the embedding of organizational and normative elements in the intentions, desires, and beliefs of participating agents. These considerations lead to the following requirements for engineering methodologies for agent societies (Dignum & Dignum, 2001): • The methodology must include formalisms for the description, construction, and control of the organizational and normative elements of a society (roles, norms, and goals). • The methodology must provide mechanisms to describe the environment of the society and the interactions between agents and the society, and to


198

• • •

Dignum & Weigand

formalize the expected outcome of roles in order to verify the overall animation of the society. The organizational and normative elements of a society must be explicitly specified, because an open society cannot rely on its embedding in the agent’s internal structure. Methods and tools are needed to verify whether or not the design of an agent society satisfies its design requirements and objectives. The methodology should provide building directives concerning the communication capability and ability to conform to the expected role behavior of participating agents.

In our opinion, none of the currently existing agent-oriented engineering methodologies, such as Gaia (Wooldridge et al., 2000) and SODA (Omicini, 2001) fulfill all of the above requirements yet.

AGENT SOCIETY FRAMEWORKS The way in which organizations describe and achieve coordination is determinant to the specification of coordination in agent societies. Different application contexts exhibit different needs with respect to coordination, and the choice of a coordination model has great impact in the design of the agent society. The overall goals of a society are domain dependent, but all societies depend on a facilitation layer that provides the social backbone of the organization (Dellarocas, 2000). This layer deals with the functioning of the society and relates to the underlying coordination model. Therefore, we argue that the first step in the development of agent societies is to identify the underlying coordination model. We have specified generic facilitation and interaction frameworks for agent societies that implement functionality derived from the type of coordination holding in the domain. The coordination model determines interaction patterns and functionality of the facilitation layer of the agent society, that is, the interaction primitives and agent roles necessary to implement the facilitation layer are specific to each type of society (market, network, or hierarchy). Moreover, coordination models provide a framework to express interaction between the activities of agents and the social behavior of the system (Ciancarini et al., 1999). In the following, we describe the characteristics of agent society frameworks based on different coordination models.



199

Market Framework The main goal of markets is to facilitate exchange between agents. In a market, heterogeneous agents will strive to find partners or clients with whom to trade their services. Being open systems, market architectures assume the heterogeneity of their members, in structure, goals, and ways of acting. Markets are particularly suitable to situations in which resources are overlapping and agents need to compete for them and are, therefore, a good choice for modeling product or service allocation problems. Being self-interested, agents will first try to solve their own local problem, after which agents can potentially negotiate with other agents to exchange services or goods in shortage or in excess. The decision to enter into or cancel a transaction is usually left to the agent. The facilitation activities of such an agent society are mainly limited to help agents find suitable partners through identification and matchmaking. Matchmakers keep track of agents in the system, their needs and possibilities, and mediate in the matching of demand and supply of services. Identification and reputation facilities are meant to build the confidence of customers as well as offer guarantees to society members. Furthermore, it is necessary to define ways to value the goods to be exchanged and determine profit and fairness of exchanges, which can be accomplished by providing banking facilities and currency specification. Interaction in markets occurs through communication and negotiation. A specific kind of market interaction is the auction that uses a highly structured negotiation protocol.

Network Framework Networks are coalitions of self-interested agents that agree to collaborate for some time in order to achieve a mutual goal. Relationships between agents are dependent on clear communication patterns and social norms. The society is responsible for making rules and norms known to its potential members. Agents in a network society are self-interested but still willing to trade some of their freedom to obtain secure relations and trust. Relationships between agents are described by contracts. Agents also enter a social contract with the network society, in which they commit themselves to act within and according to the norms and rules of the society and of the role they will assume. Besides matchmakers as in market frameworks, other types of facilitation roles in networks are gatekeepers, notaries, and monitoring agents. Gatekeepers are responsible for accepting and introducing new agents to the market. Gatekeepers negotiate the terms of a social contract between the applicant and the members of the market. Notaries keep track of collaboration contracts between agents. Monitoring agents are trusted third parties. Appointing a Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

200

Dignum & Weigand

monitor agent to a contract is the equivalent to the setting up of a supercontract between the contracting agents and the environment (personified by the monitoring agent). This supercontract specifies that the monitoring agents are allowed to check the actions of contracting agents and that the contracting agents must submit to the sanctions imposed.

Hierarchy Framework In a hierarchy, the flow of resources or information is coordinated through adjacent steps by controlling and directing it at a higher level in the managerial hierarchy. Managerial decisions, and not negotiation and communication as in markets, determine the interaction possibilities and the design of hierarchical societies. Demand parties do not select a supplier from a group of potential suppliers; they simply work with a predetermined one. In a hierarchy interaction, lines are well defined, and the facilitation level assumes the function of global control of the society and coordination of interaction with the outside world. Environments such as automated manufacturing planning and control are well suited to the hierarchical model. In such systems, reliable control of resources and information flow requires central entities that manage local resources and data but also need quick access to global ones. In the hierarchy model, agents are usually cooperative, not guided by their self-interest but by their orientation to a common global goal. In this architecture, communication lines between agents are predefined, and agents are usually not free to enter or leave their roles in the system. Agents have a local perspective, and their actions are therefore determined by their local states. In a hierarchical architecture, facilitation layer agents are mainly dedicated to the overall control and optimization of the system activities. Facilitation roles are controllers that monitor and orient the overall performance of the system or of a part of the system and interface agents that regulate communication between the society and the outside world. Table 2: Coordination in agent societies Type of society Agent ‘values’ Facilitation roles

MARKET Open Self interest Matchmaking Banking

NETWORK HIERARCHY Trust Closed Mutual interest/ Collaboration Dependency Interface Gate-keeping Control Matchmaking Notary Monitoring



201

APPLICATION EXAMPLE We are currently applying the methodology described in this paper to the development of a system that supports knowledge exchange between nonlife insurance experts at Achmea, a large financial and insurance services company in the Netherlands. This Knowledge Exchange Network preserves existing knowledge, rewards knowledge owners, and reaches knowledge seekers in a “just in time, just enough” basis. The network will serve as a knowledge repository as well as a means for support and encouragement of communication and collaboration. In this section, we briefly describe the aims of an agent society being developed to model the knowledge-sharing activities. In the next section, this example will serve as illustration for the different methodological steps. In this setting, knowledge seekers and knowledge owners want to be able to decide on trade partners and conditions. Sharing is not centrally controlled but is greatly encouraged by the management. The best-suited partner, according to each participant’s own conditions and judgement, will get the “job.” However, factors such as privacy, secrecy, and competitiveness between brands and departments may influence the channels and possibilities of sharing and must thus be considered. The project stakeholders have expressed the following requirements: • The organization aims at supporting collaboration and extending synergy and at supporting the preservation and organization-wide availability of existing knowledge • Knowledge owners are willing to share their knowledge within a group they feel they can trust; that is, they wish to be able to decide on sharing decisions and conditions; furthermore, added value of the sharing effort and fair exchange is a must (that is, the feeling that one is rewarded for share) • Knowledge seekers are not aware of existing knowledge and knowledge owners; they also wish to be able to decide on acquisition conditions and partners, and furthermore, an accreditation and certification mechanism is desired that enables them to check the level of trust and knowledge of partners These requirements identify a distributed system in which different actors, acting autonomously on behalf of a user, and each pursuing its own goals, need to interact in order to achieve their goals. Communication and negotiation are paramount. Furthermore, the number and behavior of participants cannot be


202

Dignum & Weigand

fixed a priori, and the system can be expected to expand and change during operation, in number of participants as well as in the amount and kind of knowledge shared. These characteristics indicate a situation for which the agent paradigm is well suited, and therefore, the methodology we propose can be applied.

DEVELOPMENT METHODOLOGY We propose a methodology for the modeling and construction of agent societies based on an organizational, collectivist view that specifies coordination through preestablished roles, responsibilities, and norms. Adapting from the ideas of Frederiksson and Gustavsson (2001), our methodology comprises the phases of observation, construction, and verification. A model results from the application of a set of explanatory principles to the observed properties of an existing system. The model includes the description of the coordination, environment, and behavior characteristics of the observed system. Using this model, an agent system can be constructed by populating the model with agents that will perform the modeled functionality. The resulting system can again be observed and the original model verified and possibly adapted. That is, the introduction of agents will influence the behavior of the observed system, creating the necessity for a dynamic engineering cycle.

Modeling Agent Societies The modeling process starts with the analysis of the domain, resulting in the elicitation of functional (what) and interaction (how) requirements. Interaction requirements specify the coordination structure (market, hierarchy, or network) of the society. Functional requirements determine the behavior of the society and its relationship with the environment. These requirements are the basis for a model society, in which behavior and animation can be verified and compliance to the domain requirements can be checked. This process can be compared to designing a generic enterprise model including roles as accountants, secretaries, and managers, as well as their job descriptions and relationships, and then extending it with the functions necessary to achieve the objectives of the given enterprise. These are, for example, designers and carpenters if the firm is going to manufacture chairs, or programmers and analysts when the company is a software house. More specifically, the modeling part of the methodology consists of the following levels that will be further described in the following subsections:



• • •

203

Coordination—The structure of the domain is determined, and a model is designed based on the collection of coordination models available in the library. Environment—Based on the coordination model design in the previous step, this level describes the interaction between the society and its environment in terms of global requirements and domain ontology. Behavior—Based on the models above, in this level, the intended behavior of the society is described in terms of agent roles and interaction patterns. This process is supported by a library of roles and interaction patterns.

Coordination Level The coordination level results in the choice of a coordination model applicable to the problem. Table 3 gives an overview of the specific characteristics of each coordination model that can be used to determine the applicable model for the domain. The identification of the appropriate model will point out the type of social laws and norms of conduct in the domain and describe the interaction patterns and facilitation needs of the agent society. In order to determine the type of coordination applicable to our example of a knowledge exchange network, we need to look at the wishes expressed by the stakeholders. The desired system should support collaboration and synergy and still enable participants to fulfil their own objectives. That is, collaboration and certification mechanisms are necessary. Furthermore, participants want to be able to determine their own exchange rules and to be assured that there is control over who the other participants are in the Table 3: Social characteristics of different coordination frameworks Society purpose Society goals Relation forms Communication capabilities of agents Interface to outside world

MARKET

NETWORK

HIERARCHY

Exchange Individual goals (determined by the agent) Negotiation (e.g. Contract Net Protocol) Interaction based on standards; communication concerns exchange only Usually open for agents (after identification)

Collaboration Both are possible

Production Determined by the global goals of the society Fixed (e.g. Action / Workflow loop)

Negotiable within society norms and rules Both the interaction Specified on design procedures and exchange can be negotiated Admittance Closed for agents; openn procedure for agents for data (input and output)


204

Dignum & Weigand

environment. In this situation, a market framework is not really suitable, because negotiation in a market follows fixed rules that participants must follow. Moreover, participation is open to any agent, and restriction of role or access is not possible. Also, the hierarchical model can be rejected, because it imposes a fixed partnership relation that is not possible, because partners and sources are not a priori known. However, the expressed requirements and wishes of the stakeholders point clearly to the network framework (cf. Table 3). Environment Level At the environment level, we describe the external behavior of the society, that is, the interaction between society and its environment. This includes the identification of the global functionality of the society and the specification of domain ontologies. In our agent framework, the environment consists basically of other agent societies. That is, interaction between society and its environment means essentially interaction between societies, that is, interaction between agents from different societies. However, because interaction must be governed by the rules of some society, interaction across societies is not directly possible. So, how do societies interact? We propose to draw on the linking-pin concept developed by Rensis Likert in management theory (1961). Likert redefined the role of managers in organization, by realizing that managers are members of at least two groups, and their behaviors reflect the values, norms, and objects of both groups—a manager is a subordinate in one group and a superior in another group. So rather than seeing the manager as a node in a hierarchical tree, Likert puts the manager in the intersection of two groups. Because of this dual membership, a manager can forward information or control from one group to the other. Groups may have different norms, which leave the manager with the task to “translate” between them. A Likert model of an organization is pictured as a set of tiles, where each tile has one or more overlaps with other tiles. Moreover, not only managers can be linking pins, and the set of tiles is not necessarily hierarchically ordered. We have applied the linking-pin principle to solve the problem of interaction between agent societies. Assuming that every agent is owned by an (human) subject, different agent societies will be linked by agents belonging to the same subject. In this way, the problem of communication between societies becomes an intrasubject problem. It is the responsibility of the subject to implement the communication between its various agents and to resolve potential conflicts. Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


205

Table 4: Stakeholder table for the knowledge exchange network STAKEHOLDER

SOCIETY

STAKE

Knowledge owner Knowledge seeker

{product development team} {product development team, management, call-centre} {product development team, actuaries team, legal team} {system management team}

Disseminate knowledge Collect knowledge for tasks

Expert Editor

Generate knowledge Consolidate knowledge

All relevant societies in the environment, and more in particular, the stakeholders, those agents in adjacent societies that have a certain goal or expectation toward the society, must be described. The way in which a linking pin connects two societies is related to the coordination structure. In markets, each agent brings in a goal from outside. In hierarchies, each agent has a certain contribution to the overall goals, and hence, its presence in another society must be instrumental to these goals as well. The case of networks is mixed again: the agent contributes to the network, according to the contracts, but besides that, the network can be instrumental for the linking-pin agent to fulfil his role in the other society or vice versa. A stakeholder table specifies in detail the arrangements (“contract”) between the society and its adjacent societies. Table 4 shows an example of a stakeholder table for the knowledge exchange network. The next step in the environment level is to identify the functional requirements of the domain and the concepts and relationships relevant in the domain. The different stakes users will have on the society determine the requirements. The aim of the knowledge exchange network is to exchange knowledge represented as (XML)-documents describing reports, people, applications, web sites, projects, questions, etc. (This type of exchange “goods” imposes constraints to the task and communicative components of agents, because it demands a complex matching mechanism, because matches are not only at keyword level but require knowledge about relationships, processes, etc. However, this lies outside the scope of this article and will not be further discussed). Ontologies are needed to describe the different concepts relevant to the system. There are two types of ontologies necessary: • Society ontology describes concepts related to exchange to the coordination framework of the society. In the network case, these are owner, seeker, source, etc. • Domain ontology describes concepts related to the application domain. In our example, concepts related to nonlife insurance and specific concepts used at Achmea are part of this ontology.


206

Dignum & Weigand

Behavior Level The purpose of the behavior level is to populate the society model obtained in the previous levels with the functional agent roles and interaction patterns needed to achieve the aims of the system. Here, we are concerned with the high-level definition of agent types. The social rules and norms of conduct associated with each coordination framework will determine the interaction patterns between agents in the society. To support the identification and specification of agent roles, role catalogs providing commonly occurring role models will be developed. In this level, environment requirements (identified in the environment level) are translated into domain roles and interaction rules of the society. Facilitation roles were identified in the coordination level as a result of the coordination model chosen. In the behavior level, specific roles, needed to achieve the aims of the society, as well as characteristics and constraints of each role are specified. The tasks and objectives of each role are derived from the stake each role is holding, as described in the stakeholders’ table in the Environment Level. Furthermore, the society must impose mechanisms for collaboration and certification between roles. For instance, in our example, a special kind of knowledge owner is responsible for the gathering and dissemination of information on a known, fixed list of competitors to knowledge seekers interested. In this case, society norms must enforce that such agents are required to provide all the information they are aware of. This also determines that monitors tracing this type of contracts have the task of checking if information in all companies in the list is indeed provided. In our example, the roles of matchmaker, notary, monitor, and gatekeeper were determined by the choice of a network model during the coordination level. The gatekeeper determines whether an agent can participate in the exchange or not and what kind of role can be fulfilled, the matchmaker matches supply and demand of knowledge between participants, and the notary registers and oversees the exchange commitments decided upon between participants. From the domain requirements identified in the environment level, the roles of knowledge owner, knowledge seeker, already introduced in the previous section, and the roles of editor and expert can be deduced. Editors are responsible for determining the validity and degree of expertise of knowledge items and knowledge owners, and experts can be consulted about a certain area and will be requested to answer questions of knowledge seekers. Furthermore, the role of visitor can be defined, referring to participants who are browsing through the system, are able to consult the knowledge repository, Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.


207

but cannot request some of the services. Figure 1 shows a fragment of the architecture of the society, indicating roles and possible interaction procedures. Interactions in Figure 1 are described for the binary case (one seeker, one owner). However, it is possible to form multipartner contracts. Furthermore, interactions are usually multiaction processes and not one single action. That is, the result of the interaction is usually achieved after a whole conversation. Similar to what in Esteva et al. (2001) is referred to as a scene, interaction between agents is defined through conversations following a well-defined protocol. Figure 1: Fragment of the knowledge exchange network architecture ...

negotiate_partnership

Knowledge owner

register

Knowledge seeker

Knowledge owner

request_partner make_contract

Applicant

apply_sanction

Monitor

membership_ application

appoint Matchmaker

Notary Gatekeeper

Facilitation layer for Network Society

membership_application(X, gatekeeper): This is a negotiation between any agent and the gatekeeper of the society resulting in either an acceptance, that is X will become member of the society, or a rejection. The role the agent will play is also determined in this scene. register(M, matchmaker): Knowledge owners or seekers can register their requests with the matchmaker, who will use this information in future matches request_partner(M, matchmaker): Knowledge owners or seekers request possible partners for an exchange. Results in a possibly empty list of potential partners. negotiate_partnership(M, N): Owners and seekers check the viability of an exchange and determine conditions make_contract(M, N, notary): When an agreement is reached, partners register their commitments with the notary. appoint(notary, monitor): The notary appoints a monitor for a contract. It delegates agreed tasks to the monitor. The monitor will keep track of contract status and will act when an undesired state is reached. apply_sanction(monitor, M): when a breech of contract occurs the monitor will contact the faulty party and apply the sanctions agreed upon (either described in the contract or standard in the institution).


208

Dignum & Weigand

Building Agent Societies The design part of the methodology described above results in a generic model for an agent society. This model specifies the necessary components and relationships that describe the domain. In the next step, an actual agent society will be built by populating the society model with real agents. The behavior of the resulting multiagent system mimics the original system. We assume that the design and implementation of the agents is independent from the design of the society. That is, participating agents are built somewhere else, thus their capabilities cannot be determined a priori. The design of the behavior of the society cannot rely on specific architectural characteristics of the participating agents. Therefore, the society model must impose conditions to the internal model of agents intended to participate in a society (such as the communication language allowed). The functionality of each agent role can be specified in the model in terms of requirements for interaction, communication, behavior, and interface. However, the agent actually performing the role will act according to its own design, and therefore, the behavior of the system may differ from the expected behavior specified in the model, as illustrated in Figure 2. The structure of agent societies will evolve over time as result of interaction between agents. Therefore, the methodology must describe and verify the patterns and processes that model the system dynamics. Figure 2: Influence of agent interpretation in system behavior

modelled role 1

model

modelled modelled role 3 role 2 modelled role 4

Agent A Agent B performed role 2 performed role 3 performed role 1

system

performed role 4



209

CONCLUSIONS We presented a global methodology for the design of agent societies that takes the organizational perspective as the starting point and describes the implications of the coordination model of the organization for the architecture and design method of the agent society being developed. The approach specifies the development steps for the design and development of an agentbased system for a particular domain. It provides a generic frame that directly relates to the organizational perception of a problem and allows for existing methodologies to be used for the development, modeling, and formalization of each step. Although there are several agent-based software engineering methodologies available, these are often either too specific or too formal and are not easily used and accepted. We believe that because of its organizationaloriented approach, our methodology will contribute to the acceptance of multiagent technology by organizations. We are currently applying the ideas described in this paper to develop a Knowledge Exchange Network at Achmea. In the future, a system for agentbased mediation for health care will be developed. Experience gained from these applications will be used to improve the design methodology and the coordination frameworks used. Research has to be continued in several directions. Work is needed on the formal description of agent societies based on the coordination frameworks presented. We also intend to develop libraries of conceptual interaction patterns and agent roles. These libraries will improve and facilitate the design of agent societies. Finally, we plan to look at the compatibility and integration of our ideas with current standardization efforts for agent development, such as Agent UML.

REFERENCES Artikis, A., Kamara, L., & Pitt, J. (2001). Towards an Open Agent Society Model and Animation. Proceedings of the workshop on Agent-Based Simulation II, Passau, (48–55). Bond, A. & Gasser, L. (1988). Readings in Distributed Artificial Intelligence. Morgan-Kaufmann. Brazier, F., Jonker, C., & Treur, J. (2000). Compositional Design and Reuse of a Generic Agent Model. Applied Artificial Intelligence Journal, 14, 491–538. Castelfranchi, C. (2000). Engineering Social Order. Omicini, A., Tolksdorf,


210

Dignum & Weigand

R., & Zambonelli, F. (Eds.). Engineering Societies in the Agents World, Lecture Notes in Computer Science 1972, Heidelberg: SpringerVerlag, pp. 1–19. Ciancarini, P., Omicini, A., & Zambonelli, F. (1999, July). Coordination Models for Multiagent Systems. AgentLink News, 3. Dautenhahn. (2000): Reverse Engineering of Societies — A Biological Perspective. Proceedings of AISB Symposium “Starting from Society — Application of Social Analogies to Computational Systems.” Convention of the Society for the Study of Artificial Intelligence and the Simulation of Behavior (AISB-00), University of Birmingham, England. Davidsson, P. (2000). Emergent Societies of Information Agents. Klusch, M. & Kerschberg, L. (Eds.). Co-operative Information Agents IV — The Future of Information Agents in Cyberspace, Lecture Notes in Artificial Intelligence 1860, Heidelberg: Springer-Verlag, pp. 143– 153. Davidsson, P. (2001). Categories of Artificial Societies. Omicini, A., Petta, P., & Tolkdorf, R. (Eds.). 2nd International Workshop on Emerging Societies in the Agent’s World (ESAW’01), Prague. Dellarocas, C. (2000). Contractual Agent Societies: Negotiated Shared Context and Social Control in Open Multiagent Systems. Proceedings of Workshop on Norms and Institutions in Multiagent Systems, Autonomous Agents 2000, Barcelona. Dignum, V. & Dignum, F. (2001, December). Modeling Agent Societies: Coordination Frameworks and Institutions. Submitted to MASTA 2001, 2nd Workshop on Multiagent Systems: Theory & Applications, Porto, Portugal. Dignum, V., Weigand, H., & Xu, L. (2001). Agent Societies: Towards Framework-Based Design. Wooldridge, M., Ciancarini P., & Weiss, G. (Eds.). Proceedings of the 2nd Workshop on Agent-Oriented Software Engineering, Autonomous Agents 2001, (25–31). Edmonds, B. (1999). Capturing Social Embeddedness: A Constructivist Approach. Adaptive Behavior, 7, 323–348. Esteva, M., Padget, J., & Sierra, C. (2001). Formalising a Language for Institutions and Norms. Proceedings of the 8th International Workshop on Agent Theories, Architectures and Languages, ATAL-2001, Seattle. Ferber, J. & Gutknecht, O. (1998). A Meta-model for the Analysis and Design of Organizations in Multiagent Systems. Proceedings of the 3rd Interna-



211

tional Conference on Multiagent Systems (ICMAS’98), IEEE Computer Society. Frederiksson, M. & Gustavsson, R. (2001). A Methodological Perspective on Engineering of Agent Societies. Omicini, A., Petta, P., & Tolkdorf, R. (Eds.). 2nd International Workshop on Emerging Societies in the Agent’s World (ESAW’01), Prague. Jennings, N.R., Sycara, K., & Wooldridge, M. (1998). A Roadmap for Agent Research and Development. Autonomous Agents and MultiAgent Systems, 1, Boston: Kluwer Academic Press, pp. 275–306. Jonker, C., Klusch, M., & Treur, J. (2000). Design of Collaborative Information Agents. Klusch M. & Kerschberg, L. (Eds.). Cooperative Information Agents IV — The Future of Information Agents in Cyberspace. Lecture Notes in Artificial Intelligence 1860, Heidelberg: SpringerVerlag. pp. 262–283. Likert, R. (1961). New Patterns of Management. New York: McGraw-Hill. Nouwens, J. & Bouwman, H. (1995). Living Apart Together in Electronic Commerce: The Use of Information and Communication Technology to Create Network Organizations. Steinfield, C. (Ed.). Journal of Computer Mediated Communication, Special Issue in Electronic Commerce, 1(3). Retrieved August 3, 2001 from http://www.ascusc.org/ jcmc/vol1/issue3/nouwens.html. Omicini, A. (2000). From Objects to Agent Societies: Abstractions and Methodologies for the Engineering of Open Distributed Systems. Corradi A., Omicini, A., & Poggi, A. (Eds.). WOA 2000 — Dagli Oggetti agli Agenti: Tendenze evolutive dei sistemi software, Bologna: Pitagora Editrice. Omicini, A. (2001). SODA: Societies and Infrastructures in the Analysis and Design of Agent-Based Systems. In Ciancarini P. & Wooldridge, M. (Eds.). Agent-Oriented Software Engineering, Lecture Notes in Computer Science 1957, Heidelberg: Springer-Verlag, pp. 185–194. Papadopoulos, G. & Arbab, F. (1998). Coordination Models and Languages. M.V. Zelkowitz (Ed.). Advances in Computers, 46, Academic Press, pp. 329–400. Powell, W. (1990). Neither Market nor Hierarchy: Network Forms of Organization. Research in Organizational Behavior, 12, 295–336. Sycara, K. (1998). Multiagent Systems. AI Magazine, 19(2), 79–92. Williamson, O. (1975). Markets and Hierarchies: Analysis and Antitrust Implications. New York: Free Press.


212

Dignum & Weigand

Wooldridge, M. (1997). Agent-Based Software Engineering. IEEE Proc. Software Engineering, 144(1), 26–37. Wooldridge, M., Jennings, N., & Kinny, D. (2000). The Gaia Methodology for Agent-Oriented Analysis and Design. Autonomous Agents and MultiAgent Systems, 3(3), 285–312. Zambonelli, F., Jennings, N., Omicini, A., & Wooldridge, M. (2001). AgentOriented Software Engineering for Internet Applications. Omicini, A., Zambonelli, F., Klusch, M., & Tolkdorf, R. (Eds.). Coordination of Internet Agents: Models, Technologies, and Applications, Heidelberg: Springer-Verlag, pp. 326–346.


Cooperation Between Agents

213

Chapter X

Cooperation Between Agents to Evolve Complete Programs Ricardo Aler Universidad Carlos III Madrid, Spain David Camacho Universidad Carlos III Madrid, Spain Alfredo Moscardini University of Sunderland, UK

ABSTRACT In this paper, we present a multiagent system approach with the purpose of building computer programs. Each agent in the multiagent system will be in charge of evolving a part of the program, which in this case, can be the main body of the program or one of its different subroutines. There are two kinds of agents: the manager agent and the genetic programming (GP) agents. The former is in charge of starting the system and returning the results to the user. The GP agents include skills for evolving computer programs, based on the genetic programming paradigm. There are two sorts of GP agents: some can evolve the main body of the program and the others evolve its subroutines. Both kinds of agents cooperate by telling each other their best results found so far, so that the search for a good computer program is made more efficient. In this paper, this multiagent approach is presented and tested empirically. Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

214 Aler, Camacho & Moscardini

INTRODUCTION Nowadays, the field of multiagent systems has shown a lot of interest in agents with intelligent skills, so that the agents can collaborate to solve complex problems. For instance, there are successful multiagent applications, and agent-based applications, that allow a user to solve complex problems through the use of artificial intelligence (AI) techniques, in several domains like the following: • Industrial applications—CAPlan (Muñoz-Avila, 2000), NaCoDAE (Breslow, 1998), Paris (Bergmann, 1996) • Information retrieval and knowledge integration—SIMS (Knobloc, 1997), TSIMMIS (Chawathe, 1994) • Web domains—WebPlan (Hullen, 1999), Ariadne (Knoblock, 1998), Heracles (Knoblock, 1998), Letizia (Lieberman, 1995), MetaCrawler (Selberg, 1995), MAPWeb (Camacho, 2001) Several AI techniques have been used in those previous systems, like planning (Allen, 1990), case-based reasoning (Aamodt, 1994) or case-based planning (Hammond, 1986), etc. Those techniques have been successfully integrated in agent-based or multiagent systems, obtaining excellent results. In particular, genetics algorithms (GA) have also been applied in multiagent environments. For instance, it has been applied for evolving agent behaviors and agent groups to solve complex problems. In this approach, a single genetic programming (GP) algorithm is used to evolve the behavior of all the agents in the society (this is called the homogeneous approach, because all the agents in the society share the same controlling program) (Arora, 1997; Haynes, 1997; Munde, 2000). Another approach is the coevolution of agents in which different populations evolve one agent in the team (Bull, 1997; Puppala, 1998). In this paper, we use a collaborative multiagent system to generate a computer program that is able to solve a particular problem (i.e., we follow the coevolutionary approach, but in our case, the task is to build a computer program). A computer program is made of different parts: the main program and several subroutines. Different agents will generate every one of these parts. Finally, there will be a “collecting” agent that will put all of the parts together. A way of finding good subroutines (and main programs) is to carry out a search in the space of subroutines. But in order to do so, it is necessary to have a measure to determine whether a subroutine is good or not (the so-called heuristic function, in the search paradigm). However, in many cases, it is not possible to evaluate different parts of a program separately, only a whole computer program can be properly evaluated. This can be done, for instance, Copyright © 2003, Idea Group Inc. Copying or distributing in print or electronic forms without written permission of Idea Group Inc. is prohibited.

Cooperation Between Agents

215

by running it and determining how well it solves a problem. But in order to do so, an agent specialized in searching for the first subroutine of a program must also have the rest of the subroutines and the main program, so that a complete program can be run. At this point, the agents can collaborate by sending the rest of the agents the best result they have found so far. By doing so, two points can be achieved: • An agent can evaluate a subroutine (or a main program) being considered at that moment by putting it together with the other parts sent by the other agents. • Different agents can send their best subroutine (or main program) found so far, so that the other agents can use them. This way, some agents can use improvements found by other agents as soon as they are available. By doing this, we hope that the search process will be more efficient. An example of this is science, where discoveries by one scientist are quickly transferred to other scientists, leading to quick advance. If discoveries were transferred only by word of mouth, advance would be much slower. However, it is not clear whether other agents can easily and always adopt the best results obtained by one agent, although we expect that in the long run, different agents would become adept. In this paper, we would like to empirically study this issue. Many techniques could be used to search in the space of computer programs (or subroutines), which is the basic skill of our agents. We considered that the most appropriate is genetic programming (GP) (Koza, 1994). GP is a genetic algorithm that starts with a population of randomly generated computer programs and evolves them by using the Darwinian principle of the supervivence of the fittest. This amounts to a particular beam-search algorithm (Tackett, 1994). Usually, computer programs in GP are represented as parse trees, made of functions that return a value, although this need not necessarily be the case.

GENETIC PROGRAMMING AND AUTOMATICALLY DEFINED FUNCTIONS Genetic Programming Genetic programming is a technique to automatically find computer programs that solve specific problems (Koza, 1994). It uses a genetic algorithm to search the space of possible computer programs. It has three basic elements:


216 Aler, Camacho & Moscardini

Figure 1: Crossover between two individuals represented as parse trees

•

A population P of individuals i ∈ P. Every individual is a computer program. Computer programs are usually represented by using primitive functions selected from a set F and terminals chosen from a set T. For instance, the following is an example of genetic programming individual: IF X

Intelligent Agent Software Engineering

Agent-Oriented Software Engineering X

Agent-Oriented Software Engineering XI - AOSE XI

Software Engineering for Multi-Agent Systems II

Agent engineering

Oracle Intelligent Agent. User's Guide

Software Engineering

Software Engineering

Software Engineering

Agent-Oriented Software Engineering VII, 7 conf., AOSE 2006

Agent-Oriented Software Engineering II, 2 conf., AOSE 2001

Agent-Oriented Software Engineering, 1 conf., AOSE 2000

Agent-Oriented Software Engineering 8 conf., AOSE 2007

Agent-Oriented Software Engineering VI, 6 conf., AOSE 2005

Developing Intelligent Agent Systems: A Practical Guide

Intelligent agent technology: research and development

Software Security Engineering

Value-Based Software Engineering

Software Engineering 3

Software engineering measurement

Software Engineering, 9th Edition

Programming and Software Engineering)

Engineering Software for Accessibility

Value-Based Software Engineering

Software Engineering (6th Edition)

Rationale-Based Software Engineering

Programming and Software Engineering)

Advances in Software Engineering

Agile software engineering

Software Engineering Handbook

Automotive Software Engineering

Intelligent Agent Software Engineering

Agent-Oriented Software Engineering X

Agent-Oriented Software Engineering XI - AOSE XI

Software Engineering for Multi-Agent Systems II

Agent engineering

Oracle Intelligent Agent. User's Guide

Software Engineering

Software Engineering

Software Engineering

Agent-Oriented Software Engineering VII, 7 conf., AOSE 2006

Agent-Oriented Software Engineering II, 2 conf., AOSE 2001

Agent-Oriented Software Engineering, 1 conf., AOSE 2000

Agent-Oriented Software Engineering 8 conf., AOSE 2007

Agent-Oriented Software Engineering VI, 6 conf., AOSE 2005

Developing Intelligent Agent Systems: A Practical Guide

Intelligent agent technology: research and development

Software Security Engineering

Value-Based Software Engineering

Software Engineering 3

Software engineering measurement

Software Engineering, 9th Edition

Programming and Software Engineering)

Engineering Software for Accessibility

Value-Based Software Engineering

Software Engineering (6th Edition)

Rationale-Based Software Engineering

Programming and Software Engineering)

Advances in Software Engineering

Agile software engineering

Software Engineering Handbook

Automotive Software Engineering

Recommend Documents